libretech
4947b64569
CI / ci (pull_request) Failing after 5m37s
backup.sh now supports BACKUP_REMOTE_RSYNC alongside BACKUP_REMOTE. The rsync path writes <root>/YYYY-MM-DD/<archive> on the target with --link-dest pointing at the previous day's directory, so unchanged archives become hard links and daily snapshots cost almost zero extra bytes. BACKUP_REMOTE_SSH_KEY routes the rsync ssh leg to a dedicated identity (e.g. rsync.net restricted accounts). Timer moved to 03:00 Europe/Berlin (was 03:17 UTC) per #41. docs/operations.md: full restore procedure (parallel stack first, then atomic swap) plus the rsync vs rclone trade-off. Closes most of #41 — the only remaining task is the operator's choice of off-host target.
217 lines
7.7 KiB
Markdown
217 lines
7.7 KiB
Markdown
# Operations
|
|
|
|
This document covers deployment, backups, and rollback for the
|
|
librenot.es production environment. For developer / contributor
|
|
docs see the top-level [README](../README.md).
|
|
|
|
## Deployment
|
|
|
|
The repository ships a CI workflow at `.gitea/workflows/deploy.yml`
|
|
that builds and pushes a Docker image on every push to `main` and
|
|
on every `vX.Y.Z` tag, then SSHes to the deployment host and runs
|
|
`docker compose pull && up -d`.
|
|
|
|
The workflow is gated on the repository variable
|
|
`DEPLOY_ENABLED=true`. Deployment is opt-in; flipping the variable
|
|
disables the workflow without removing the file.
|
|
|
|
### Required secrets
|
|
|
|
The workflow expects these secrets and variables on the repo:
|
|
|
|
| Name | Type | Purpose |
|
|
| ---- | ---- | ------- |
|
|
| `REGISTRY` | secret | registry hostname, e.g. `git.librete.ch` |
|
|
| `REGISTRY_USER` | secret | robot account or PAT username with `package:write` |
|
|
| `REGISTRY_PASS` | secret | robot/PAT token |
|
|
| `DEPLOY_HOST` | secret | `user@host` SSH target, e.g. `root@netcup` |
|
|
| `DEPLOY_KEY` | secret | passphrase-less private key (PEM) |
|
|
| `DEPLOY_PATH` | secret | absolute path on host with the compose stack, e.g. `/srv/librenotes` |
|
|
| `HEALTH_URL` | secret | e.g. `https://ln.cloud.librete.ch/healthz` |
|
|
| `DEPLOY_ENABLED` | variable | `true` to enable the workflow |
|
|
|
|
The image is pushed to `${REGISTRY}/public/librenotes`; main pushes
|
|
publish `:main` and `:<sha7>`, tag pushes publish `:vX.Y.Z` and
|
|
`:latest`.
|
|
|
|
### Production compose stack
|
|
|
|
On the deployment host, place `compose.yaml` and
|
|
`compose.netcup.yaml` from this repo at `$DEPLOY_PATH`, together
|
|
with an `.env` file containing `LIBRENOTES_IMAGE` plus the runtime
|
|
configuration (JWT secret, SMTP credentials, public base URL).
|
|
See `.env.netcup.example` for the full key list.
|
|
|
|
Bring it up with:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Rollback
|
|
|
|
Tag pushes publish immutable tags (`:vX.Y.Z`). To pin or roll back,
|
|
edit `LIBRENOTES_IMAGE` in `/srv/librenotes/.env` and re-run pull +
|
|
up. The deployment workflow itself never rewrites the file — failed
|
|
health checks abort the workflow and the operator redeploys
|
|
manually.
|
|
|
|
```sh
|
|
# Example: roll back to v0.1.0
|
|
ssh netcup
|
|
cd /srv/librenotes
|
|
perl -i -pe 's|^LIBRENOTES_IMAGE=.*|LIBRENOTES_IMAGE=git.librete.ch/public/librenotes:v0.1.0|' .env
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Gitea Actions runner
|
|
|
|
Workflows run on the netcup `act_runner` (see `runner/` stack in
|
|
the netcup umbrella). Both `ci.yml` and `deploy.yml` declare
|
|
`container: image: git.librete.ch/libretech/runner-image:v1` —
|
|
a bespoke Ubuntu 24.04 image (built and signed by us, hosted on
|
|
the same Gitea instance) that bundles `git`, `make`, `node`,
|
|
`perl`, `ssh`, and a docker CLI. The image's `runner` user is
|
|
pre-joined to `docker` group gid 998 so the auto-mounted
|
|
`/var/run/docker.sock` is writable without `--user root`.
|
|
|
|
The runner config (`runner/config.yaml`) whitelists
|
|
`/var/run/docker.sock` under `valid_volumes` to allow the
|
|
auto-mount.
|
|
|
|
## Backups
|
|
|
|
`scripts/backup.sh` is a self-contained backup driver suitable for
|
|
running from cron, a systemd timer, or the supplied
|
|
`scripts/librenotes-backup.{service,timer}` units.
|
|
|
|
Required tools on the backup host: `sqlite3`, `tar`, `gzip`,
|
|
`rsync`, and optionally `rclone` for object-store off-site copy.
|
|
On Debian/Ubuntu: `apt install sqlite3 rsync rclone`.
|
|
|
|
### What is backed up
|
|
|
|
- The SQLite database (`$LIBRENOTES_DB`) via the SQLite
|
|
`.backup` command, which produces a consistent online snapshot
|
|
without needing to stop the application.
|
|
- The per-tenant note directory (`$LIBRENOTES_DATA_DIR`) as a
|
|
gzipped tar.
|
|
- An `info.txt` in each archive recording the backup timestamp,
|
|
hostname, and version.
|
|
|
|
### Off-site copy
|
|
|
|
The script supports two off-site transports; either or both can be
|
|
configured. Without either, backups stay on local disk only.
|
|
|
|
- `BACKUP_REMOTE` — `rclone` destination (e.g.
|
|
`s3:librenotes-backups`, `b2:librenotes`). Uses `rclone copy` to
|
|
upload each new archive. Best for object-store targets.
|
|
- `BACKUP_REMOTE_RSYNC` — `rsync` destination (e.g.
|
|
`backup@rsync.net:librenotes/`). Each run rsyncs into
|
|
`<root>/YYYY-MM-DD/` and passes
|
|
`--link-dest=../<previous-day>/`, so unchanged archives become
|
|
hard links — daily snapshots cost (almost) nothing extra. Set
|
|
`BACKUP_REMOTE_SSH_KEY` to the path of a passphrase-less private
|
|
key if the host's default identity is wrong (recommended for
|
|
rsync.net-style restricted accounts).
|
|
|
|
For netcup, the operator's choice is captured in
|
|
[issue #41](https://git.librete.ch/public/librenotes/issues/41).
|
|
Until a target is chosen, leave both unset and run the timer
|
|
locally to verify archives generate cleanly.
|
|
|
|
### Retention
|
|
|
|
`scripts/backup-prune.sh` enforces the policy "keep 30 daily
|
|
+ 12 monthly". Run it from the same timer as the backup. Files
|
|
keep the form `librenotes-YYYYMMDD-HHMMSS.tar.gz` so the prune
|
|
script can sort and select by name alone.
|
|
|
|
### Restore test
|
|
|
|
`scripts/backup-restore-test.sh` picks the most recent archive,
|
|
extracts it into a scratch directory, runs `sqlite3 .schema` on
|
|
the database to confirm it is readable, and verifies that the
|
|
note tar lists at least one entry. It is wired into a separate
|
|
weekly timer so a silent backup-corruption regression cannot hide
|
|
indefinitely.
|
|
|
|
### Systemd
|
|
|
|
Drop `scripts/librenotes-backup.{service,timer}` into
|
|
`/etc/systemd/system/`, then:
|
|
|
|
```sh
|
|
systemctl daemon-reload
|
|
systemctl enable --now librenotes-backup.timer
|
|
```
|
|
|
|
Confirm with:
|
|
|
|
```sh
|
|
systemctl list-timers librenotes-backup.timer
|
|
```
|
|
|
|
### Restore procedure
|
|
|
|
A restore is a four-step process. Run it on a *spare* directory
|
|
(e.g. `/srv/librenotes-restore-YYYYMMDD/`) — never overwrite the
|
|
live `/srv/librenotes/data` or `/srv/librenotes/state` until the
|
|
parallel stack passes verification.
|
|
|
|
```sh
|
|
# 1. Pick an archive (local or pulled back from the off-host target).
|
|
ARCHIVE=/var/backups/librenotes/librenotes-20260415-031712.tar.gz
|
|
|
|
# 2. Lay out a parallel stack dir.
|
|
RESTORE_DIR=/srv/librenotes-restore
|
|
mkdir -p "$RESTORE_DIR"/{data,state}
|
|
chown -R 65532:65532 "$RESTORE_DIR"/{data,state} # distroless nonroot uid
|
|
|
|
# 3. Unpack the archive into a scratch dir, then move the pieces
|
|
# into place.
|
|
SCRATCH=$(mktemp -d)
|
|
tar -C "$SCRATCH" -xzf "$ARCHIVE"
|
|
install -m 0644 -o 65532 -g 65532 "$SCRATCH/librenotes.db" \
|
|
"$RESTORE_DIR/state/librenotes.db"
|
|
tar -C "$RESTORE_DIR/data" -xzf "$SCRATCH/notes.tar.gz"
|
|
chown -R 65532:65532 "$RESTORE_DIR/data"
|
|
rm -rf "$SCRATCH"
|
|
|
|
# 4. Bring up a parallel compose stack (dedicated docker compose
|
|
# project name + non-conflicting LIBRENOTES_BASE_URL) and verify.
|
|
cd "$RESTORE_DIR"
|
|
cp /srv/librenotes/.env .env
|
|
perl -i -pe 's|^LIBRENOTES_BASE_URL=.*|LIBRENOTES_BASE_URL=http://localhost:18080|' .env
|
|
cp /srv/librenotes/{compose.yaml,compose.netcup.yaml} .
|
|
# Override port + project name so the parallel stack does not
|
|
# clash with the live one.
|
|
docker compose -p librenotes-restore \
|
|
-f compose.yaml \
|
|
--profile=restore \
|
|
up -d \
|
|
-e LIBRENOTES_PORT=18080
|
|
curl -fsS http://localhost:18080/healthz
|
|
# Sign in with a known test user, click around, confirm note state.
|
|
docker compose -p librenotes-restore down
|
|
```
|
|
|
|
Once the parallel stack proves the archive is intact, swap into
|
|
production with a maintenance window:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f compose.netcup.yaml down
|
|
mv /srv/librenotes/data /srv/librenotes/data.bak-$(date +%s)
|
|
mv /srv/librenotes/state /srv/librenotes/state.bak-$(date +%s)
|
|
mv /srv/librenotes-restore/data /srv/librenotes/data
|
|
mv /srv/librenotes-restore/state /srv/librenotes/state
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
curl -fsS https://ln.cloud.librete.ch/healthz
|
|
```
|
|
|
|
If anything goes wrong, the `.bak-*` directories are still there
|
|
to restore from.
|