librenotes/docs/operations.md

Operations

This document covers deployment, backups, and rollback for the librenot.es production environment. For developer / contributor docs see the top-level README.

Deployment

The repository ships a CI workflow at .gitea/workflows/deploy.yml that builds and pushes a Docker image on every push to main and on every vX.Y.Z tag, then SSHes to the deployment host and runs docker compose pull && up -d.

The workflow is gated on the repository variable DEPLOY_ENABLED=true. Deployment is opt-in; flipping the variable disables the workflow without removing the file.

Required secrets

The workflow expects these secrets and variables on the repo:

Name	Type	Purpose
REGISTRY	secret	registry hostname, e.g. git.librete.ch
REGISTRY_USER	secret	robot account or PAT username with package:write
REGISTRY_PASS	secret	robot/PAT token
DEPLOY_HOST	secret	user@host SSH target, e.g. root@netcup
DEPLOY_KEY	secret	passphrase-less private key (PEM)
DEPLOY_PATH	secret	absolute path on host with the compose stack, e.g. /srv/librenotes
HEALTH_URL	secret	e.g. https://ln.cloud.librete.ch/healthz
DEPLOY_ENABLED	variable	true to enable the workflow

The image is pushed to ${REGISTRY}/public/librenotes; main pushes publish :main and :<sha7>, tag pushes publish :vX.Y.Z and :latest.

Production compose stack

On the deployment host, place compose.yaml and compose.netcup.yaml from this repo at $DEPLOY_PATH, together with an .env file containing LIBRENOTES_IMAGE plus the runtime configuration (JWT secret, SMTP credentials, public base URL). See .env.netcup.example for the full key list.

Bring it up with:

docker compose -f compose.yaml -f compose.netcup.yaml pull
docker compose -f compose.yaml -f compose.netcup.yaml up -d

Rollback

Tag pushes publish immutable tags (:vX.Y.Z). To pin or roll back, edit LIBRENOTES_IMAGE in /srv/librenotes/.env and re-run pull + up. The deployment workflow itself never rewrites the file — failed health checks abort the workflow and the operator redeploys manually.

# Example: roll back to v0.1.0
ssh netcup
cd /srv/librenotes
perl -i -pe 's|^LIBRENOTES_IMAGE=.*|LIBRENOTES_IMAGE=git.librete.ch/public/librenotes:v0.1.0|' .env
docker compose -f compose.yaml -f compose.netcup.yaml pull
docker compose -f compose.yaml -f compose.netcup.yaml up -d

Gitea Actions runner

Workflows run on the netcup act_runner (see runner/ stack in the netcup umbrella). Both ci.yml and deploy.yml declare container: image: git.librete.ch/public/runner-image:v1 — a bespoke Ubuntu 24.04 image (built and signed by us, hosted on the same Gitea instance) that bundles git, make, node, perl, ssh, and a docker CLI. The image's runner user is pre-joined to docker group gid 998 so the auto-mounted /var/run/docker.sock is writable without --user root.

The runner config (runner/config.yaml) whitelists /var/run/docker.sock under valid_volumes to allow the auto-mount.

Backups

scripts/backup.sh is a self-contained backup driver suitable for running from cron, a systemd timer, or the supplied scripts/librenotes-backup.{service,timer} units.

Required tools on the backup host: sqlite3, tar, gzip, rsync, and optionally rclone for object-store off-site copy. On Debian/Ubuntu: apt install sqlite3 rsync rclone.

What is backed up

• The SQLite database ($LIBRENOTES_DB) via the SQLite .backup command, which produces a consistent online snapshot without needing to stop the application.
• The per-tenant note directory ($LIBRENOTES_DATA_DIR) as a gzipped tar.
• An info.txt in each archive recording the backup timestamp, hostname, and version.

Off-site copy

The script supports two off-site transports; either or both can be configured. Without either, backups stay on local disk only.

• BACKUP_REMOTE — rclone destination (e.g. s3:librenotes-backups, b2:librenotes). Uses rclone copy to upload each new archive. Best for object-store targets.
• BACKUP_REMOTE_RSYNC — rsync destination (e.g. backup@rsync.net:librenotes/). Each run rsyncs into <root>/YYYY-MM-DD/ and passes --link-dest=../<previous-day>/, so unchanged archives become hard links — daily snapshots cost (almost) nothing extra. Set BACKUP_REMOTE_SSH_KEY to the path of a passphrase-less private key if the host's default identity is wrong (recommended for rsync.net-style restricted accounts).

For netcup, the operator's choice is captured in issue #41. Until a target is chosen, leave both unset and run the timer locally to verify archives generate cleanly.

Retention

scripts/backup-prune.sh enforces the policy "keep 30 daily

• 12 monthly". Run it from the same timer as the backup. Files keep the form librenotes-YYYYMMDD-HHMMSS.tar.gz so the prune script can sort and select by name alone.

Restore test

scripts/backup-restore-test.sh picks the most recent archive, extracts it into a scratch directory, runs sqlite3 .schema on the database to confirm it is readable, and verifies that the note tar lists at least one entry. It is wired into a separate weekly timer so a silent backup-corruption regression cannot hide indefinitely.

Systemd

Drop scripts/librenotes-backup.{service,timer} into /etc/systemd/system/, then:

systemctl daemon-reload
systemctl enable --now librenotes-backup.timer

Confirm with:

systemctl list-timers librenotes-backup.timer

Restore procedure

A restore is a four-step process. Run it on a spare directory (e.g. /srv/librenotes-restore-YYYYMMDD/) — never overwrite the live /srv/librenotes/data or /srv/librenotes/state until the parallel stack passes verification.

# 1. Pick an archive (local or pulled back from the off-host target).
ARCHIVE=/var/backups/librenotes/librenotes-20260415-031712.tar.gz

# 2. Lay out a parallel stack dir.
RESTORE_DIR=/srv/librenotes-restore
mkdir -p "$RESTORE_DIR"/{data,state}
chown -R 65532:65532 "$RESTORE_DIR"/{data,state}    # distroless nonroot uid

# 3. Unpack the archive into a scratch dir, then move the pieces
#    into place.
SCRATCH=$(mktemp -d)
tar -C "$SCRATCH" -xzf "$ARCHIVE"
install -m 0644 -o 65532 -g 65532 "$SCRATCH/librenotes.db" \
        "$RESTORE_DIR/state/librenotes.db"
tar -C "$RESTORE_DIR/data" -xzf "$SCRATCH/notes.tar.gz"
chown -R 65532:65532 "$RESTORE_DIR/data"
rm -rf "$SCRATCH"

# 4. Bring up a parallel compose stack (dedicated docker compose
#    project name + non-conflicting LIBRENOTES_BASE_URL) and verify.
cd "$RESTORE_DIR"
cp /srv/librenotes/.env .env
perl -i -pe 's|^LIBRENOTES_BASE_URL=.*|LIBRENOTES_BASE_URL=http://localhost:18080|' .env
cp /srv/librenotes/{compose.yaml,compose.netcup.yaml} .
# Override port + project name so the parallel stack does not
# clash with the live one.
docker compose -p librenotes-restore \
  -f compose.yaml \
  --profile=restore \
  up -d \
  -e LIBRENOTES_PORT=18080
curl -fsS http://localhost:18080/healthz
# Sign in with a known test user, click around, confirm note state.
docker compose -p librenotes-restore down

Once the parallel stack proves the archive is intact, swap into production with a maintenance window:

docker compose -f compose.yaml -f compose.netcup.yaml down
mv /srv/librenotes/data /srv/librenotes/data.bak-$(date +%s)
mv /srv/librenotes/state /srv/librenotes/state.bak-$(date +%s)
mv /srv/librenotes-restore/data /srv/librenotes/data
mv /srv/librenotes-restore/state /srv/librenotes/state
docker compose -f compose.yaml -f compose.netcup.yaml up -d
curl -fsS https://ln.cloud.librete.ch/healthz

If anything goes wrong, the .bak-* directories are still there to restore from.