Files

Michael Czechowski bcccba92f7 Add deploy workflow and backup tooling

CI deployment (.gitea/workflows/deploy.yml):
- Two jobs (build, deploy) gated on the repo variable
  DEPLOY_ENABLED=true so the workflow exists but does nothing
  until secrets and host are configured.
- Build pushes two image tags per run: rolling :main + the short
  SHA on main, or vX.Y.Z + :latest on tag pushes. Immutable per
  commit/tag tags make rollback trivial.
- Deploy SSHes to DEPLOY_HOST, runs docker compose pull && up -d
  in DEPLOY_PATH, then polls HEALTH_URL for up to a minute. A
  failed health check fails the workflow, which is the alert.
- Required secrets and the rollback procedure are documented in
  docs/operations.md.

Backup tooling (scripts/):
- backup.sh: SQLite online .backup snapshot + tarball of the
  per-tenant data dir + info.txt header, all wrapped into a
  single librenotes-YYYYMMDD-HHMMSS.tar.gz. Optional BACKUP_REMOTE
  triggers an rclone copy for off-site storage.
- backup-prune.sh: enforces retention "30 daily + 12 monthly".
  Sorts archives by filename (date is in the name so lex order
  matches chronological) and keeps the newest 30 plus the newest
  archive for each of the most recent 12 months.
- backup-restore-test.sh: extracts the most recent archive into
  a tmpdir, runs sqlite3 .schema (proves DB readability), and
  asserts the notes tar has at least one entry. Failure is the
  alert. Wired into a separate weekly timer.
- librenotes-backup.{service,timer}: systemd units for the daily
  03:17 UTC run with 5min jitter; ProtectSystem=strict, only
  /var/backups/librenotes is writable.
- librenotes-backup-verify.{service,timer}: weekly Monday
  04:00 UTC restore test.

Closes #26 and #27.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-28 22:49:40 +02:00

3.8 KiB

Raw Permalink Blame History

Operations

This document covers deployment, backups, and rollback for the librenot.es production environment. For developer / contributor docs see the top-level README.

Deployment

The repository ships a CI workflow at .gitea/workflows/deploy.yml that builds and pushes a Docker image on every push to main and on every vX.Y.Z tag, then SSHes to the deployment host and runs docker compose pull && up -d.

The workflow is gated on the repository variable DEPLOY_ENABLED=true. Deployment is opt-in; flipping the variable disables the workflow without removing the file.

Required secrets

The workflow expects these secrets and variables on the repo:

Name	Type	Purpose
`REGISTRY`	secret	hostname of the OCI registry
`REGISTRY_USER`	secret	robot account
`REGISTRY_PASS`	secret	robot token
`DEPLOY_HOST`	secret	`user@host` SSH target
`DEPLOY_KEY`	secret	passphrase-less private key
`DEPLOY_PATH`	secret	absolute path on host with `docker-compose.*.yml`
`HEALTH_URL`	secret	e.g. `https://librenot.es/healthz`
`DEPLOY_ENABLED`	variable	`true` to enable the workflow

Production compose stack

On the deployment host, place docker-compose.yml and docker-compose.prod.yml from this repo at $DEPLOY_PATH, together with an .env file containing the runtime configuration (JWT secret, SMTP credentials, public base URL, image tag).

Bring it up with:

docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Rollback

Image tags are immutable per commit / version. To roll back, set LIBRENOTES_IMAGE to the previous tag in .env and run the same up -d command. The deployment workflow does not auto-rollback on health-check failure — failed health alerts the operator via the workflow itself, who can then redeploy the prior tag manually.

# Example: roll back to v0.1.2
sed -i 's/^LIBRENOTES_IMAGE=.*/LIBRENOTES_IMAGE=registry.librete.ch\/librenotes:v0.1.2/' .env
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Backups

scripts/backup.sh is a self-contained backup driver suitable for running from cron, a systemd timer, or the supplied scripts/librenotes-backup.{service,timer} units.

Required tools on the backup host: sqlite3, tar, gzip, and optionally rclone for off-site copy. On Debian/Ubuntu: apt install sqlite3 rclone.

What is backed up

The SQLite database ($LIBRENOTES_DB) via the SQLite .backup command, which produces a consistent online snapshot without needing to stop the application.
The per-tenant note directory ($LIBRENOTES_DATA_DIR) as a gzipped tar.
An info.txt in each archive recording the backup timestamp, hostname, and version.

Off-site copy

Set BACKUP_REMOTE to an rclone destination (e.g. s3:librenotes-backups). When set, the script invokes rclone copy to upload the archive after creation. Without it, backups stay on the local disk only.

Retention

scripts/backup-prune.sh enforces the policy "keep 30 daily

12 monthly". Run it from the same timer as the backup. Files keep the form librenotes-YYYYMMDD-HHMMSS.tar.gz so the prune script can sort and select by name alone.

Restore test

scripts/backup-restore-test.sh picks the most recent archive, extracts it into a scratch directory, runs sqlite3 .schema on the database to confirm it is readable, and verifies that the note tar lists at least one entry. It is wired into a separate weekly timer so a silent backup-corruption regression cannot hide indefinitely.

Systemd

Drop scripts/librenotes-backup.{service,timer} into /etc/systemd/system/, then:

systemctl daemon-reload
systemctl enable --now librenotes-backup.timer

Confirm with:

systemctl list-timers librenotes-backup.timer

3.8 KiB Raw Permalink Blame History