librenotes/docs/operations.md

# Operations

This document covers deployment, backups, and rollback for the
librenot.es production environment. For developer / contributor
docs see the top-level [README](../README.md).

## Deployment

The repository ships a CI workflow at `.gitea/workflows/deploy.yml`
that builds and pushes a Docker image on every push to `main` and
on every `vX.Y.Z` tag, then SSHes to the deployment host and runs
`docker compose pull && up -d`.

The workflow is gated on the repository variable
`DEPLOY_ENABLED=true`. Deployment is opt-in; flipping the variable
disables the workflow without removing the file.

### Required secrets

The workflow expects these secrets and variables on the repo:

| Name | Type | Purpose |
| ---- | ---- | ------- |
| `REGISTRY` | secret | registry hostname, e.g. `git.librete.ch` |
| `REGISTRY_USER` | secret | robot account or PAT username with `package:write` |
| `REGISTRY_PASS` | secret | robot/PAT token |
| `DEPLOY_HOST` | secret | `user@host` SSH target, e.g. `root@netcup` |
| `DEPLOY_KEY` | secret | passphrase-less private key (PEM) |
| `DEPLOY_PATH` | secret | absolute path on host with the compose stack, e.g. `/srv/librenotes` |
| `HEALTH_URL` | secret | e.g. `https://ln.cloud.librete.ch/healthz` |
| `DEPLOY_ENABLED` | variable | `true` to enable the workflow |

The image is pushed to `${REGISTRY}/public/librenotes`; main pushes
publish `:main` and `:<sha7>`, tag pushes publish `:vX.Y.Z` and
`:latest`.

### Production compose stack

On the deployment host, place `compose.yaml` and
`compose.netcup.yaml` from this repo at `$DEPLOY_PATH`, together
with an `.env` file containing `LIBRENOTES_IMAGE` plus the runtime
configuration (JWT secret, SMTP credentials, public base URL).
See `.env.netcup.example` for the full key list.

Bring it up with:

```sh
docker compose -f compose.yaml -f compose.netcup.yaml pull
docker compose -f compose.yaml -f compose.netcup.yaml up -d
```

### Rollback

Tag pushes publish immutable tags (`:vX.Y.Z`). To pin or roll back,
edit `LIBRENOTES_IMAGE` in `/srv/librenotes/.env` and re-run pull +
up. The deployment workflow itself never rewrites the file — failed
health checks abort the workflow and the operator redeploys
manually.

```sh
# Example: roll back to v0.1.0
ssh netcup
cd /srv/librenotes
perl -i -pe 's|^LIBRENOTES_IMAGE=.*|LIBRENOTES_IMAGE=git.librete.ch/public/librenotes:v0.1.0|' .env
docker compose -f compose.yaml -f compose.netcup.yaml pull
docker compose -f compose.yaml -f compose.netcup.yaml up -d
```

### Gitea Actions runner

Workflows run on the netcup `act_runner` (see `runner/` stack in
the netcup umbrella). Both `ci.yml` and `deploy.yml` declare
`container: image: git.librete.ch/public/runner-image:v1` —
a bespoke Ubuntu 24.04 image (built and signed by us, hosted on
the same Gitea instance) that bundles `git`, `make`, `node`,
`perl`, `ssh`, and a docker CLI. The image's `runner` user is
pre-joined to `docker` group gid 998 so the auto-mounted
`/var/run/docker.sock` is writable without `--user root`.

The runner config (`runner/config.yaml`) whitelists
`/var/run/docker.sock` under `valid_volumes` to allow the
auto-mount.

## Backups

`scripts/backup.sh` is a self-contained backup driver suitable for
running from cron, a systemd timer, or the supplied
`scripts/librenotes-backup.{service,timer}` units.

Required tools on the backup host: `sqlite3`, `tar`, `gzip`,
`rsync`, and optionally `rclone` for object-store off-site copy.
On Debian/Ubuntu: `apt install sqlite3 rsync rclone`.

### What is backed up

- The SQLite database (`$LIBRENOTES_DB`) via the SQLite
  `.backup` command, which produces a consistent online snapshot
  without needing to stop the application.
- The per-tenant note directory (`$LIBRENOTES_DATA_DIR`) as a
  gzipped tar.
- An `info.txt` in each archive recording the backup timestamp,
  hostname, and version.

### Off-site copy

The script supports two off-site transports; either or both can be
configured. Without either, backups stay on local disk only.

- `BACKUP_REMOTE` — `rclone` destination (e.g.
  `s3:librenotes-backups`, `b2:librenotes`). Uses `rclone copy` to
  upload each new archive. Best for object-store targets.
- `BACKUP_REMOTE_RSYNC` — `rsync` destination (e.g.
  `backup@rsync.net:librenotes/`). Each run rsyncs into
  `<root>/YYYY-MM-DD/` and passes
  `--link-dest=../<previous-day>/`, so unchanged archives become
  hard links — daily snapshots cost (almost) nothing extra. Set
  `BACKUP_REMOTE_SSH_KEY` to the path of a passphrase-less private
  key if the host's default identity is wrong (recommended for
  rsync.net-style restricted accounts).

For netcup, the operator's choice is captured in
[issue #41](https://git.librete.ch/public/librenotes/issues/41).
Until a target is chosen, leave both unset and run the timer
locally to verify archives generate cleanly.

### Retention

`scripts/backup-prune.sh` enforces the policy "keep 30 daily
+ 12 monthly". Run it from the same timer as the backup. Files
keep the form `librenotes-YYYYMMDD-HHMMSS.tar.gz` so the prune
script can sort and select by name alone.

### Restore test

`scripts/backup-restore-test.sh` picks the most recent archive,
extracts it into a scratch directory, runs `sqlite3 .schema` on
the database to confirm it is readable, and verifies that the
note tar lists at least one entry. It is wired into a separate
weekly timer so a silent backup-corruption regression cannot hide
indefinitely.

### Systemd

Drop `scripts/librenotes-backup.{service,timer}` into
`/etc/systemd/system/`, then:

```sh
systemctl daemon-reload
systemctl enable --now librenotes-backup.timer
```

Confirm with:

```sh
systemctl list-timers librenotes-backup.timer
```

### Restore procedure

A restore is a four-step process. Run it on a *spare* directory
(e.g. `/srv/librenotes-restore-YYYYMMDD/`) — never overwrite the
live `/srv/librenotes/data` or `/srv/librenotes/state` until the
parallel stack passes verification.

```sh
# 1. Pick an archive (local or pulled back from the off-host target).
ARCHIVE=/var/backups/librenotes/librenotes-20260415-031712.tar.gz

# 2. Lay out a parallel stack dir.
RESTORE_DIR=/srv/librenotes-restore
mkdir -p "$RESTORE_DIR"/{data,state}
chown -R 65532:65532 "$RESTORE_DIR"/{data,state}    # distroless nonroot uid

# 3. Unpack the archive into a scratch dir, then move the pieces
#    into place.
SCRATCH=$(mktemp -d)
tar -C "$SCRATCH" -xzf "$ARCHIVE"
install -m 0644 -o 65532 -g 65532 "$SCRATCH/librenotes.db" \
        "$RESTORE_DIR/state/librenotes.db"
tar -C "$RESTORE_DIR/data" -xzf "$SCRATCH/notes.tar.gz"
chown -R 65532:65532 "$RESTORE_DIR/data"
rm -rf "$SCRATCH"

# 4. Bring up a parallel compose stack (dedicated docker compose
#    project name + non-conflicting LIBRENOTES_BASE_URL) and verify.
cd "$RESTORE_DIR"
cp /srv/librenotes/.env .env
perl -i -pe 's|^LIBRENOTES_BASE_URL=.*|LIBRENOTES_BASE_URL=http://localhost:18080|' .env
cp /srv/librenotes/{compose.yaml,compose.netcup.yaml} .
# Override port + project name so the parallel stack does not
# clash with the live one.
docker compose -p librenotes-restore \
  -f compose.yaml \
  --profile=restore \
  up -d \
  -e LIBRENOTES_PORT=18080
curl -fsS http://localhost:18080/healthz
# Sign in with a known test user, click around, confirm note state.
docker compose -p librenotes-restore down
```

Once the parallel stack proves the archive is intact, swap into
production with a maintenance window:

```sh
docker compose -f compose.yaml -f compose.netcup.yaml down
mv /srv/librenotes/data /srv/librenotes/data.bak-$(date +%s)
mv /srv/librenotes/state /srv/librenotes/state.bak-$(date +%s)
mv /srv/librenotes-restore/data /srv/librenotes/data
mv /srv/librenotes-restore/state /srv/librenotes/state
docker compose -f compose.yaml -f compose.netcup.yaml up -d
curl -fsS https://ln.cloud.librete.ch/healthz
```

If anything goes wrong, the `.bak-*` directories are still there
to restore from.