libretech
c261a23da8
CI / ci (pull_request) Successful in 11m39s
operations.md lists the full secret/variable matrix the deploy workflow expects, the new compose.yaml + compose.netcup.yaml invocation, and a note on why both workflow jobs pin catthehacker/ubuntu:runner-latest. self-hosting.md updates the example LIBRENOTES_IMAGE value to the Gitea Packages path and switches the quick-start to compose.netcup.yaml.
140 lines
4.8 KiB
Markdown
140 lines
4.8 KiB
Markdown
# Operations
|
|
|
|
This document covers deployment, backups, and rollback for the
|
|
librenot.es production environment. For developer / contributor
|
|
docs see the top-level [README](../README.md).
|
|
|
|
## Deployment
|
|
|
|
The repository ships a CI workflow at `.gitea/workflows/deploy.yml`
|
|
that builds and pushes a Docker image on every push to `main` and
|
|
on every `vX.Y.Z` tag, then SSHes to the deployment host and runs
|
|
`docker compose pull && up -d`.
|
|
|
|
The workflow is gated on the repository variable
|
|
`DEPLOY_ENABLED=true`. Deployment is opt-in; flipping the variable
|
|
disables the workflow without removing the file.
|
|
|
|
### Required secrets
|
|
|
|
The workflow expects these secrets and variables on the repo:
|
|
|
|
| Name | Type | Purpose |
|
|
| ---- | ---- | ------- |
|
|
| `REGISTRY` | secret | registry hostname, e.g. `git.librete.ch` |
|
|
| `REGISTRY_USER` | secret | robot account or PAT username with `package:write` |
|
|
| `REGISTRY_PASS` | secret | robot/PAT token |
|
|
| `DEPLOY_HOST` | secret | `user@host` SSH target, e.g. `root@netcup` |
|
|
| `DEPLOY_KEY` | secret | passphrase-less private key (PEM) |
|
|
| `DEPLOY_PATH` | secret | absolute path on host with the compose stack, e.g. `/srv/librenotes` |
|
|
| `HEALTH_URL` | secret | e.g. `https://ln.cloud.librete.ch/healthz` |
|
|
| `DEPLOY_ENABLED` | variable | `true` to enable the workflow |
|
|
|
|
The image is pushed to `${REGISTRY}/public/librenotes`; main pushes
|
|
publish `:main` and `:<sha7>`, tag pushes publish `:vX.Y.Z` and
|
|
`:latest`.
|
|
|
|
### Production compose stack
|
|
|
|
On the deployment host, place `compose.yaml` and
|
|
`compose.netcup.yaml` from this repo at `$DEPLOY_PATH`, together
|
|
with an `.env` file containing `LIBRENOTES_IMAGE` plus the runtime
|
|
configuration (JWT secret, SMTP credentials, public base URL).
|
|
See `.env.netcup.example` for the full key list.
|
|
|
|
Bring it up with:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Rollback
|
|
|
|
Image tags are immutable per commit / version. To roll back, set
|
|
`LIBRENOTES_IMAGE` to the previous tag in `.env` and run the same
|
|
`up -d` command. The deployment workflow does not auto-rollback
|
|
on health-check failure — failed health alerts the operator via
|
|
the workflow itself, who can then redeploy the prior tag manually.
|
|
|
|
```sh
|
|
# Example: roll back to v0.1.0
|
|
perl -i -pe 's|^LIBRENOTES_IMAGE=.*|LIBRENOTES_IMAGE=git.librete.ch/public/librenotes:v0.1.0|' .env
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Gitea Actions runner
|
|
|
|
Workflows run on the netcup `act_runner` (see `runner/` stack in
|
|
the netcup umbrella). Both jobs declare `container: catthehacker/ubuntu:runner-latest`
|
|
because:
|
|
|
|
- The default runner label image (`node:20-bookworm`) lacks `make`
|
|
and `docker`, both required by the workflows.
|
|
- The `runner-latest` image bundles `make`, `git`, `curl`, `ssh`,
|
|
`node`, plus a docker CLI.
|
|
|
|
The runner config (`runner/config.yaml`) declares
|
|
`/var/run/docker.sock` as a `valid_volume` so the build job can
|
|
mount the host socket and push images via `docker/build-push-action`.
|
|
|
|
## Backups
|
|
|
|
`scripts/backup.sh` is a self-contained backup driver suitable for
|
|
running from cron, a systemd timer, or the supplied
|
|
`scripts/librenotes-backup.{service,timer}` units.
|
|
|
|
Required tools on the backup host: `sqlite3`, `tar`, `gzip`, and
|
|
optionally `rclone` for off-site copy. On Debian/Ubuntu:
|
|
`apt install sqlite3 rclone`.
|
|
|
|
### What is backed up
|
|
|
|
- The SQLite database (`$LIBRENOTES_DB`) via the SQLite
|
|
`.backup` command, which produces a consistent online snapshot
|
|
without needing to stop the application.
|
|
- The per-tenant note directory (`$LIBRENOTES_DATA_DIR`) as a
|
|
gzipped tar.
|
|
- An `info.txt` in each archive recording the backup timestamp,
|
|
hostname, and version.
|
|
|
|
### Off-site copy
|
|
|
|
Set `BACKUP_REMOTE` to an `rclone` destination (e.g.
|
|
`s3:librenotes-backups`). When set, the script invokes
|
|
`rclone copy` to upload the archive after creation. Without it,
|
|
backups stay on the local disk only.
|
|
|
|
### Retention
|
|
|
|
`scripts/backup-prune.sh` enforces the policy "keep 30 daily
|
|
+ 12 monthly". Run it from the same timer as the backup. Files
|
|
keep the form `librenotes-YYYYMMDD-HHMMSS.tar.gz` so the prune
|
|
script can sort and select by name alone.
|
|
|
|
### Restore test
|
|
|
|
`scripts/backup-restore-test.sh` picks the most recent archive,
|
|
extracts it into a scratch directory, runs `sqlite3 .schema` on
|
|
the database to confirm it is readable, and verifies that the
|
|
note tar lists at least one entry. It is wired into a separate
|
|
weekly timer so a silent backup-corruption regression cannot hide
|
|
indefinitely.
|
|
|
|
### Systemd
|
|
|
|
Drop `scripts/librenotes-backup.{service,timer}` into
|
|
`/etc/systemd/system/`, then:
|
|
|
|
```sh
|
|
systemctl daemon-reload
|
|
systemctl enable --now librenotes-backup.timer
|
|
```
|
|
|
|
Confirm with:
|
|
|
|
```sh
|
|
systemctl list-timers librenotes-backup.timer
|
|
```
|