libretech
4ba7572565
CI / ci (pull_request) Failing after 5m52s
The deploy workflow is now a single job that builds, pushes, and deploys in one runner. Tag computation moved to docker/metadata-action, the per-deploy .env perl rewrite is gone (host pins LIBRENOTES_IMAGE once; main pushes update :main rolling, releases pin to :vX.Y.Z by manual edit), and both jobs run in our bespoke runner image whose runner user already has socket access via group membership. ci.yml moves to the same image so go/make/node are all available without per-step apt installs. Drops compose.prod.yaml (unused, redundant with compose.netcup.yaml).
142 lines
4.8 KiB
Markdown
142 lines
4.8 KiB
Markdown
# Operations
|
|
|
|
This document covers deployment, backups, and rollback for the
|
|
librenot.es production environment. For developer / contributor
|
|
docs see the top-level [README](../README.md).
|
|
|
|
## Deployment
|
|
|
|
The repository ships a CI workflow at `.gitea/workflows/deploy.yml`
|
|
that builds and pushes a Docker image on every push to `main` and
|
|
on every `vX.Y.Z` tag, then SSHes to the deployment host and runs
|
|
`docker compose pull && up -d`.
|
|
|
|
The workflow is gated on the repository variable
|
|
`DEPLOY_ENABLED=true`. Deployment is opt-in; flipping the variable
|
|
disables the workflow without removing the file.
|
|
|
|
### Required secrets
|
|
|
|
The workflow expects these secrets and variables on the repo:
|
|
|
|
| Name | Type | Purpose |
|
|
| ---- | ---- | ------- |
|
|
| `REGISTRY` | secret | registry hostname, e.g. `git.librete.ch` |
|
|
| `REGISTRY_USER` | secret | robot account or PAT username with `package:write` |
|
|
| `REGISTRY_PASS` | secret | robot/PAT token |
|
|
| `DEPLOY_HOST` | secret | `user@host` SSH target, e.g. `root@netcup` |
|
|
| `DEPLOY_KEY` | secret | passphrase-less private key (PEM) |
|
|
| `DEPLOY_PATH` | secret | absolute path on host with the compose stack, e.g. `/srv/librenotes` |
|
|
| `HEALTH_URL` | secret | e.g. `https://ln.cloud.librete.ch/healthz` |
|
|
| `DEPLOY_ENABLED` | variable | `true` to enable the workflow |
|
|
|
|
The image is pushed to `${REGISTRY}/public/librenotes`; main pushes
|
|
publish `:main` and `:<sha7>`, tag pushes publish `:vX.Y.Z` and
|
|
`:latest`.
|
|
|
|
### Production compose stack
|
|
|
|
On the deployment host, place `compose.yaml` and
|
|
`compose.netcup.yaml` from this repo at `$DEPLOY_PATH`, together
|
|
with an `.env` file containing `LIBRENOTES_IMAGE` plus the runtime
|
|
configuration (JWT secret, SMTP credentials, public base URL).
|
|
See `.env.netcup.example` for the full key list.
|
|
|
|
Bring it up with:
|
|
|
|
```sh
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Rollback
|
|
|
|
Tag pushes publish immutable tags (`:vX.Y.Z`). To pin or roll back,
|
|
edit `LIBRENOTES_IMAGE` in `/srv/librenotes/.env` and re-run pull +
|
|
up. The deployment workflow itself never rewrites the file — failed
|
|
health checks abort the workflow and the operator redeploys
|
|
manually.
|
|
|
|
```sh
|
|
# Example: roll back to v0.1.0
|
|
ssh netcup
|
|
cd /srv/librenotes
|
|
perl -i -pe 's|^LIBRENOTES_IMAGE=.*|LIBRENOTES_IMAGE=git.librete.ch/public/librenotes:v0.1.0|' .env
|
|
docker compose -f compose.yaml -f compose.netcup.yaml pull
|
|
docker compose -f compose.yaml -f compose.netcup.yaml up -d
|
|
```
|
|
|
|
### Gitea Actions runner
|
|
|
|
Workflows run on the netcup `act_runner` (see `runner/` stack in
|
|
the netcup umbrella). Both `ci.yml` and `deploy.yml` declare
|
|
`container: image: git.librete.ch/libretech/runner-image:v1` —
|
|
a bespoke Ubuntu 24.04 image (built and signed by us, hosted on
|
|
the same Gitea instance) that bundles `git`, `make`, `node`,
|
|
`perl`, `ssh`, and a docker CLI. The image's `runner` user is
|
|
pre-joined to `docker` group gid 998 so the auto-mounted
|
|
`/var/run/docker.sock` is writable without `--user root`.
|
|
|
|
The runner config (`runner/config.yaml`) whitelists
|
|
`/var/run/docker.sock` under `valid_volumes` to allow the
|
|
auto-mount.
|
|
|
|
## Backups
|
|
|
|
`scripts/backup.sh` is a self-contained backup driver suitable for
|
|
running from cron, a systemd timer, or the supplied
|
|
`scripts/librenotes-backup.{service,timer}` units.
|
|
|
|
Required tools on the backup host: `sqlite3`, `tar`, `gzip`, and
|
|
optionally `rclone` for off-site copy. On Debian/Ubuntu:
|
|
`apt install sqlite3 rclone`.
|
|
|
|
### What is backed up
|
|
|
|
- The SQLite database (`$LIBRENOTES_DB`) via the SQLite
|
|
`.backup` command, which produces a consistent online snapshot
|
|
without needing to stop the application.
|
|
- The per-tenant note directory (`$LIBRENOTES_DATA_DIR`) as a
|
|
gzipped tar.
|
|
- An `info.txt` in each archive recording the backup timestamp,
|
|
hostname, and version.
|
|
|
|
### Off-site copy
|
|
|
|
Set `BACKUP_REMOTE` to an `rclone` destination (e.g.
|
|
`s3:librenotes-backups`). When set, the script invokes
|
|
`rclone copy` to upload the archive after creation. Without it,
|
|
backups stay on the local disk only.
|
|
|
|
### Retention
|
|
|
|
`scripts/backup-prune.sh` enforces the policy "keep 30 daily
|
|
+ 12 monthly". Run it from the same timer as the backup. Files
|
|
keep the form `librenotes-YYYYMMDD-HHMMSS.tar.gz` so the prune
|
|
script can sort and select by name alone.
|
|
|
|
### Restore test
|
|
|
|
`scripts/backup-restore-test.sh` picks the most recent archive,
|
|
extracts it into a scratch directory, runs `sqlite3 .schema` on
|
|
the database to confirm it is readable, and verifies that the
|
|
note tar lists at least one entry. It is wired into a separate
|
|
weekly timer so a silent backup-corruption regression cannot hide
|
|
indefinitely.
|
|
|
|
### Systemd
|
|
|
|
Drop `scripts/librenotes-backup.{service,timer}` into
|
|
`/etc/systemd/system/`, then:
|
|
|
|
```sh
|
|
systemctl daemon-reload
|
|
systemctl enable --now librenotes-backup.timer
|
|
```
|
|
|
|
Confirm with:
|
|
|
|
```sh
|
|
systemctl list-timers librenotes-backup.timer
|
|
```
|