diff --git a/CHANGELOG.md b/CHANGELOG.md index d81d80e..0da36a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,8 +66,22 @@ ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon. than its minimum, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. +### Backups + +`llmgw server backup` writes the store and `/etc/llmgw` — configuration, +secrets, and the certificate authority's key — to one archive readable by root +only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it +nightly. The control plane publishes the newest archive's time as +`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is +more than 26 hours old or there has never been one. docs/install.md now says how +to restore, including why running engines survive it: the engine-key secret +comes back with the store. + ### Fixes +- Tests that slept a fixed time and then asserted now wait for their + condition. They are the likeliest cause of CI's intermittent failures on a + loaded runner. - `ctl` reported a change the API had queued (202 Accepted) as an error, as `ctl restart` did on its first real use. diff --git a/deploy/monitoring/llmgw-rules.yml b/deploy/monitoring/llmgw-rules.yml index 128f11e..fc022b5 100644 --- a/deploy/monitoring/llmgw-rules.yml +++ b/deploy/monitoring/llmgw-rules.yml @@ -123,6 +123,17 @@ groups: summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}" description: See CertificateExpiresSoon; a week is not long. + - alert: BackupTooOld + expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600 + labels: { severity: warning } + annotations: + summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old" + description: >- + llmgw-backup.timer runs nightly. Also fires when there has never + been a backup (the series is 0 then). `journalctl -u + llmgw-backup.service` on the control plane says why the last one + failed; docs/install.md, "Backing up", says how to set it up. + - alert: PoolSaturated expr: llmgw_pool_saturation > 1 for: 5m diff --git a/deploy/systemd/llmgw-backup.service b/deploy/systemd/llmgw-backup.service new file mode 100644 index 0000000..268fbe5 --- /dev/null +++ b/deploy/systemd/llmgw-backup.service @@ -0,0 +1,11 @@ +[Unit] +Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw +After=llmgw-server.service postgresql.service + +[Service] +Type=oneshot +# Root, because the archive includes /etc/llmgw/secrets and the certificate +# authority's key, which the llmgw user deliberately cannot read all of. +ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14 +Nice=10 +IOSchedulingClass=idle diff --git a/deploy/systemd/llmgw-backup.timer b/deploy/systemd/llmgw-backup.timer new file mode 100644 index 0000000..7db9a3a --- /dev/null +++ b/deploy/systemd/llmgw-backup.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Back up llmgw every night + +[Timer] +OnCalendar=*-*-* 03:15:00 +# A machine that was off at 03:15 backs up when it next starts, rather than +# waiting another day with a gap nobody notices until the alert. +Persistent=true +RandomizedDelaySec=10m + +[Install] +WantedBy=timers.target diff --git a/docs/install.md b/docs/install.md index c6d7bc9..b282b0a 100644 --- a/docs/install.md +++ b/docs/install.md @@ -961,36 +961,74 @@ the choice is yours rather than silent. ## Backing up -Three things, in descending order of how badly you want them. +`llmgw server backup` writes one archive holding everything a control plane +cannot be rebuilt without: -**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2). -Losing it does not stop the cluster, and it means every node identity has to be -reissued the next time you touch one. Nothing in llmgw can reconstruct it. +- **the store** — users, API keys, the tenancy tree, limits and budgets, usage + history, models added at runtime, measured capacity, and the secret every + engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a + consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running + gateway); +- **`/etc/llmgw`** — the configuration, the store's address, and the + certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that + key; losing it means reissuing every node's identity. -**The store.** Users, API keys, organisations, teams, projects, every limit and -budget, all usage history, models added at runtime, and measured capacity. +Archives go to `/var/backups/llmgw`, readable by root only because of the CA +key, and the newest 14 are kept. Run it nightly with the timer: ``` -# SQLite -sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'" - -# PostgreSQL -pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump +sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now llmgw-backup.timer +sudo systemctl start llmgw-backup.service # one now, to see it work +ls -l /var/backups/llmgw ``` -`.backup` and `pg_dump` are both safe against a running gateway. Copying the -SQLite file with `cp` while it is being written is not. +The control plane publishes when the newest archive was written as +`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when +it is more than 26 hours old — including when there has never been one. + +**Copy the archives off the machine.** They sit on the disk they back up and do +not survive losing it. Anything that copies files will do — rsync from the +mirror, a NAS mount, an offsite bucket — as long as it keeps the archives +readable by root only. **Keys are stored hashed and cannot be recovered from a backup or from anywhere else.** Restoring an old store restores which keys exist and who owns them, not their secrets. A key created after the backup is gone and has to be -reissued — which is why the store is worth backing up on a schedule rather than -before upgrades. +reissued. + +### Restoring + +On a machine with llmgw installed (the same version or newer) and, for +PostgreSQL, an empty database and role as `server init --postgres` would create +them: + +``` +sudo systemctl stop llmgw-server +mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw/llmgw-backup--