Back up the store and the certificate authority, and alert when it stops
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m32s

There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 01:44:32 +02:00
1 parent b661bfead8
commit 1da4a2e1ef
12 files changed
+549 -18

No files matched your search

+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw
After=llmgw-server.service postgresql.service
[Service]
Type=oneshot
# Root, because the archive includes /etc/llmgw/secrets and the certificate
# authority's key, which the llmgw user deliberately cannot read all of.
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
Nice=10
IOSchedulingClass=idle
+12
View File
@@ -0,0 +1,12 @@
[Unit]
Description=Back up llmgw every night
[Timer]
OnCalendar=*-*-* 03:15:00
# A machine that was off at 03:15 backs up when it next starts, rather than
# waiting another day with a gap nobody notices until the alert.
Persistent=true
RandomizedDelaySec=10m
[Install]
WantedBy=timers.target