Back up the store and the certificate authority, and alert when it stops
There were no backups: docs/install.md gave a pg_dump command, and nothing ran it. The store holds every key, person, quota and usage record, the models added at runtime, and now the secret each engine's key is derived from; the certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk lost all of it. `llmgw server backup` writes both to one archive: pg_dump's custom format, or a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's key, so it is 0600; the directory is listable, so the control plane — not root — can see when the newest was written. pg_dump is given the connection through its environment, since a DSN carries the password and a process's arguments are readable by every user. Written beside its name and renamed, so an interrupted backup is never the newest; the newest 14 are kept, and nothing the command did not write is ever removed. llmgw-backup.timer runs it nightly and catches up after downtime. The control plane publishes the newest archive's time, and BackupTooOld fires past 26 hours — or when there has never been one. The install guide now covers restoring, and says to copy archives off the machine, since they share its disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b661bfead8
commit
1da4a2e1ef
12 files changed
+549
-18
No files matched your search
@@ -0,0 +1,11 @@
|
||||
[Unit]
|
||||
Description=llmgw backup: the store and /etc/llmgw, to /var/backups/llmgw
|
||||
After=llmgw-server.service postgresql.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Root, because the archive includes /etc/llmgw/secrets and the certificate
|
||||
# authority's key, which the llmgw user deliberately cannot read all of.
|
||||
ExecStart=/usr/local/bin/llmgw server backup --config /etc/llmgw/server.yaml --keep 14
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Back up llmgw every night
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 03:15:00
|
||||
# A machine that was off at 03:15 backs up when it next starts, rather than
|
||||
# waiting another day with a gap nobody notices until the alert.
|
||||
Persistent=true
|
||||
RandomizedDelaySec=10m
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in new issue
Block a user