Back up the store and the certificate authority, and alert when it stops
Release / build (push) Successful in 1m1s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m32s

There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.

`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.

llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 01:44:32 +02:00
1 parent b661bfead8
commit 1da4a2e1ef
12 files changed
+549 -18

No files matched your search

+56 -18
View File
@@ -961,36 +961,74 @@ the choice is yours rather than silent.
## Backing up
Three things, in descending order of how badly you want them.
`llmgw server backup` writes one archive holding everything a control plane
cannot be rebuilt without:
**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
Losing it does not stop the cluster, and it means every node identity has to be
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
history, models added at runtime, measured capacity, and the secret every
engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
gateway);
- **`/etc/llmgw`** — the configuration, the store's address, and the
certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
key; losing it means reissuing every node's identity.
**The store.** Users, API keys, organisations, teams, projects, every limit and
budget, all usage history, models added at runtime, and measured capacity.
Archives go to `/var/backups/llmgw`, readable by root only because of the CA
key, and the newest 14 are kept. Run it nightly with the timer:
```
# SQLite
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
# PostgreSQL
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service # one now, to see it work
ls -l /var/backups/llmgw
```
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
SQLite file with `cp` while it is being written is not.
The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.
**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
readable by root only.
**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued — which is why the store is worth backing up on a schedule rather than
before upgrades.
reissued.
### Restoring
On a machine with llmgw installed (the same version or newer) and, for
PostgreSQL, an empty database and role as `server init --postgres` would create
them:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
# the store — PostgreSQL
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
# the store — SQLite
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
sudo systemctl start llmgw-server
sudo rm -rf /tmp/restore
```
Running engines keep serving throughout: agents reconnect to the restored
control plane and their replicas are adopted, and because the engine-key secret
came back with the store, the router can still reach them.
**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. Worth remembering that a pinned version may no longer be upstream to
download: that is exactly what pinning it protected you from, and a backup is
what makes the protection outlive the disk.
all again. A pinned version may no longer be upstream to download: that is
exactly what pinning it protected you from, and a copy of the directory is what
makes the protection outlive the disk.
## Reclaiming disk
+9
View File
@@ -169,6 +169,7 @@ rather than repeated.
| `monitoring` | `Monitoring` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
@@ -1265,6 +1266,14 @@ never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `backup_dir`
`string`
BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw.
### `router`
`Router`