server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:09:59 +02:00
1 parent d29f694aaf
commit 10466416df
11 files changed
+889 -25

No files matched your search

+47 -19
View File
@@ -1529,35 +1529,63 @@ anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued.
### Copying archives off the machine
```yaml
backup_recipients: [age1…] # required: an archive leaves only encrypted
backup_copy_to:
- /mnt/backups/llmgw # a mounted share or another disk
- ssh://backup@10.0.0.20/srv/llmgw-backups
backup_copy_key: /etc/llmgw/secrets/backup-ssh-key
```
Each archive is copied there as soon as it is written, and the same number is
kept there as here (`--keep`, 14). An ssh host's key is learnt the first time
and checked every time after, in `/etc/llmgw/backup_known_hosts`. A copy that
fails makes the backup fail, so the timer's unit shows it.
### Restoring
On a machine with llmgw installed (the same version or newer) and, for
PostgreSQL, an empty database and role as `server init --postgres` would create
them:
On a machine with llmgw installed — the same version as the archive's, or
newer — and, for PostgreSQL, the database and role the store's address names:
```
sudo systemctl stop llmgw-server
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
# an encrypted one: open it with the private key, from wherever it is kept
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
cat /tmp/restore/MANIFEST # version and time it was taken
# /etc/llmgw: configuration, secrets, certificates and the CA
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
# the store — PostgreSQL
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
# the store — SQLite
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
sudo systemctl start llmgw-server
sudo rm -rf /tmp/restore
sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz --dry-run
sudo llmgw server restore llmgw-backup-<host>-<time>.tar.gz.age --identity key.txt
```
It checks before it touches anything — that the archive is whole and opens,
that this llmgw is not older than the one that wrote it, where the store goes
and that `pg_restore` is there for a PostgreSQL dump — and `--dry-run` stops
there. Then it stops `llmgw-server`, moves `/etc/llmgw` aside to
`/etc/llmgw.before-restore-<time>`, puts the archive's in its place with each
file's mode and owner, restores the store (a SQLite file replaced, the old one
kept beside it; PostgreSQL by `pg_restore --clean` in one transaction), starts
the server and waits for it to answer. `--store` restores the store somewhere
else than the archive's `server.yaml` says.
Running engines keep serving throughout: agents reconnect to the restored
control plane and their replicas are adopted, and because the engine-key secret
came back with the store, the router can still reach them.
### A disaster recovery drill
Worth doing once before it is needed, on a spare machine, and again after a
major upgrade:
1. Install the same llmgw version on the spare machine (step 1).
2. Copy the newest archive there from wherever `backup_copy_to` put it, and
the private key that opens it.
3. `sudo llmgw server restore <archive> --identity <key> --dry-run`, and read
what it says. For PostgreSQL, create the database and role first, or point
`--store` at a scratch database.
4. Without `--dry-run`. Then `llmgw ctl status`, `llmgw keys list --all`, and
a request with a key that existed before.
5. Note how long it took: that is the number to plan an outage around.
The drill machine now holds the cluster's CA key and secrets: take it down
afterwards, or keep it as the standby it has become.
**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. A pinned version may no longer be upstream to download: that is
exactly what pinning it protected you from, and a copy of the directory is what
+19
View File
@@ -245,6 +245,8 @@ rather than repeated.
| `notify` | `notify.Config` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `backup_copy_to` | `[]string` | [Server](#server) |
| `backup_copy_key` | `string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
@@ -2008,6 +2010,23 @@ the certificate authority's key and every secret in the store, and is
copied off the machine; encrypted, the copy is safe wherever it lands.
Empty writes archives unencrypted, readable by root only.
### `backup_copy_to`
`[]string`
BackupCopyTo is where each archive is copied once written, so it
survives this machine: a directory (a mounted share, another disk) or
ssh://user@host[:port]/path, reached with BackupCopyKey and checked
against /etc/llmgw/backup_known_hosts, which learns a host the first
time. The same number of archives is kept there as here. Refused
without backup_recipients: an archive leaves the machine encrypted.
### `backup_copy_key`
`string`
BackupCopyKey is the SSH private key for ssh:// copies.
### `releases`
`string`