server restore: an archive checked, dry-run, and put back — configuration with modes and owners, the store in SQLite or PostgreSQL; backup_copy_to copies archives off the machine
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d29f694aaf
commit
10466416df
11 files changed
+889
-25
No files matched your search
+47
-19
@@ -1529,35 +1529,63 @@ anywhere else.** Restoring an old store restores which keys exist and who owns
|
||||
them, not their secrets. A key created after the backup is gone and has to be
|
||||
reissued.
|
||||
|
||||
### Copying archives off the machine
|
||||
|
||||
```yaml
|
||||
backup_recipients: [age1…] # required: an archive leaves only encrypted
|
||||
backup_copy_to:
|
||||
- /mnt/backups/llmgw # a mounted share or another disk
|
||||
- ssh://backup@10.0.0.20/srv/llmgw-backups
|
||||
backup_copy_key: /etc/llmgw/secrets/backup-ssh-key
|
||||
```
|
||||
|
||||
Each archive is copied there as soon as it is written, and the same number is
|
||||
kept there as here (`--keep`, 14). An ssh host's key is learnt the first time
|
||||
and checked every time after, in `/etc/llmgw/backup_known_hosts`. A copy that
|
||||
fails makes the backup fail, so the timer's unit shows it.
|
||||
|
||||
### Restoring
|
||||
|
||||
On a machine with llmgw installed (the same version or newer) and, for
|
||||
PostgreSQL, an empty database and role as `server init --postgres` would create
|
||||
them:
|
||||
On a machine with llmgw installed — the same version as the archive's, or
|
||||
newer — and, for PostgreSQL, the database and role the store's address names:
|
||||
|
||||
```
|
||||
sudo systemctl stop llmgw-server
|
||||
mkdir /tmp/restore && sudo tar -xzf /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz -C /tmp/restore
|
||||
# an encrypted one: open it with the private key, from wherever it is kept
|
||||
age -d -i key.txt llmgw-backup-<host>-<time>.tar.gz.age | sudo tar -xz -C /tmp/restore
|
||||
cat /tmp/restore/MANIFEST # version and time it was taken
|
||||
|
||||
# /etc/llmgw: configuration, secrets, certificates and the CA
|
||||
sudo cp -a /tmp/restore/etc/llmgw/. /etc/llmgw/
|
||||
|
||||
# the store — PostgreSQL
|
||||
sudo -u postgres pg_restore --clean --if-exists --no-owner --role=llmgw -d llmgw /tmp/restore/store.pgdump
|
||||
# the store — SQLite
|
||||
sudo install -o llmgw -g llmgw -m 0600 /tmp/restore/store.sqlite /var/lib/llmgw/llmgw.db
|
||||
|
||||
sudo systemctl start llmgw-server
|
||||
sudo rm -rf /tmp/restore
|
||||
sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz --dry-run
|
||||
sudo llmgw server restore llmgw-backup-<host>-<time>.tar.gz.age --identity key.txt
|
||||
```
|
||||
|
||||
It checks before it touches anything — that the archive is whole and opens,
|
||||
that this llmgw is not older than the one that wrote it, where the store goes
|
||||
and that `pg_restore` is there for a PostgreSQL dump — and `--dry-run` stops
|
||||
there. Then it stops `llmgw-server`, moves `/etc/llmgw` aside to
|
||||
`/etc/llmgw.before-restore-<time>`, puts the archive's in its place with each
|
||||
file's mode and owner, restores the store (a SQLite file replaced, the old one
|
||||
kept beside it; PostgreSQL by `pg_restore --clean` in one transaction), starts
|
||||
the server and waits for it to answer. `--store` restores the store somewhere
|
||||
else than the archive's `server.yaml` says.
|
||||
|
||||
Running engines keep serving throughout: agents reconnect to the restored
|
||||
control plane and their replicas are adopted, and because the engine-key secret
|
||||
came back with the store, the router can still reach them.
|
||||
|
||||
### A disaster recovery drill
|
||||
|
||||
Worth doing once before it is needed, on a spare machine, and again after a
|
||||
major upgrade:
|
||||
|
||||
1. Install the same llmgw version on the spare machine (step 1).
|
||||
2. Copy the newest archive there from wherever `backup_copy_to` put it, and
|
||||
the private key that opens it.
|
||||
3. `sudo llmgw server restore <archive> --identity <key> --dry-run`, and read
|
||||
what it says. For PostgreSQL, create the database and role first, or point
|
||||
`--store` at a scratch database.
|
||||
4. Without `--dry-run`. Then `llmgw ctl status`, `llmgw keys list --all`, and
|
||||
a request with a key that existed before.
|
||||
5. Note how long it took: that is the number to plan an outage around.
|
||||
|
||||
The drill machine now holds the cluster's CA key and secrets: take it down
|
||||
afterwards, or keep it as the standby it has become.
|
||||
|
||||
**The mirror's directory** is the one thing you can rebuild, by downloading it
|
||||
all again. A pinned version may no longer be upstream to download: that is
|
||||
exactly what pinning it protected you from, and a copy of the directory is what
|
||||
|
||||
Reference in new issue
Block a user