There were no backups: docs/install.md gave a pg_dump command, and nothing
ran it. The store holds every key, person, quota and usage record, the models
added at runtime, and now the secret each engine's key is derived from; the
certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk
lost all of it.
`llmgw server backup` writes both to one archive: pg_dump's custom format, or
a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's
key, so it is 0600; the directory is listable, so the control plane — not
root — can see when the newest was written. pg_dump is given the connection
through its environment, since a DSN carries the password and a process's
arguments are readable by every user. Written beside its name and renamed,
so an interrupted backup is never the newest; the newest 14 are kept, and
nothing the command did not write is ever removed.
llmgw-backup.timer runs it nightly and catches up after downtime. The control
plane publishes the newest archive's time, and BackupTooOld fires past 26
hours — or when there has never been one. The install guide now covers
restoring, and says to copy archives off the machine, since they share its
disk.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The agent's defaults said eth0 before normalize looked for the interface
holding the management address, so it never looked: a node on ens3 or
bond0 had NCCL and Gloo bootstrap over an interface it does not have. It
is now found, with eth0 only as the last resort.
The agent unit sets LimitMEMLOCK=infinity. RDMA pins what it registers,
and under systemd's 8 MB default an engine without the capability that
lets root ignore the limit fails its first collective over InfiniBand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The page described nginx and a curl loop, which was accurate when the mirror
was a directory somebody kept by hand. It is a subcommand now, so it says so:
the two listeners and why they are two, the API and what each endpoint answers,
and the certificates coming from the authority the fleet already has.
nginx stays, further down. It serves a directory perfectly well and anyone
already running it should not be told to install something to replace it.
The systemd unit spells out --base in a comment because it is the flag that
goes wrong quietly: it ends up in catalog entries, and a wrong one makes
`mirror unused` call files unreferenced while they are in use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 3 had a block duplicated into itself, including an unchecked
'EndpointSource becomes dynamic' sitting directly under the checked line
saying it was wired. Several phase 5 items had shipped without being ticked.
The units were only ever on the machines and in a scratch directory, which
makes them undocumented operational knowledge rather than part of the project.
KillMode=process in particular is not obvious and not optional: without it
systemd takes the engines down with the agent, turning a thirty-second upgrade
into a model reload on every node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
