4 Commits

Include renames

Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	1da4a2e1ef	Back up the store and the certificate authority, and alert when it stops ... There were no backups: docs/install.md gave a pg_dump command, and nothing ran it. The store holds every key, person, quota and usage record, the models added at runtime, and now the secret each engine's key is derived from; the certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk lost all of it. `llmgw server backup` writes both to one archive: pg_dump's custom format, or a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's key, so it is 0600; the directory is listable, so the control plane — not root — can see when the newest was written. pg_dump is given the connection through its environment, since a DSN carries the password and a process's arguments are readable by every user. Written beside its name and renamed, so an interrupted backup is never the newest; the newest 14 are kept, and nothing the command did not write is ever removed. llmgw-backup.timer runs it nightly and catches up after downtime. The control plane publishes the newest archive's time, and BackupTooOld fires past 26 hours — or when there has never been one. The install guide now covers restoring, and says to copy archives off the machine, since they share its disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-30 01:44:32 +02:00
Ilker ManapandClaude Opus 5.5	4d9284ce9f	Find each node's management interface, and lift the agent's memlock limit ... The agent's defaults said eth0 before normalize looked for the interface holding the management address, so it never looked: a node on ens3 or bond0 had NCCL and Gloo bootstrap over an interface it does not have. It is now found, with eth0 only as the last resort. The agent unit sets LimitMEMLOCK=infinity. RDMA pins what it registers, and under systemd's 8 MB default an engine without the capability that lets root ignore the limit fails its first collective over InfiniBand. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-09-29 01:44:59 +02:00
Ilker ManapandClaude Opus 5	afb3797e57	Document the mirror that exists rather than the one you assemble ... The page described nginx and a curl loop, which was accurate when the mirror was a directory somebody kept by hand. It is a subcommand now, so it says so: the two listeners and why they are two, the API and what each endpoint answers, and the certificates coming from the authority the fleet already has. nginx stays, further down. It serves a directory perfectly well and anyone already running it should not be told to install something to replace it. The systemd unit spells out --base in a comment because it is the flag that goes wrong quietly: it ends up in catalog entries, and a wrong one makes `mirror unused` call files unreferenced while they are in use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-27 09:43:02 +02:00
Ilker ManapandClaude Opus 5	555e4678cb	Ship the systemd units, and stop the plan lying about what is done ... Phase 3 had a block duplicated into itself, including an unchecked 'EndpointSource becomes dynamic' sitting directly under the checked line saying it was wired. Several phase 5 items had shipped without being ticked. The units were only ever on the machines and in a scratch directory, which makes them undocumented operational knowledge rather than part of the project. KillMode=process in particular is not obvious and not optional: without it systemd takes the engines down with the agent, turning a thirty-second upgrade into a model reload on every node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-16 10:04:44 +02:00