Ilker ManapandClaude Opus 5.5
1da4a2e1ef
There were no backups: docs/install.md gave a pg_dump command, and nothing ran it. The store holds every key, person, quota and usage record, the models added at runtime, and now the secret each engine's key is derived from; the certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk lost all of it. `llmgw server backup` writes both to one archive: pg_dump's custom format, or a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's key, so it is 0600; the directory is listable, so the control plane — not root — can see when the newest was written. pg_dump is given the connection through its environment, since a DSN carries the password and a process's arguments are readable by every user. Written beside its name and renamed, so an interrupted backup is never the newest; the newest 14 are kept, and nothing the command did not write is ever removed. llmgw-backup.timer runs it nightly and catches up after downtime. The control plane publishes the newest archive's time, and BackupTooOld fires past 26 hours — or when there has never been one. The install guide now covers restoring, and says to copy archives off the machine, since they share its disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
systemd units
Two roles, two units. Copy the one you need to /etc/systemd/system/, put the
matching configuration in /etc/llmgw/, and systemctl enable --now it.
The control plane — llmgw-server.service
Runs unprivileged. It needs to read its own TLS key, so that file must belong to
the llmgw user and nothing else:
useradd --system --home-dir /var/lib/llmgw --shell /usr/sbin/nologin llmgw
chown llmgw:llmgw /etc/llmgw/server-key.pem
chmod 0400 /etc/llmgw/server-key.pem
StateDirectory=llmgw creates /var/lib/llmgw with the right owner, which is
where the store belongs. ProtectSystem=strict makes the rest of the filesystem
read-only, so anything else it writes to has to be named explicitly.
The agent — llmgw-agent.service
Runs as root, because it launches engines and needs device access.
KillMode=process is the line that matters and the reason this file exists
rather than being generated. Without it, systemd kills the whole control group
on stop — including the engines, which hold a loaded model and are serving
traffic. Engines are meant to outlive an agent restart: the router talks to them
directly, and the agent adopts them again when it comes back. Taking them down
with the agent turns a thirty-second upgrade into a full model reload on every
node.
systemd will log Found left-over process ... in control group while starting unit. Ignoring. on the next start. That is the mechanism working, not a fault.