Ilker ManapandClaude Opus 5.5
9bf8256577
The first backup on the live control plane was written, and the control plane reported none: /var/backups/llmgw already existed, made by an earlier move to PostgreSQL for its own dumps, owned by postgres and 0700. MkdirAll leaves an existing directory as it is, so the archive landed where the llmgw user could not list it, and llmgw_backup_last_success_timestamp_seconds said 0. Opening that directory up would have exposed the dumps already in it, so backups get their own, /var/backups/llmgw-server. And `llmgw server backup` now warns when the directory it wrote to is one the control plane cannot list, rather than leaving that to be found from an alert. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
systemd units
Two roles, two units. Copy the one you need to /etc/systemd/system/, put the
matching configuration in /etc/llmgw/, and systemctl enable --now it.
The control plane — llmgw-server.service
Runs unprivileged. It needs to read its own TLS key, so that file must belong to
the llmgw user and nothing else:
useradd --system --home-dir /var/lib/llmgw --shell /usr/sbin/nologin llmgw
chown llmgw:llmgw /etc/llmgw/server-key.pem
chmod 0400 /etc/llmgw/server-key.pem
StateDirectory=llmgw creates /var/lib/llmgw with the right owner, which is
where the store belongs. ProtectSystem=strict makes the rest of the filesystem
read-only, so anything else it writes to has to be named explicitly.
The agent — llmgw-agent.service
Runs as root, because it launches engines and needs device access.
KillMode=process is the line that matters and the reason this file exists
rather than being generated. Without it, systemd kills the whole control group
on stop — including the engines, which hold a loaded model and are serving
traffic. Engines are meant to outlive an agent restart: the router talks to them
directly, and the agent adopts them again when it comes back. Taking them down
with the agent turns a thirty-second upgrade into a full model reload on every
node.
systemd will log Found left-over process ... in control group while starting unit. Ignoring. on the next start. That is the mechanism working, not a fault.