Ship the systemd units, and stop the plan lying about what is done

Phase 3 had a block duplicated into itself, including an unchecked
'EndpointSource becomes dynamic' sitting directly under the checked line
saying it was wired. Several phase 5 items had shipped without being ticked.

The units were only ever on the machines and in a scratch directory, which
makes them undocumented operational knowledge rather than part of the project.
KillMode=process in particular is not obvious and not optional: without it
systemd takes the engines down with the agent, turning a thirty-second upgrade
into a model reload on every node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 10:04:44 +02:00
1 parent 378383dd69
commit 555e4678cb
4 files changed
+92 -15

No files matched your search

+34
View File
@@ -0,0 +1,34 @@
# systemd units
Two roles, two units. Copy the one you need to `/etc/systemd/system/`, put the
matching configuration in `/etc/llmgw/`, and `systemctl enable --now` it.
## The control plane — `llmgw-server.service`
Runs unprivileged. It needs to read its own TLS key, so that file must belong to
the `llmgw` user and nothing else:
```
useradd --system --home-dir /var/lib/llmgw --shell /usr/sbin/nologin llmgw
chown llmgw:llmgw /etc/llmgw/server-key.pem
chmod 0400 /etc/llmgw/server-key.pem
```
`StateDirectory=llmgw` creates `/var/lib/llmgw` with the right owner, which is
where the store belongs. `ProtectSystem=strict` makes the rest of the filesystem
read-only, so anything else it writes to has to be named explicitly.
## The agent — `llmgw-agent.service`
Runs as root, because it launches engines and needs device access.
`KillMode=process` is the line that matters and the reason this file exists
rather than being generated. Without it, systemd kills the whole control group
on stop — including the engines, which hold a loaded model and are serving
traffic. Engines are meant to outlive an agent restart: the router talks to them
directly, and the agent adopts them again when it comes back. Taking them down
with the agent turns a thirty-second upgrade into a full model reload on every
node.
systemd will log `Found left-over process ... in control group while starting
unit. Ignoring.` on the next start. That is the mechanism working, not a fault.
+17
View File
@@ -0,0 +1,17 @@
[Unit]
Description=llmgw node agent
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
# Root, because the agent launches engines and needs device access. Engines
# outlive a restart of this unit and are adopted again on the next start, so
# KillMode must not take the whole process group with it.
KillMode=process
Restart=on-failure
RestartSec=2s
StateDirectory=llmgw-agent
[Install]
WantedBy=multi-user.target
+19
View File
@@ -0,0 +1,19 @@
[Unit]
Description=llmgw control plane
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/local/bin/llmgw server --config /etc/llmgw/server.yaml
User=llmgw
Group=llmgw
Restart=on-failure
RestartSec=2s
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
StateDirectory=llmgw
[Install]
WantedBy=multi-user.target
+22 -15
View File
@@ -1349,12 +1349,6 @@ work_dir: /var/lib/llmgw-agent
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
- [x] `EndpointSource` wired from the reconciler to the router
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
protocol does not have. The failure path already carries an engine log
tail with every status report, so the common case is covered; what is
missing is reading a healthy shard's output on demand.
- [ ] The router's `EndpointSource` becomes dynamic
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
protocol does not have. The failure path already carries an engine log
tail with every status report, so the common case is covered; what is
@@ -1362,9 +1356,13 @@ work_dir: /var/lib/llmgw-agent
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
moves its replicas elsewhere while the router keeps serving. Measured in
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
rebuilt on a survivor and ~370 requests complete with zero failures. The real
agent path still needs the stream, so this is proof of the control logic, not
of the transport.
rebuilt on a survivor and ~370 requests complete with zero failures.
- **Since verified against real agents and engines**, which turned up what the
simulated fleet could not: replica ids were reused after a control plane
restart, so a restarted control plane collided with engines still running from
its previous life and every placement landed on a port an orphan was holding.
Ids now carry random bytes, adopted state wins over merely-placed state, and a
replica that never comes up is failed rather than left in `starting` forever.
### Phase 4 — Multi-node and fabrics (2 weeks)
- [x] Fabric identity from the agent's explicit labels
@@ -1401,9 +1399,10 @@ The launch path is now built too:
same fabric and never spreads it across two.
### Phase 5 — Production features (3 weeks)
- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3
because everything that genuinely needs persisting — keys, usage, capacity
profiles — is a phase 5 feature. Writing it earlier would have meant
- [x] **SQLite store and migrations**, behind an interface, with per-dialect
migration directories so Postgres is a driver swap rather than a rewrite.
It arrived here rather than in phase 3 because everything that genuinely
needs persisting is a phase 5 feature; writing it earlier would have meant
designing a schema against guesses (§6.5).
- [x] Tenancy: org/team/project/key hierarchy, key storage and authentication,
limit resolution (most restrictive along the chain), `llmgw keys`
@@ -1427,9 +1426,14 @@ The launch path is now built too:
- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
- [ ] Approximate radix tree (§7.5 tier 1), with its thresholds and decision metrics
- [ ] Fallback models and pool spillover
- [ ] `ctl bench` → `CapacityProfile` (§6.2.1), `ctl plan`
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
level meeting the SLO, plus the queue depth and KV usage observed there,
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
rather than printed
- [ ] OTel tracing
- [ ] systemd units, documentation
- [x] systemd units and `docs/onboarding.md`
- [ ] Reference documentation beyond the plan itself
### Phase 6 — v2 ideas
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
@@ -1441,7 +1445,10 @@ The launch path is now built too:
- [ ] Prefill/decode disaggregation (§12)
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
- [ ] External provider support (as a fallback)
- [ ] Admin web UI
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
placement failures, usage) and a public landing page
- [ ] An admin UI that can change things, which needs roles this deliberately
does not have
---