Ship the systemd units, and stop the plan lying about what is done
CI / test (push) Successful in 1m16s
CI / test (push) Successful in 1m16s
Phase 3 had a block duplicated into itself, including an unchecked 'EndpointSource becomes dynamic' sitting directly under the checked line saying it was wired. Several phase 5 items had shipped without being ticked. The units were only ever on the machines and in a scratch directory, which makes them undocumented operational knowledge rather than part of the project. KillMode=process in particular is not obvious and not optional: without it systemd takes the engines down with the agent, turning a thirty-second upgrade into a model reload on every node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
378383dd69
commit
555e4678cb
4 files changed
+92
-15
No files matched your search
@@ -0,0 +1,34 @@
|
||||
# systemd units
|
||||
|
||||
Two roles, two units. Copy the one you need to `/etc/systemd/system/`, put the
|
||||
matching configuration in `/etc/llmgw/`, and `systemctl enable --now` it.
|
||||
|
||||
## The control plane — `llmgw-server.service`
|
||||
|
||||
Runs unprivileged. It needs to read its own TLS key, so that file must belong to
|
||||
the `llmgw` user and nothing else:
|
||||
|
||||
```
|
||||
useradd --system --home-dir /var/lib/llmgw --shell /usr/sbin/nologin llmgw
|
||||
chown llmgw:llmgw /etc/llmgw/server-key.pem
|
||||
chmod 0400 /etc/llmgw/server-key.pem
|
||||
```
|
||||
|
||||
`StateDirectory=llmgw` creates `/var/lib/llmgw` with the right owner, which is
|
||||
where the store belongs. `ProtectSystem=strict` makes the rest of the filesystem
|
||||
read-only, so anything else it writes to has to be named explicitly.
|
||||
|
||||
## The agent — `llmgw-agent.service`
|
||||
|
||||
Runs as root, because it launches engines and needs device access.
|
||||
|
||||
`KillMode=process` is the line that matters and the reason this file exists
|
||||
rather than being generated. Without it, systemd kills the whole control group
|
||||
on stop — including the engines, which hold a loaded model and are serving
|
||||
traffic. Engines are meant to outlive an agent restart: the router talks to them
|
||||
directly, and the agent adopts them again when it comes back. Taking them down
|
||||
with the agent turns a thirty-second upgrade into a full model reload on every
|
||||
node.
|
||||
|
||||
systemd will log `Found left-over process ... in control group while starting
|
||||
unit. Ignoring.` on the next start. That is the mechanism working, not a fault.
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=llmgw node agent
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
|
||||
# Root, because the agent launches engines and needs device access. Engines
|
||||
# outlive a restart of this unit and are adopted again on the next start, so
|
||||
# KillMode must not take the whole process group with it.
|
||||
KillMode=process
|
||||
Restart=on-failure
|
||||
RestartSec=2s
|
||||
StateDirectory=llmgw-agent
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=llmgw control plane
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/llmgw server --config /etc/llmgw/server.yaml
|
||||
User=llmgw
|
||||
Group=llmgw
|
||||
Restart=on-failure
|
||||
RestartSec=2s
|
||||
NoNewPrivileges=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
PrivateTmp=true
|
||||
StateDirectory=llmgw
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+22
-15
@@ -1349,12 +1349,6 @@ work_dir: /var/lib/llmgw-agent
|
||||
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
|
||||
- [x] `EndpointSource` wired from the reconciler to the router
|
||||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||||
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
|
||||
protocol does not have. The failure path already carries an engine log
|
||||
tail with every status report, so the common case is covered; what is
|
||||
missing is reading a healthy shard's output on demand.
|
||||
- [ ] The router's `EndpointSource` becomes dynamic
|
||||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||||
- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the
|
||||
protocol does not have. The failure path already carries an engine log
|
||||
tail with every status report, so the common case is covered; what is
|
||||
@@ -1362,9 +1356,13 @@ work_dir: /var/lib/llmgw-agent
|
||||
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
|
||||
moves its replicas elsewhere while the router keeps serving. Measured in
|
||||
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
|
||||
rebuilt on a survivor and ~370 requests complete with zero failures. The real
|
||||
agent path still needs the stream, so this is proof of the control logic, not
|
||||
of the transport.
|
||||
rebuilt on a survivor and ~370 requests complete with zero failures.
|
||||
- **Since verified against real agents and engines**, which turned up what the
|
||||
simulated fleet could not: replica ids were reused after a control plane
|
||||
restart, so a restarted control plane collided with engines still running from
|
||||
its previous life and every placement landed on a port an orphan was holding.
|
||||
Ids now carry random bytes, adopted state wins over merely-placed state, and a
|
||||
replica that never comes up is failed rather than left in `starting` forever.
|
||||
|
||||
### Phase 4 — Multi-node and fabrics (2 weeks)
|
||||
- [x] Fabric identity from the agent's explicit labels
|
||||
@@ -1401,9 +1399,10 @@ The launch path is now built too:
|
||||
same fabric and never spreads it across two.
|
||||
|
||||
### Phase 5 — Production features (3 weeks)
|
||||
- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3
|
||||
because everything that genuinely needs persisting — keys, usage, capacity
|
||||
profiles — is a phase 5 feature. Writing it earlier would have meant
|
||||
- [x] **SQLite store and migrations**, behind an interface, with per-dialect
|
||||
migration directories so Postgres is a driver swap rather than a rewrite.
|
||||
It arrived here rather than in phase 3 because everything that genuinely
|
||||
needs persisting is a phase 5 feature; writing it earlier would have meant
|
||||
designing a schema against guesses (§6.5).
|
||||
- [x] Tenancy: org/team/project/key hierarchy, key storage and authentication,
|
||||
limit resolution (most restrictive along the chain), `llmgw keys`
|
||||
@@ -1427,9 +1426,14 @@ The launch path is now built too:
|
||||
- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||||
- [ ] Approximate radix tree (§7.5 tier 1), with its thresholds and decision metrics
|
||||
- [ ] Fallback models and pool spillover
|
||||
- [ ] `ctl bench` → `CapacityProfile` (§6.2.1), `ctl plan`
|
||||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||||
- [ ] `ctl plan <model> --target-rps N`, which needs capacity profiles stored
|
||||
rather than printed
|
||||
- [ ] OTel tracing
|
||||
- [ ] systemd units, documentation
|
||||
- [x] systemd units and `docs/onboarding.md`
|
||||
- [ ] Reference documentation beyond the plan itself
|
||||
|
||||
### Phase 6 — v2 ideas
|
||||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||||
@@ -1441,7 +1445,10 @@ The launch path is now built too:
|
||||
- [ ] Prefill/decode disaggregation (§12)
|
||||
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
|
||||
- [ ] External provider support (as a fallback)
|
||||
- [ ] Admin web UI
|
||||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||||
placement failures, usage) and a public landing page
|
||||
- [ ] An admin UI that can change things, which needs roles this deliberately
|
||||
does not have
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in new issue
Block a user