From 555e4678cbe4079954df0a5fabf67fa7bce8485d Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Wed, 16 Sep 2026 10:04:44 +0200 Subject: [PATCH] Ship the systemd units, and stop the plan lying about what is done Phase 3 had a block duplicated into itself, including an unchecked 'EndpointSource becomes dynamic' sitting directly under the checked line saying it was wired. Several phase 5 items had shipped without being ticked. The units were only ever on the machines and in a scratch directory, which makes them undocumented operational knowledge rather than part of the project. KillMode=process in particular is not obvious and not optional: without it systemd takes the engines down with the agent, turning a thirty-second upgrade into a model reload on every node. Co-Authored-By: Claude Opus 5 (1M context) --- deploy/systemd/README.md | 34 ++++++++++++++++++++++++++ deploy/systemd/llmgw-agent.service | 17 +++++++++++++ deploy/systemd/llmgw-server.service | 19 +++++++++++++++ llmgw-plan.md | 37 +++++++++++++++++------------ 4 files changed, 92 insertions(+), 15 deletions(-) create mode 100644 deploy/systemd/README.md create mode 100644 deploy/systemd/llmgw-agent.service create mode 100644 deploy/systemd/llmgw-server.service diff --git a/deploy/systemd/README.md b/deploy/systemd/README.md new file mode 100644 index 00000000..a1ebc3a2 --- /dev/null +++ b/deploy/systemd/README.md @@ -0,0 +1,34 @@ +# systemd units + +Two roles, two units. Copy the one you need to `/etc/systemd/system/`, put the +matching configuration in `/etc/llmgw/`, and `systemctl enable --now` it. + +## The control plane — `llmgw-server.service` + +Runs unprivileged. It needs to read its own TLS key, so that file must belong to +the `llmgw` user and nothing else: + +``` +useradd --system --home-dir /var/lib/llmgw --shell /usr/sbin/nologin llmgw +chown llmgw:llmgw /etc/llmgw/server-key.pem +chmod 0400 /etc/llmgw/server-key.pem +``` + +`StateDirectory=llmgw` creates `/var/lib/llmgw` with the right owner, which is +where the store belongs. `ProtectSystem=strict` makes the rest of the filesystem +read-only, so anything else it writes to has to be named explicitly. + +## The agent — `llmgw-agent.service` + +Runs as root, because it launches engines and needs device access. + +`KillMode=process` is the line that matters and the reason this file exists +rather than being generated. Without it, systemd kills the whole control group +on stop — including the engines, which hold a loaded model and are serving +traffic. Engines are meant to outlive an agent restart: the router talks to them +directly, and the agent adopts them again when it comes back. Taking them down +with the agent turns a thirty-second upgrade into a full model reload on every +node. + +systemd will log `Found left-over process ... in control group while starting +unit. Ignoring.` on the next start. That is the mechanism working, not a fault. diff --git a/deploy/systemd/llmgw-agent.service b/deploy/systemd/llmgw-agent.service new file mode 100644 index 00000000..fd529570 --- /dev/null +++ b/deploy/systemd/llmgw-agent.service @@ -0,0 +1,17 @@ +[Unit] +Description=llmgw node agent +After=network-online.target +Wants=network-online.target + +[Service] +ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml +# Root, because the agent launches engines and needs device access. Engines +# outlive a restart of this unit and are adopted again on the next start, so +# KillMode must not take the whole process group with it. +KillMode=process +Restart=on-failure +RestartSec=2s +StateDirectory=llmgw-agent + +[Install] +WantedBy=multi-user.target diff --git a/deploy/systemd/llmgw-server.service b/deploy/systemd/llmgw-server.service new file mode 100644 index 00000000..d2adc6fc --- /dev/null +++ b/deploy/systemd/llmgw-server.service @@ -0,0 +1,19 @@ +[Unit] +Description=llmgw control plane +After=network-online.target +Wants=network-online.target + +[Service] +ExecStart=/usr/local/bin/llmgw server --config /etc/llmgw/server.yaml +User=llmgw +Group=llmgw +Restart=on-failure +RestartSec=2s +NoNewPrivileges=true +ProtectSystem=strict +ProtectHome=true +PrivateTmp=true +StateDirectory=llmgw + +[Install] +WantedBy=multi-user.target diff --git a/llmgw-plan.md b/llmgw-plan.md index 4f9e6efb..bce1a0b5 100644 --- a/llmgw-plan.md +++ b/llmgw-plan.md @@ -1349,12 +1349,6 @@ work_dir: /var/lib/llmgw-agent - [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff - [x] `EndpointSource` wired from the reconciler to the router - [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement -- [ ] `ctl logs`: needs a request/response pair on the agent stream, which the - protocol does not have. The failure path already carries an engine log - tail with every status report, so the common case is covered; what is - missing is reading a healthy shard's output on demand. -- [ ] The router's `EndpointSource` becomes dynamic -- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement - [ ] `ctl logs`: needs a request/response pair on the agent stream, which the protocol does not have. The failure path already carries an engine log tail with every status report, so the common case is covered; what is @@ -1362,9 +1356,13 @@ work_dir: /var/lib/llmgw-agent - **Exit criterion (met against a simulated agent fleet):** shutting down a node moves its replicas elsewhere while the router keeps serving. Measured in `test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is - rebuilt on a survivor and ~370 requests complete with zero failures. The real - agent path still needs the stream, so this is proof of the control logic, not - of the transport. + rebuilt on a survivor and ~370 requests complete with zero failures. +- **Since verified against real agents and engines**, which turned up what the + simulated fleet could not: replica ids were reused after a control plane + restart, so a restarted control plane collided with engines still running from + its previous life and every placement landed on a port an orphan was holding. + Ids now carry random bytes, adopted state wins over merely-placed state, and a + replica that never comes up is failed rather than left in `starting` forever. ### Phase 4 — Multi-node and fabrics (2 weeks) - [x] Fabric identity from the agent's explicit labels @@ -1401,9 +1399,10 @@ The launch path is now built too: same fabric and never spreads it across two. ### Phase 5 — Production features (3 weeks) -- [ ] **SQLite store and migrations.** It arrives here rather than in phase 3 - because everything that genuinely needs persisting — keys, usage, capacity - profiles — is a phase 5 feature. Writing it earlier would have meant +- [x] **SQLite store and migrations**, behind an interface, with per-dialect + migration directories so Postgres is a driver swap rather than a rewrite. + It arrived here rather than in phase 3 because everything that genuinely + needs persisting is a phase 5 feature; writing it earlier would have meant designing a schema against guesses (§6.5). - [x] Tenancy: org/team/project/key hierarchy, key storage and authentication, limit resolution (most restrictive along the chain), `llmgw keys` @@ -1427,9 +1426,14 @@ The launch path is now built too: - [ ] Routing profiles from configuration plus the `x-llmgw-profile` header - [ ] Approximate radix tree (§7.5 tier 1), with its thresholds and decision metrics - [ ] Fallback models and pool spillover -- [ ] `ctl bench` → `CapacityProfile` (§6.2.1), `ctl plan` +- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency + level meeting the SLO, plus the queue depth and KV usage observed there, + so `weight`, `target_waiting` and `kv_ceiling` stop being guesses +- [ ] `ctl plan --target-rps N`, which needs capacity profiles stored + rather than printed - [ ] OTel tracing -- [ ] systemd units, documentation +- [x] systemd units and `docs/onboarding.md` +- [ ] Reference documentation beyond the plan itself ### Phase 6 — v2 ideas - [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration @@ -1441,7 +1445,10 @@ The launch path is now built too: - [ ] Prefill/decode disaggregation (§12) - [ ] LoRA adapter routing (affinity to a replica that already has the adapter) - [ ] External provider support (as a fallback) -- [ ] Admin web UI +- [x] Read-only dashboard on the admin listener (nodes, replicas, models, + placement failures, usage) and a public landing page +- [ ] An admin UI that can change things, which needs roles this deliberately + does not have ---