Two bugs found while verifying mTLS against real machines.
session cancelled its own derived context and then asked that context
whether the agent was shutting down. The answer was always yes, so every
stream error read as a clean close: the log said err=<nil>, the backoff
reset, and an agent whose certificate the control plane refused hammered it
forever at the base interval with nothing to diagnose. It now keeps the
parent context to answer that question, and a refusal reaches the operator
naming both the certificate and the claim.
Cancelling the context also did not stop the agent. HTTP/2 watches the
context while the request body is being written, and a bidirectional stream
spends its life blocked on the read side instead, so the read never returned.
Measured on a deployed node: `systemctl stop llmgw-agent` took 90 seconds and
ended in SIGKILL, on every restart. Closing the response when the context
ends is what actually unblocks it.
TestAgentStopsWhenContextEnds hangs reliably without the second fix, which is
the only reason to trust that it tests it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5