One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NVML is the only source of NVLink topology, and that topology is what keeps a
tensor-parallel group off the slow path inside a node. It needs cgo, so it sits
behind the nvml tag and the default build keeps using nvidia-smi.
Discover() picks the backend and falls back to nvidia-smi when NVML fails,
logging the fallback rather than performing it silently: on a node that is meant
to have working NVML, a quiet downgrade hides a broken driver installation and
costs the scheduler its topology.
CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML
was declared only in the untagged file, so the tests did not compile under
-tags nvml. It is now declared where both builds can see it.
Not verified: the NVML path has never run against a driver, because there is no
GPU machine available. The NVLink domains it reports should be treated as
unverified until someone runs `llmgw agent inventory` on a real node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>