75a8e76706bbfa15f5dbe9e72dc5e648a02e8ef6
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
75a8e76706 |
Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s
Not persisting replica state left one hole: restarting the control plane made it forget every replica while the engines kept running. They held GPUs the inventory believed were free, so the next placement failed to allocate, with an error naming the new process rather than the old one. The fix belongs on the agent, not in a database. A row can say "running" about a process that died; the fleet cannot. - the supervisor writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica. - Adopt takes over an engine a previous agent left running. It is not this process's child, so it cannot be waited on: it is watched by polling and stopped by signalling its group directly. Its earlier output is gone and the log says so rather than presenting emptiness as silence. - every Register reports what the agent is running, not only the first, because a reconnect may be a control plane restart and has to recover the same way. - the reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped; nothing will route to it and it is holding cards. - an engine the agent cannot identify is killed. A process holding GPUs that nothing can describe is worse than no process, because placement hands those GPUs out again. Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they do not have, and reported them stopped without killing anything. TestAgentReconnects had been passing for no reason. httptest's CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c hijacks the connection and httptest stops tracking hijacked ones — so the agent never reconnected, and the assertion on the connection map passed on a stale entry. Tests now sever connections through their own listener and count opened streams. Plan §11 also records what multi-node still needs, since placement landing early makes it look further along than it is: workers never become Ready because the agent health-checks every shard while only rank 0 serves HTTP; RayCommand is written and tested but never called; and startup is fire-and-forget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
2373726efd |
Agent stream: connect the fleet to the control plane
CI / test (push) Failing after 1m29s
One bidirectional connect-go stream per node, carrying registration, heartbeats and shard status up, and start, stop and drain down. The agent dials out: GPU nodes sit behind firewalls far more often than a control plane does, and an outbound connection needs no inbound rule on every node in the fleet. - registration carries the whole hardware inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it; they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement. - agentsvc is both the stream handler and the reconciler's dispatcher, because it holds the open streams and is therefore the only thing that can deliver a command to a node. - the agent validates a profile before launching anything, since an engine that refuses a profile does so after minutes of loading, then supervises the process and reports every transition with the engine's output tail on failure. - reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock it over again. Engines keep serving throughout, because the router talks to them directly. llmgw agent is real now, and llmgw server runs a control plane whenever any model declares a profile. The static endpoint list stays for router-only runs. Verified with real binaries as separate processes: the agent registers, a replica is placed, an engine launches through the vLLM adapter's actual argument construction, streaming traffic is served, and killing the agent moves the node through unreachable to lost with placement explaining why nothing can run. config.Model and catalog.Model were the same thing described twice, so the router now reads catalog entries directly and the validation rules live in one place. Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted network until that lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
55417546fc |
Phase 2 (2/2): NVML backend behind a build tag
CI / test (push) Successful in 1m37s
NVML is the only source of NVLink topology, and that topology is what keeps a tensor-parallel group off the slow path inside a node. It needs cgo, so it sits behind the nvml tag and the default build keeps using nvidia-smi. Discover() picks the backend and falls back to nvidia-smi when NVML fails, logging the fallback rather than performing it silently: on a node that is meant to have working NVML, a quiet downgrade hides a broken driver installation and costs the scheduler its topology. CI now compiles the tagged build. That immediately paid for itself: ErrNoNVML was declared only in the untagged file, so the tests did not compile under -tags nvml. It is now declared where both builds can see it. Not verified: the NVML path has never run against a driver, because there is no GPU machine available. The NVLink domains it reports should be treated as unverified until someone runs `llmgw agent inventory` on a real node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6a13e3dddf |
Phase 2 (1/2): node discovery, engine adapters, process supervisor
CI / test (push) Successful in 45s
Everything here works without GPUs and is covered by tests. The control plane connection is not built yet, so llmgw agent itself is still a stub and the phase exit criterion is not met. - discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the cgo-free path, with health flags for uncorrected ECC and memory already held by another process. A node without IB is not an error, it is a single-node island. - engines: vLLM and SGLang command and environment construction, kept free of process handling so it stays testable. The NCCL environment is the part the plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters, because leaving it open lets NCCL pick an adapter on a fabric where the peer is unreachable, and the job then hangs instead of failing. - validation before launch: no mixed GPU types in a tensor-parallel group, the node/GPU/TP arithmetic has to add up, and fp8 is refused below compute capability 8.9 rather than silently falling back to a slower path. - supervisor: process groups, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader; a bounded log tail, since engine output is the only diagnosis when loading fails; Starting separate from Loading, because a 70B model legitimately takes minutes; and orphan adoption through pid files fingerprinted with the command line, since pids get reused and adopting an unrelated process is worse than leaking the original. - readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a loading model. - llmgw agent inventory reports what the scheduler would see on a node. One plan correction: GPU class sizes come from the driver in GiB, which does not always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g, not l40s-48g. Consistency matters more than matching the box, because that number decides whether a model fits. Plan §2 and §5.1 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |