CI's test job has failed intermittently — two of three runs on some days —
while every run here passed, including six repetitions on one CPU under the
race detector with eight packages at once. The difference is a shared,
loaded Linux runner, and the tests most exposed to it are the ones that
sleep a fixed time and then assert:
- admission: idle admission bounded at 50ms of wall clock against a 1ms
tick, and a queued request checked for after a 50ms sleep;
- scrape: a 10ms scrape timeout, which on a slow runner reports the engine
unreachable and then the violation a second time, in a test that counts
one;
- supervisor: log lines, three lines of output and a removed state file
each checked after a fixed sleep, and an exited worker bounded at 1.5s;
- agent: a heartbeat checked for after four intervals.
Each now waits for its condition up to a deadline of seconds, and the two
wall-clock bounds are set against what they exist to distinguish — the
admission queue's tick raised to ten seconds, the exited worker bounded
below the five-second grace it must not wait out — rather than against how
fast this laptop is.
Without the CI logs this cannot be proven to be the failure; these are the
tests that fail when the machine is slow, and they no longer do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.
The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.
Not yet run over real InfiniBand or RoCE; this cluster has neither.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane asks one node on each physical fabric to run
ibnetdiscover every five minutes and parses the text itself, so a
format the parser has not met is a control plane upgrade, not one on
every node. From the map it works out the switch tiers (leaf, spine,
core, by distance from the adapters), which adapter is which node's,
which cables trained below the rate the rest of their kind run at, and
whether the rails line up: every node's Nth adapter on its own leaf.
A page per fabric (/admin/fabrics) draws it: switches in their tiers,
adapters fanned out under their leaves, cables coloured by rate and
bundled, degraded ones marching red with a marker, and the cluster's
own adapters live - down ports glowing red, and particles running each
cable as fast as its adapter is sending and receiving, from the
counters. It zooms, pans, fades what a hovered switch or adapter does
not touch, opens a panel of detail on a click, colours by rail, finds a
node, and refreshes every fifteen seconds. The data is also at
/api/v1/fabrics/{id}/topology.
Placement reads the map: a replica spanning nodes takes nodes of one
leaf together when it can, and nodes whose cable trained short last.
Static assets are named by their content, as the stylesheet already
was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A fabric's identity was its master subnet manager, which changes on a
failover: for the minute its ports took to agree, one fabric read as
two, and a name covering it as two separate fabrics. Each agent now
also lists every subnet manager on a fabric (saquery SMIR, through one
port per master, once a minute); managers listed together are one
fabric, named by the lowest GUID among them, so the identity survives
the master changing.
The same list says what a fabric can survive. The cluster page's new
subnet manager table, /api/v1/fabrics, doctor and metrics
(llmgw_fabric_master_sm, llmgw_fabric_subnet_managers,
llmgw_fabric_problems) show each fabric's master, priority and
standbys, and say when a fabric has no standby, its ports disagree on
the master, ports are up with no subnet manager (INIT), or the master
changed in the last hour. Every change of master a port reports is
kept, and a fabric's last day of them is listed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The agent's defaults said eth0 before normalize looked for the interface
holding the management address, so it never looked: a node on ens3 or
bond0 had NCCL and Gloo bootstrap over an interface it does not have. It
is now found, with eth0 only as the last resort.
The agent unit sets LimitMEMLOCK=infinity. RDMA pins what it registers,
and under systemd's 8 MB default an engine without the capability that
lets root ignore the limit fails its first collective over InfiniBand.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM's mp launch gave workers no rank and the leader's address as their
own VLLM_HOST_IP; every node now runs serve with --nnodes, --node-rank,
--master-addr and --master-port, and the workers --headless, as vLLM
documents it.
NCCL_SOCKET_IFNAME and GLOO_SOCKET_IFNAME were the control plane's one
interface name for every node; the agent already worked out its own -
the one holding its management address - and now uses it.
On RoCE, NCCL's default GID index is a v1 link-local entry on most hosts,
which does not route. The agent finds the RoCE v2 entry holding an IPv4
address and passes it as NCCL_IB_GID_INDEX; InfiniBand gets none.
A worker whose only process needs the leader - a Ray join - started
without waiting for it, since the wait was honoured for sidecars alone;
it found no head, exited and failed the replica. It now waits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.
The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.
A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.
An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.
Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.
And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.
A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default arrangement is the simple one: every node keeps its own copy.
So a source naming a remote file is now fetched before the engine is
launched, rather than left to fail as a missing path minutes into a load.
Eight shards of a tensor-parallel replica start at the same instant and
want the same file; they produce one download. An interrupted fetch resumes
instead of starting over, which matters at forty gigabytes, and the
download is written beside its target and renamed only when complete — a
partial file under the model's name would be found by the next scan,
reported as present, and handed to an engine that fails on it.
The exotic arrangements need no code, which is the point. Weights on a
Lustre mount or an image exported over the fabric are simply present, and
present means nothing is fetched. The escape hatch from a distribution
mechanism is a distribution mechanism that checks first.
Wiring it up caught a regression in my own design. A bare model id —
"Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang
resolve themselves, and treating every non-remote source as a local path
made every existing vLLM profile fail. Only an absolute or explicitly
relative path is now checked as a file; anything else is passed through
untouched. Three agent tests failed on the same change for a better reason:
they used "/models/x", and the new check is exactly the one that catches
that.
Whole repositories of shards are deliberately not fetched here. vLLM and
SGLang do that well already, and a second snapshot downloader would be work
spent arriving back where we started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit taught the reconciler to let go of shards an agent says
it is not running. It made no difference on the cluster, because the service
returned early when the agent's list was empty — which is precisely the case
that carries the news.
So the fix was dead code in production while its unit tests passed, and they
passed because they called the reconciler directly. The test added here goes
through the real stream instead, and asserts only that an agent running
nothing reaches the reconciler at all.
That is the lesson worth keeping: a test that calls the function under test
directly cannot tell you the caller never calls it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adoption handled "the agent has something we do not know about". The mirror
image was missing: "we think the agent has something it does not".
Both follow from the same rule — the fleet is authoritative about what is
running (plan §6.5) — and without the second half an engine that dies while
its agent survives leaves the control plane believing a replica is Ready
forever. Measured on the cluster: the engine was killed, the agent restarted
and registered with nothing running, and the replica stayed Ready pointing at
a dead port for two and a half minutes until I stopped watching. Traffic
survived, because the breaker took the endpoint out and retries covered it —
24 requests, none failed — so the only symptom was a cluster quietly running a
third short.
A registration follows a fresh connection, so anything dispatched over a
previous one is not running by definition; there is no window where a shard is
legitimately started but unreported.
The previous commit's disown rule was too broad, which this turned up: when a
known replica had a live shard it disowned everything the agent reported for
it, including the shards we already knew. A reconnecting agent would have been
told to stop the replica that was serving. It now disowns only what it does
not recognise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A node's identity was whatever the agent said it was. With mutual TLS the
control plane compares the client certificate's common name against the
claimed node id and refuses the connection when they disagree, so identity
becomes something a node proves rather than something it picks.
A public authority cannot supply these certificates: Let's Encrypt and its
peers stopped issuing the clientAuth usage, and Go verifies client chains
against it. Hence internal/tlsconf/ca.go, a small authority whose key lives
on one machine and whose root is trusted everywhere, and `llmgw ctl cert`
over it so issuing is reproducible instead of remembered openssl.
The server certificate's hosts matter more than they look: hostname
verification reads subject alternative names and ignores the common name
entirely, so a certificate without the address agents dial — bare IP
included — is rejected no matter what its subject says. There is a test for
exactly that, because it is the failure everyone hits once.
requireIdentity stays wired to false until the configuration lands, so this
commit changes no running behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not persisting replica state left one hole: restarting the control plane made it
forget every replica while the engines kept running. They held GPUs the
inventory believed were free, so the next placement failed to allocate, with an
error naming the new process rather than the old one.
The fix belongs on the agent, not in a database. A row can say "running" about a
process that died; the fleet cannot.
- the supervisor writes the full shard spec, not just a pid and command line. A
pid identifies a process; the spec is what lets it be described back to the
control plane and matched to a replica.
- Adopt takes over an engine a previous agent left running. It is not this
process's child, so it cannot be waited on: it is watched by polling and
stopped by signalling its group directly. Its earlier output is gone and the
log says so rather than presenting emptiness as silence.
- every Register reports what the agent is running, not only the first, because
a reconnect may be a control plane restart and has to recover the same way.
- the reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped; nothing will
route to it and it is holding cards.
- an engine the agent cannot identify is killed. A process holding GPUs that
nothing can describe is worse than no process, because placement hands those
GPUs out again.
Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they
do not have, and reported them stopped without killing anything.
TestAgentReconnects had been passing for no reason. httptest's
CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c
hijacks the connection and httptest stops tracking hijacked ones — so the agent
never reconnected, and the assertion on the connection map passed on a stale
entry. Tests now sever connections through their own listener and count opened
streams.
Plan §11 also records what multi-node still needs, since placement landing early
makes it look further along than it is: workers never become Ready because the
agent health-checks every shard while only rank 0 serves HTTP; RayCommand is
written and tested but never called; and startup is fire-and-forget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
