The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second half of leadership. A standby that refuses to place but still holds
an agent's stream is worse than useless: commands travel down the stream the
agent opened, so the leader's placements fail with "no agent connected" for a
node that is plainly connected.
So a standby refuses the stream, before reading the opening message, and says
why. And an agent takes a list rather than an address, trying each in turn — a
standby refusing, a stopped control plane refusing the connection and a crashed
one never answering are the same thing from a node, and the answer to all three
is somewhere else.
`server` and `servers` are both accepted, with `server` first when both are
given: every agent configuration written before there could be a second control
plane names one address, and an operator adding a standby expects the one
already working to stay preferred. With a single address the behaviour is
exactly what it was, including the backoff — which is skipped only while there
is another address that has not just been tried, because the delay exists to
stop hammering one control plane and the next one has not been hammered.
Removed a check that asked whether one field was empty, which is how the first
run of these tests failed: the list was populated and `server` was not, and an
agent configured entirely correctly was refused by its own validation.
The tests run a real agent against a real service over h2c. The first attempt
used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error
that said nothing about why — which is exactly what an operator would have seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by answering "where can we see this live". `ctl weights` was offering a
file for deletion that I had deleted two days earlier, and the node's disk
figure was the one it had when its agent started. Discover() is called at
registration and nowhere else, so both were frozen.
Disk is a fact about a moment, not about a machine. Sampled once, the figure
is wrong on exactly the node worth watching — the one busy filling up — and it
stays wrong until that agent restarts. The heartbeat already refreshes GPU
memory for the same reason, with a comment about noticing a foreign process
"after registration"; disk belonged beside it and I did not put it there.
The stale list cut both ways. A model fetched after registration was invisible
until a restart, so placement could not prefer the node that had just spent ten
minutes downloading it, and the disk check could not see the space it used.
Storage goes in every heartbeat, being a few numbers per filesystem. The file
list goes only when a fingerprint of paths and sizes changes, with a flag
saying the list is meant — without that, an empty list would be ambiguous
between "this node now holds nothing" and "nothing to report", and the second
reading would empty a node's inventory five seconds after it registered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of the same problem: nothing stopped a model being placed on a node
that could not hold it, and nothing ever removed the weights of a model that
had been deleted from the catalog.
Placement now has a disk check, shaped exactly like the memory one it sits
beside, with one addition: it is skipped on a node that already holds the
weights. A node holding the file needs no room for it, and without that the
node best placed to run a model — because it already has it — is the one most
likely to be rejected, since holding the file is what used the space.
Reclaiming is the dangerous half, so the whole design is about what it will
not do. llmgw deletes only files it fetched itself, and that is a calculation
rather than a heuristic: the cache name is a hash of the source, so the set of
wanted files can be computed from the catalog exactly. Anything without that
name was put there by an operator — scp, a shared mount, a copy by hand — and
is none of this system's business however unreferenced it looks. A model that
took a week to copy must not be removable by a command someone ran to tidy up.
The node checks every path again before unlinking it, and that duplication is
the point. The control plane knows the catalog; the node knows its disk. A
control plane that is wrong, newer than the agent, or tampered with must not be
able to talk a node into deleting something, so the agent refuses anything
outside its own cache directory or without the cache name — including a path
that tries to traverse out of it.
Listing is the default and deleting takes --collect. The summary says "asked to
remove", because each node decides, and one that refuses a path it cannot prove
it fetched is this working rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit taught the reconciler to let go of shards an agent says
it is not running. It made no difference on the cluster, because the service
returned early when the agent's list was empty — which is precisely the case
that carries the news.
So the fix was dead code in production while its unit tests passed, and they
passed because they called the reconciler directly. The test added here goes
through the real stream instead, and asserts only that an agent running
nothing reaches the reconciler at all.
That is the lesson worth keeping: a test that calls the function under test
directly cannot tell you the caller never calls it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adoption handled "the agent has something we do not know about". The mirror
image was missing: "we think the agent has something it does not".
Both follow from the same rule — the fleet is authoritative about what is
running (plan §6.5) — and without the second half an engine that dies while
its agent survives leaves the control plane believing a replica is Ready
forever. Measured on the cluster: the engine was killed, the agent restarted
and registered with nothing running, and the replica stayed Ready pointing at
a dead port for two and a half minutes until I stopped watching. Traffic
survived, because the breaker took the endpoint out and retries covered it —
24 requests, none failed — so the only symptom was a cluster quietly running a
third short.
A registration follows a fresh connection, so anything dispatched over a
previous one is not running by definition; there is no window where a shard is
legitimately started but unreported.
The previous commit's disown rule was too broad, which this turned up: when a
known replica had a live shard it disowned everything the agent reported for
it, including the shards we already knew. A reconnecting agent would have been
told to stop the replica that was serving. It now disowns only what it does
not recognise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A node's identity was whatever the agent said it was. With mutual TLS the
control plane compares the client certificate's common name against the
claimed node id and refuses the connection when they disagree, so identity
becomes something a node proves rather than something it picks.
A public authority cannot supply these certificates: Let's Encrypt and its
peers stopped issuing the clientAuth usage, and Go verifies client chains
against it. Hence internal/tlsconf/ca.go, a small authority whose key lives
on one machine and whose root is trusted everywhere, and `llmgw ctl cert`
over it so issuing is reproducible instead of remembered openssl.
The server certificate's hosts matter more than they look: hostname
verification reads subject alternative names and ignores the common name
entirely, so a certificate without the address agents dial — bare IP
included — is rejected no matter what its subject says. There is a test for
exactly that, because it is the failure everyone hits once.
requireIdentity stays wired to false until the configuration lands, so this
commit changes no running behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not persisting replica state left one hole: restarting the control plane made it
forget every replica while the engines kept running. They held GPUs the
inventory believed were free, so the next placement failed to allocate, with an
error naming the new process rather than the old one.
The fix belongs on the agent, not in a database. A row can say "running" about a
process that died; the fleet cannot.
- the supervisor writes the full shard spec, not just a pid and command line. A
pid identifies a process; the spec is what lets it be described back to the
control plane and matched to a replica.
- Adopt takes over an engine a previous agent left running. It is not this
process's child, so it cannot be waited on: it is watched by polling and
stopped by signalling its group directly. Its earlier output is gone and the
log says so rather than presenting emptiness as silence.
- every Register reports what the agent is running, not only the first, because
a reconnect may be a control plane restart and has to recover the same way.
- the reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped; nothing will
route to it and it is holding cards.
- an engine the agent cannot identify is killed. A process holding GPUs that
nothing can describe is worse than no process, because placement hands those
GPUs out again.
Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they
do not have, and reported them stopped without killing anything.
TestAgentReconnects had been passing for no reason. httptest's
CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c
hijacks the connection and httptest stops tracking hijacked ones — so the agent
never reconnected, and the assertion on the connection map passed on a stale
entry. Tests now sever connections through their own listener and count opened
streams.
Plan §11 also records what multi-node still needs, since placement landing early
makes it look further along than it is: workers never become Ready because the
agent health-checks every shard while only rank 0 serves HTTP; RayCommand is
written and tested but never called; and startup is fire-and-forget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>