The previous commit taught the reconciler to let go of shards an agent says
it is not running. It made no difference on the cluster, because the service
returned early when the agent's list was empty — which is precisely the case
that carries the news.
So the fix was dead code in production while its unit tests passed, and they
passed because they called the reconciler directly. The test added here goes
through the real stream instead, and asserts only that an agent running
nothing reaches the reconciler at all.
That is the lesson worth keeping: a test that calls the function under test
directly cannot tell you the caller never calls it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adoption handled "the agent has something we do not know about". The mirror
image was missing: "we think the agent has something it does not".
Both follow from the same rule — the fleet is authoritative about what is
running (plan §6.5) — and without the second half an engine that dies while
its agent survives leaves the control plane believing a replica is Ready
forever. Measured on the cluster: the engine was killed, the agent restarted
and registered with nothing running, and the replica stayed Ready pointing at
a dead port for two and a half minutes until I stopped watching. Traffic
survived, because the breaker took the endpoint out and retries covered it —
24 requests, none failed — so the only symptom was a cluster quietly running a
third short.
A registration follows a fresh connection, so anything dispatched over a
previous one is not running by definition; there is no window where a shard is
legitimately started but unreported.
The previous commit's disown rule was too broad, which this turned up: when a
known replica had a live shard it disowned everything the agent reported for
it, including the shards we already knew. A reconnecting agent would have been
told to stop the replica that was serving. It now disowns only what it does
not recognise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two tests for behaviour that looked wrong on the cluster and was not. Three
replicas all landed on one node, which is correct when it is the only node
registered: spreading is a preference and a model left unplaced to preserve a
preference would be worse. Over successive ticks with somewhere to go, they
spread.
What neither test asserts, because it is not true: nothing rebalances them
afterwards. A replica that moved on its own would interrupt the traffic it is
serving, and v1 has no autoscaler to decide it is worth it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit's adoption fix made a deeper problem visible instead of
fixing it. Replica ids were the model name and a counter, and the counter
restarted with the process — so a control plane that came back placed a
replica called llama-1 while an engine started before the restart, still
running on a different node, also called itself llama-1.
Two nodes then claimed the same shard id. Adoption handed it back and forth
between them, every placement landed on a port an orphan was still bound to,
and the engine failed to bind, which failed the replica, which placed another
one on the same port. On the real cluster that loop ran for minutes:
couldn't bind HTTP server socket, hostname: 0.0.0.0, port: 20002
Ids now carry four random bytes, which costs nothing and makes the collision
impossible. The "already known" branch in adoption goes back to meaning what
it says: an agent reconnecting, not a name clash across restarts.
Ports a stray still holds are also reserved now. Stopping is not instant and
an engine keeps its socket until it exits, so the allocator has to know about
sockets no replica claims — otherwise the same loop reappears in the window
between telling an agent to stop something and it actually stopping.
Two of the tests written for the previous commit had quietly become vacuous
under the new ids, asserting things about a literal llama-1 that no longer
exists. They capture the id now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl bench` measures what a replica can actually serve, so weight,
target_waiting and kv_ceiling stop being numbers somebody typed. Concurrency
climbs until the SLO breaks and the last level that met it is the capacity:
peak throughput arrives long after latency has become unusable, so throughput
alone overstates what the hardware can serve. Queue depth and KV usage are
read from the engine while it is still meeting the target, which is exactly
what admission should treat as healthy.
Prompts are unique per request by default. With one repeated prompt the
engine's cache answers most of the prefill and the measurement describes the
cache rather than the hardware; --shared-prefix asks that question
deliberately, and it now produces byte-identical prompts, which the first
version did not — it dropped the random header and still picked words at
random, so the cache missed and the flag quietly measured the thing it exists
to avoid.
Running it against the cluster turned up two real defects:
A replica stuck in Starting never timed out. Every other way out of that state
needs the agent to report something, and when it never does the replica sits
there holding GPUs and counting towards the model's replica count while
serving nothing. Two of them had been like that for eight minutes past a
two-minute startup timeout, and nothing would ever have moved them.
They were stuck because adoption lost to a replica we had only placed. Replica
ids come from the model name and a counter, so a control plane that restarts
and places before its agents reconnect builds a different replica under the
same id on a freshly allocated port; the agent's report of the engine that was
actually serving was then discarded as already-known. The cluster pointed at
port 20004 while the engine answered on 20000. The fleet is authoritative
about what is running, so what we merely placed now loses to what an agent
reports — unless our copy has a shard that has actually come up, in which case
two things really are running and the stray is the one we did not ask for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.
Three things had to give way for it.
A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.
Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.
The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A node's identity was whatever the agent said it was. With mutual TLS the
control plane compares the client certificate's common name against the
claimed node id and refuses the connection when they disagree, so identity
becomes something a node proves rather than something it picks.
A public authority cannot supply these certificates: Let's Encrypt and its
peers stopped issuing the clientAuth usage, and Go verifies client chains
against it. Hence internal/tlsconf/ca.go, a small authority whose key lives
on one machine and whose root is trusted everywhere, and `llmgw ctl cert`
over it so issuing is reproducible instead of remembered openssl.
The server certificate's hosts matter more than they look: hostname
verification reads subject alternative names and ignores the common name
entirely, so a certificate without the address agents dial — bare IP
included — is rejected no matter what its subject says. There is a test for
exactly that, because it is the failure everyone hits once.
requireIdentity stays wired to false until the configuration lands, so this
commit changes no running behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.
- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
answer when a node looks idle but nothing will schedule onto it, and it was
invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
the difference between cordon and drain is exactly what an operator needs to
be sure of before running either.
Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.
Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.
ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Placement and environment generation had landed early, which made multi-node
look further along than it was. A replica spanning nodes could not come up at
all, and all three reasons came down to assumptions that only hold on one node.
Not every rank serves HTTP. The agent health-checked every shard at its own
port, but only the leader answers; workers timed out, reported Failed and took
the replica with them. Engine.ServesHTTP now says which ranks serve, and a
worker is judged by staying alive. That is deliberately weaker evidence: the
replica's real readiness is the leader answering, which only happens once every
worker has joined.
A shard is not always one process. Engine.Commands returns a sequence — every
command but the last is a sidecar kept alive for the shard's lifetime and torn
down with it, and the last one's exit is the shard's. vLLM under Ray is why: the
leader runs a Ray head and then the server against it, while a worker runs only
the join and no vLLM process at all. RayCommand had been written and tested but
never called, so launch_mode: ray would have run vllm serve against a cluster
that did not exist. Sidecars are killed with the shard, because a Ray head left
behind holds the port the next replica needs.
Start order was backwards. Workers went first, on the theory that the head
should wait for them. Both engines put the rendezvous point on rank 0, so a
worker started first has nothing to connect to. vLLM's server blocks until the
Ray cluster has the GPUs it needs, so the leader can start immediately and waits
on its own. A command can now also require that rendezvous point to be listening
before the next starts, bounded by a timeout so a head that never appears fails
the shard instead of hanging it.
Stop was also killing only the engine, leaving sidecars behind.
None of this is verified against real engines — it is tested with stub
processes. The Ray sequencing, and which launch mode a given vLLM version wants,
can only be confirmed on hardware.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Not persisting replica state left one hole: restarting the control plane made it
forget every replica while the engines kept running. They held GPUs the
inventory believed were free, so the next placement failed to allocate, with an
error naming the new process rather than the old one.
The fix belongs on the agent, not in a database. A row can say "running" about a
process that died; the fleet cannot.
- the supervisor writes the full shard spec, not just a pid and command line. A
pid identifies a process; the spec is what lets it be described back to the
control plane and matched to a replica.
- Adopt takes over an engine a previous agent left running. It is not this
process's child, so it cannot be waited on: it is watched by polling and
stopped by signalling its group directly. Its earlier output is gone and the
log says so rather than presenting emptiness as silence.
- every Register reports what the agent is running, not only the first, because
a reconnect may be a control plane restart and has to recover the same way.
- the reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped; nothing will
route to it and it is holding cards.
- an engine the agent cannot identify is killed. A process holding GPUs that
nothing can describe is worse than no process, because placement hands those
GPUs out again.
Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they
do not have, and reported them stopped without killing anything.
TestAgentReconnects had been passing for no reason. httptest's
CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c
hijacks the connection and httptest stops tracking hijacked ones — so the agent
never reconnected, and the assertion on the connection map passed on a stale
entry. Tests now sever connections through their own listener and count opened
streams.
Plan §11 also records what multi-node still needs, since placement landing early
makes it look further along than it is: workers never become Ready because the
agent health-checks every shard while only rank 0 serves HTTP; RayCommand is
written and tested but never called; and startup is fire-and-forget.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.
Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.
The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.
- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
fallbacks resolve at load rather than at request time, because a typo in a
fallback should stop a deployment instead of surfacing during the outage when
the fallback is finally needed. The node/GPU/TP arithmetic is checked here
too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
already allocated and several from one snapshot would be handed the same GPUs.
Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
that will never complete while holding GPUs, so every shard is stopped and the
replica rebuilt, with a jittered exponential backoff carried on the model
rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
network blip must not tear down replicas that are still serving, and
rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.
The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.
Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two hard constraints from plan §1.1 are enforced structurally rather than
checked after the fact: there is no code path that can combine nodes from two
fabrics, and only GPUs of the profile's class are ever considered. Getting
either wrong produces a replica that starts, passes health checks, and then
hangs or crawls under load — which looks like success until traffic arrives.
- inventory: nodes, hardware, fabric attachments, allocations. A re-registering
agent cannot clear an operator's cordon, since a cordon is usually in place
because something is wrong with the node. A GPU with an ECC fault or another
process on it is not free in any useful sense. Replicas are released whole,
because they fail whole.
- placement: highest-priority profile that fits, with every rejected candidate
explained. "Unschedulable" on its own tells an operator nothing about which
of six constraints bit.
- bin packing fills partly used nodes first so whole nodes stay free for large
models that cannot be split; multi-node replicas get the emptiest machines
instead, being the least movable thing in the cluster.
- NVLink-aware GPU selection keeps a tensor-parallel group inside one domain
where the topology is known, falling back to lowest free indices otherwise.
Property-based tests generate 400 random clusters and assert the invariants on
every placement produced: one fabric per multi-node replica, only that fabric's
HCAs per shard, one GPU class, no unhealthy or already-allocated GPU. The
table-driven tests cover the cases someone thought of; these cover the rest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5