Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.
Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.
It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.
Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.
ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.
Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.
Three things had to give way for it.
A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.
Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.
The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.
Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.
The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>