ea4dd37f80af44432cedc856b67b8f5fba577366
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9d8cd5e284 |
Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing, and they are the same thing seen from two distances. Idle tracking. The control plane knew what was placed and whether it was healthy, never whether anyone wanted it — so on a fleet with more models than cards there was no way to answer the first question anyone asks, which is what can I stop. The router is the only thing that knows, so it now reports it: a timestamp written in Acquire, where a request already meets an endpoint under a lock already held, and collected once a tick as a map. That shape is the point. Per-request notification would be a firehose carrying one bit — the newest timestamp wins — and this costs the same on four replicas as on four hundred. It travels through an interface in the direction opposite EndpointSink, for the same reason that one exists: the control plane decides what should run, the router knows whether anyone wants it, and neither should need to be in the other's process to say so. Unknown is kept distinct from idle throughout. A router that has just restarted knows nothing, and a zero timestamp would be the oldest idle replica in every comparison — so one restart would nominate an entire fleet for eviction. Absent from the map, "-" in the table, negative in the JSON, and IdleFor returns a second value rather than a duration anything could act on by mistake. ctl status. A per-node listing is right for four nodes and wrong for two hundred: how full am I, what is broken, what is holding cards nobody is using are aggregates, and reading them off a long table is a chore, which means it gets skipped. Capacity per GPU class with usage bars, free whole nodes per fabric — the number that decides whether a multi-node replica can land, since free cards on the wrong fabric are not capacity for a model that spans nodes — replica and model counts, models that cannot be placed, and what has gone unasked-for. Derived from the same state the tables show, so it cannot disagree with them. Unhealthy GPUs count as neither used nor free, because counting them free is how a cluster looks like it has room it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6f3bde099e |
Let a multi-node replica be adopted one agent at a time
CI / test (push) Successful in 1m20s
A replica spanning two nodes is reported by two agents, registering
separately about a second apart. Adoption assumed one call carried a whole
replica, so the first half was adopted with WantShards=2 and immediately
failed for "replica has 1 of 2 shards" — and when the second node
registered, its half was condemned as unclaimed because the replica was
already written off.
big-model therefore could not survive a control plane restart. Seen after
today's deploy: adopted, failed, both halves stopped, rebuilt from
scratch, backing off 9s, 20s, 38s, and never becoming ready. The first
reconcile tick also runs before any agent has reconnected, which is the
harmless-looking "no node reports a fabric" that precedes it.
Two changes. Adopt merges a group into a replica it is still assembling
instead of treating it as a competing claim, and takes the endpoint from
whichever group carries rank 0 — the leader's agent is not necessarily the
first to register. deriveState leaves a replica that is short of shards
while still assembling in Starting rather than Failed, which puts it under
the startup timeout that already exists: a sibling that never arrives
still fails, one second later than it used to rather than one second
before its sibling reports in. AdoptedAt is cleared once the replica is
whole, so losing a shard after that is a failure again.
The existing multi-node adoption test passed throughout, because it hands
both shards to a single Adopt call — which no caller makes. Same shape as
the agentsvc bug in
|
||
|
|
7b4acd9284 |
A replica missing a shard is not ready
CI / test (push) Successful in 1m23s
Restarting the control plane over a running two-node replica adopted half of it. Every shard it still had was healthy, so it was reported Ready — with one shard of two and no endpoint at all, because the missing one was rank 0. The router was handed a replica nothing could route to. The cause is that a replica had no idea how many shards it should have. It could tell "every shard I have is ready" and nothing else, which is the same sentence whether or not half of it is gone. It now carries the count from its profile, set at placement and again at adoption, and a replica short of shards fails rather than pretending. Onboarding also wrote no engine paths, so a node onboarded by the script could not find llama-server and failed every shard placed on it with "executable file not found". The script now takes them, and says so when none are given — a node with no engine paths can run the sim engine and nothing else. The cluster now runs both at once: two-node simulated replicas on each fabric, and real llama.cpp inference on the CPU, which is a better test bed than either alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
05bdc7d6d4 |
Add ctl bench, and fix two holes it walked into
`ctl bench` measures what a replica can actually serve, so weight, target_waiting and kv_ceiling stop being numbers somebody typed. Concurrency climbs until the SLO breaks and the last level that met it is the capacity: peak throughput arrives long after latency has become unusable, so throughput alone overstates what the hardware can serve. Queue depth and KV usage are read from the engine while it is still meeting the target, which is exactly what admission should treat as healthy. Prompts are unique per request by default. With one repeated prompt the engine's cache answers most of the prefill and the measurement describes the cache rather than the hardware; --shared-prefix asks that question deliberately, and it now produces byte-identical prompts, which the first version did not — it dropped the random header and still picked words at random, so the cache missed and the flag quietly measured the thing it exists to avoid. Running it against the cluster turned up two real defects: A replica stuck in Starting never timed out. Every other way out of that state needs the agent to report something, and when it never does the replica sits there holding GPUs and counting towards the model's replica count while serving nothing. Two of them had been like that for eight minutes past a two-minute startup timeout, and nothing would ever have moved them. They were stuck because adoption lost to a replica we had only placed. Replica ids come from the model name and a counter, so a control plane that restarts and places before its agents reconnect builds a different replica under the same id on a freshly allocated port; the agent's report of the engine that was actually serving was then discarded as already-known. The cluster pointed at port 20004 while the engine answered on 20000. The fleet is authoritative about what is running, so what we merely placed now loses to what an agent reports — unless our copy has a shard that has actually come up, in which case two things really are running and the stray is the one we did not ask for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e1eb1d4149 |
Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f0134a2f95 |
Phase 3 (2/3): catalog and reconciler
CI / test (push) Successful in 24s
The desired-versus-actual loop, with a Dispatcher interface where the agent stream will go. The loop's logic is where the interesting failures live, not the transport, so it is tested against a fake dispatcher and a synthetic clock. - catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and fallbacks resolve at load rather than at request time, because a typo in a fallback should stop a deployment instead of surfacing during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load rather than minutes into loading. - reconciler: one replica placed per pass, since placement depends on what is already allocated and several from one snapshot would be handed the same GPUs. Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves. - a failed shard fails the whole replica. The others are waiting on collectives that will never complete while holding GPUs, so every shard is stopped and the replica rebuilt, with a jittered exponential backoff carried on the model rather than on the replica that was just destroyed. - node expiry has two stages, Unreachable then Lost, deliberately far apart: a network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading. - scale-down drains before stopping, so in-flight requests finish. The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the reconciliation half end to end. The router continuing to serve throughout needs the EndpointSource wiring and the agent stream. Writing the tests caught a real property of the design worth recording: advancing the clock makes every node look silent, so placement correctly stops. The test harness now heartbeats nodes as it advances time, except where a node going silent is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |