a8bb417618d269f2c2f35d901a82ae2cfefda01e
19
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e725adc2f0 |
Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it went, or every candidate turned down), scale-up/down, drain, stop, eviction, preemption, reaping and restarts beneath it. A replica's start is a span open from placement until it serves; the start command carries its context in a new ControlMessage.trace field, and the agent continues the trace through the weights fetch, engine launch, rendezvous and readiness wait. Stops, drains, adoption and engine installs are traced on the node, model jobs are a trace with a span per step linked to the replica's start, and agent.yaml takes the same tracing block as server.yaml. Off, every span is the no-op one and commands carry nothing. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
a3ee48aced |
Restart in place at once when only the old replica is in the way
A restart with no room beside it waited the drain grace period before stopping the old replica, and stopped it even when the replacement would fit nowhere. It now stops at once when the replica alone blocks its replacement, gives the restart up (saying why) when nothing would fit, and ctl restart --in-place stops first on request. Refs #2 (item 3). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
adb1b3cf2a |
Add replicas under load, up to a model's max, and take them back when quiet
A model's max never did anything: desiredFor returned the minimum, or one for an on-demand model someone was asking for, whatever the queue. Six small CPU nodes run out of slots quickly, and a model with room to grow stayed at its minimum while requests waited. The router already scrapes each engine's running and waiting requests. The control plane now reads them per model on every tick, alongside when each replica was last used, and the reconciler keeps how many replicas load has added above a model's base. One or more requests waiting per ready replica, held for a minute, adds one; ten minutes with nothing waiting removes one of those again. It never goes above max or below the base, moves one step at a time, and does not add while the last addition is still placing or starting — on this fleet a replica is minutes of weights and gigabytes of memory, and a spike should not leave a pile of replicas behind it. Decisions are logged with the load behind them. Nothing live changes until a model's max is raised above its min. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
37a80a9b44 |
Park a model that fails every time it starts, and say so everywhere
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not know — was started 155 times in eight hours, failing the same way each time, and phi-4, which fitted on no node, logged "no placement" 250 times in two days. Neither showed up anywhere an operator looked: the backoff never gave up, no metric said a model was failing or unplaceable, and the one alert for a model with nothing serving could not fire, because its series was published only while a replica was ready. After five failures in a row a model is parked: not placed again until its definition changes — a new source or profile — or `llmgw ctl retry` is run. Status, the model list and the admin API show it with its last error. The placement warning is logged when its reason changes or every ten minutes. A request for a parked or unplaceable model is answered at once with why, rather than "loading; retry shortly", which a client retries forever. Each request still clears the previous placement result and triggers a fresh attempt, so a model that fits once room is made is not refused on a stale answer; the router waits for that attempt, not for its full activation window. Metrics now cover model health (desired, ready at 0 for what should run, consecutive start failures, parked, unplaceable), engine installs per node, leadership, and certificate expiry read from disk at each scrape, with alerts for each. `ctl status` counts a model against what it should have now, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
ad6d2d348d |
Restart replicas without taking them out of service first
There was no way to restart a running replica. A running engine picks up an engine key, an edited profile or a new engine version only when it starts, so after v0.10.3 the nine replicas already running stayed without a key — reachable from any node — with nothing to do about it short of draining nodes, which on a small CPU fleet has nowhere to move a model to. `llmgw ctl restart REPLICA... | --model M | --all` queues replicas to be replaced. A replica being replaced keeps serving but stops counting toward its model, so the reconciler places a replacement beside it; once that is ready the old one is drained and stopped as a node drain would. Replicas of one model go one at a time, so a model never needs the memory of every replica twice. Where no second copy fits anywhere, the old replica is stopped after the drain grace period and the new one starts in its place, and the log says so, since that is the one case with an outage. `ctl replicas` shows "(restart queued)" and "(restart replacing)" while it happens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
9853936e0b |
Give every engine a key only the router holds
Engines listen on every address their node has, with no key. Checked on the live cluster: from a neighbouring node, a request straight to an engine's port was answered — past the gateway's authentication, quotas and accounting, by anyone who could reach the node. Each replica's engine is now started with a key and refuses requests without it. The key is derived from the replica id with an HMAC over a secret kept in the store, rather than stored per replica: the router has to send the right key to replicas this process never started — adopted after a restart, or placed by the other control plane — and a function of a shared secret gives that without a table to keep in step. A key taken from one node opens that replica only. The secret is made on first use and copied with the rest of the store when moving to PostgreSQL; a new one would lock the router out of every running engine at the moment of the move. The key reaches llama.cpp and vLLM through their environment (LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every user on the node can read; SGLang only takes it as an argument. The agent's readiness check, the router's proxy and its metrics scrape send it, the client's own Authorization still never reaches the engine, and `ctl bench` fetches the key of the replica it is pointed at from the admin API. Engines started before this keep running without a key until their replica is next started. With no store there is nowhere to keep the secret, and engines start without one, with a warning. Also formats two files that went in unformatted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
|
03e38e7dda |
Carry the profile fingerprint to the agent and back
CI / test (push) Successful in 1m39s
The stale-profile flag could not fire where it matters. The control plane keeps no record of a running replica across its own restart — it rebuilds them from what the agents report — and adoption left the fingerprint empty, so every replica read as unknown afterwards. That is precisely the moment the catalog is most likely to have been edited: change the configuration, restart the server, and nothing would ever say the fleet disagrees with it. The fingerprint now travels with StartShard, the agent keeps it beside the pool and fabric it already echoes back, and adoption takes the replica's version from the agent rather than from the catalog it is about to be compared against. An agent too old to report it sends nothing, which still reads as unknown rather than as matching. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
9b32531ce3 |
Stop an idle model to make room for one somebody asked for
CI / test (push) Successful in 1m29s
On-demand loading worked right up to the moment the cluster was full and then stopped meaning anything: the model somebody had just asked for could not be placed, and the cards were held by a model nobody had asked for in an hour. That is the only situation in which any of it matters. Deliberately narrow. Only idle replicas of other on-demand models — a model at min 1 is a promise that it stays up, and breaking it to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, since evicting a multi-node one frees several machines and costs tens of minutes to rebuild. One at a time, because freeing everything that qualifies would empty a cluster for a request that needed one node. And never a replica with no reported usage: after a router restart that is all of them, and unknown is not idle. Writing the test found a gap the feature had rather than a bug in the test. Two on-demand models on a cluster with room for one would take turns evicting each other, at a model load per cycle — worse for both than one of them waiting. A replica is now safe from eviction for a few minutes after it starts serving, which bounds that. Which in turn found a latent bug: deriveState stamped ReadyAt with time.Now() inside a package that has an injectable clock, so the new lifetime rule could not be tested at all — the fake clock said three minutes and the wall clock said eight months. Invisible while nothing compared ReadyAt to anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
9d8cd5e284 |
Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing, and they are the same thing seen from two distances. Idle tracking. The control plane knew what was placed and whether it was healthy, never whether anyone wanted it — so on a fleet with more models than cards there was no way to answer the first question anyone asks, which is what can I stop. The router is the only thing that knows, so it now reports it: a timestamp written in Acquire, where a request already meets an endpoint under a lock already held, and collected once a tick as a map. That shape is the point. Per-request notification would be a firehose carrying one bit — the newest timestamp wins — and this costs the same on four replicas as on four hundred. It travels through an interface in the direction opposite EndpointSink, for the same reason that one exists: the control plane decides what should run, the router knows whether anyone wants it, and neither should need to be in the other's process to say so. Unknown is kept distinct from idle throughout. A router that has just restarted knows nothing, and a zero timestamp would be the oldest idle replica in every comparison — so one restart would nominate an entire fleet for eviction. Absent from the map, "-" in the table, negative in the JSON, and IdleFor returns a second value rather than a duration anything could act on by mistake. ctl status. A per-node listing is right for four nodes and wrong for two hundred: how full am I, what is broken, what is holding cards nobody is using are aggregates, and reading them off a long table is a chore, which means it gets skipped. Capacity per GPU class with usage bars, free whole nodes per fabric — the number that decides whether a multi-node replica can land, since free cards on the wrong fabric are not capacity for a model that spans nodes — replica and model counts, models that cannot be placed, and what has gone unasked-for. Derived from the same state the tables show, so it cannot disagree with them. Unhealthy GPUs count as neither used nor free, because counting them free is how a cluster looks like it has room it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
4f9fef2a98 |
Make settling outlast an agent reconnect backoff
CI / test (push) Has been cancelled
The window worked on the cluster and settled on the wrong answer. After a restart node05 registered in one second and the other four arrived seventeen seconds later, because a disconnected agent reconnects on a backoff that reaches thirty seconds. A five-second steady period called a fleet of one settled and resumed placement while four nodes were still coming — the exact situation the window exists to prevent, reached by being impatient about it. Nothing was misplaced this time only because the model that wanted placing needed two nodes on a fabric and could not have it. With a single-node model it would have landed on node05 and stayed there. A minute now, and steady for half of it rather than a quarter. Adoption is unaffected, so a restart still recovers everything already running immediately; only new placement waits. The test that covers it drives Reconcile the way the ticker does, because the node count is noticed on a pass rather than the instant a node appears — which the first version of the test got wrong and hid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e05d5d48ee |
Do not place on a fleet that has not reported in yet
CI / test (push) Has been cancelled
A control plane that has just started knows nothing, and its first reconcile tick runs before any agent has registered. "needs 2 nodes on one fabric, but no node reports a fabric" in the log is that, and it looked harmless because adoption catches up. What is not harmless is placing on the fraction of the fleet that happened to report first. After one restart both replicas of qwen2.5-0.5b landed on node04 — the only node registered at that instant — when four were coming, and nothing ever moved them, because the desired count was met. Placement now waits, and leaves by either of two doors, because either alone is wrong. Waiting for the window to expire would cost that much downtime on every restart even when the whole fleet reconnected in a second; waiting only for the node count to stop changing would hang forever on a cluster whose nodes never all come back. So: the count holds steady for a quarter of the window, or the window expires. Only placement waits. Node expiry, stuck-replica expiry, draining and reaping all stay correct on a partial fleet, and adoption in particular must not wait — it is how a restart recovers what is already running, and delaying it would leave live engines unaccounted for exactly while the reconciler decides what to build. The window is paid once per process. A node joining a running cluster is an ordinary event, not a reason to stop placing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
6f3bde099e |
Let a multi-node replica be adopted one agent at a time
CI / test (push) Successful in 1m20s
A replica spanning two nodes is reported by two agents, registering
separately about a second apart. Adoption assumed one call carried a whole
replica, so the first half was adopted with WantShards=2 and immediately
failed for "replica has 1 of 2 shards" — and when the second node
registered, its half was condemned as unclaimed because the replica was
already written off.
big-model therefore could not survive a control plane restart. Seen after
today's deploy: adopted, failed, both halves stopped, rebuilt from
scratch, backing off 9s, 20s, 38s, and never becoming ready. The first
reconcile tick also runs before any agent has reconnected, which is the
harmless-looking "no node reports a fabric" that precedes it.
Two changes. Adopt merges a group into a replica it is still assembling
instead of treating it as a competing claim, and takes the endpoint from
whichever group carries rank 0 — the leader's agent is not necessarily the
first to register. deriveState leaves a replica that is short of shards
while still assembling in Starting rather than Failed, which puts it under
the startup timeout that already exists: a sibling that never arrives
still fails, one second later than it used to rather than one second
before its sibling reports in. AdoptedAt is cleared once the replica is
whole, so losing a shard after that is a failure again.
The existing multi-node adoption test passed throughout, because it hands
both shards to a single Adopt call — which no caller makes. Same shape as
the agentsvc bug in
|
||
|
|
aef1c143c5 |
Let go of shards an agent says it is not running
CI / test (push) Successful in 1m19s
Adoption handled "the agent has something we do not know about". The mirror image was missing: "we think the agent has something it does not". Both follow from the same rule — the fleet is authoritative about what is running (plan §6.5) — and without the second half an engine that dies while its agent survives leaves the control plane believing a replica is Ready forever. Measured on the cluster: the engine was killed, the agent restarted and registered with nothing running, and the replica stayed Ready pointing at a dead port for two and a half minutes until I stopped watching. Traffic survived, because the breaker took the endpoint out and retries covered it — 24 requests, none failed — so the only symptom was a cluster quietly running a third short. A registration follows a fresh connection, so anything dispatched over a previous one is not running by definition; there is no window where a shard is legitimately started but unreported. The previous commit's disown rule was too broad, which this turned up: when a known replica had a live shard it disowned everything the agent reported for it, including the shards we already knew. A reconnecting agent would have been told to stop the replica that was serving. It now disowns only what it does not recognise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
e1eb1d4149 |
Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
1d1a6aebb0 |
Phase 4: make a multi-node replica able to start
CI / test (push) Failing after 53s
Placement and environment generation had landed early, which made multi-node look further along than it was. A replica spanning nodes could not come up at all, and all three reasons came down to assumptions that only hold on one node. Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers; workers timed out, reported Failed and took the replica with them. Engine.ServesHTTP now says which ranks serve, and a worker is judged by staying alive. That is deliberately weaker evidence: the replica's real readiness is the leader answering, which only happens once every worker has joined. A shard is not always one process. Engine.Commands returns a sequence — every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's. vLLM under Ray is why: the leader runs a Ray head and then the server against it, while a worker runs only the join and no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist. Sidecars are killed with the shard, because a Ray head left behind holds the port the next replica needs. Start order was backwards. Workers went first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own. A command can now also require that rendezvous point to be listening before the next starts, bounded by a timeout so a head that never appears fails the shard instead of hanging it. Stop was also killing only the engine, leaving sidecars behind. None of this is verified against real engines — it is tested with stub processes. The Ray sequencing, and which launch mode a given vLLM version wants, can only be confirmed on hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
75a8e76706 |
Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s
Not persisting replica state left one hole: restarting the control plane made it forget every replica while the engines kept running. They held GPUs the inventory believed were free, so the next placement failed to allocate, with an error naming the new process rather than the old one. The fix belongs on the agent, not in a database. A row can say "running" about a process that died; the fleet cannot. - the supervisor writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica. - Adopt takes over an engine a previous agent left running. It is not this process's child, so it cannot be waited on: it is watched by polling and stopped by signalling its group directly. Its earlier output is gone and the log says so rather than presenting emptiness as silence. - every Register reports what the agent is running, not only the first, because a reconnect may be a control plane restart and has to recover the same way. - the reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped; nothing will route to it and it is holding cards. - an engine the agent cannot identify is killed. A process holding GPUs that nothing can describe is worse than no process, because placement hands those GPUs out again. Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they do not have, and reported them stopped without killing anything. TestAgentReconnects had been passing for no reason. httptest's CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c hijacks the connection and httptest stops tracking hijacked ones — so the agent never reconnected, and the assertion on the connection map passed on a stale entry. Tests now sever connections through their own listener and count opened streams. Plan §11 also records what multi-node still needs, since placement landing early makes it look further along than it is: workers never become Ready because the agent health-checks every shard while only rank 0 serves HTTP; RayCommand is written and tested but never called; and startup is fire-and-forget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
f0134a2f95 |
Phase 3 (2/3): catalog and reconciler
CI / test (push) Successful in 24s
The desired-versus-actual loop, with a Dispatcher interface where the agent stream will go. The loop's logic is where the interesting failures live, not the transport, so it is tested against a fake dispatcher and a synthetic clock. - catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and fallbacks resolve at load rather than at request time, because a typo in a fallback should stop a deployment instead of surfacing during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load rather than minutes into loading. - reconciler: one replica placed per pass, since placement depends on what is already allocated and several from one snapshot would be handed the same GPUs. Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves. - a failed shard fails the whole replica. The others are waiting on collectives that will never complete while holding GPUs, so every shard is stopped and the replica rebuilt, with a jittered exponential backoff carried on the model rather than on the replica that was just destroyed. - node expiry has two stages, Unreachable then Lost, deliberately far apart: a network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading. - scale-down drains before stopping, so in-flight requests finish. The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the reconciliation half end to end. The router continuing to serve throughout needs the EndpointSource wiring and the agent stream. Writing the tests caught a real property of the design worth recording: advancing the clock makes every node look silent, so placement correctly stops. The test harness now heartbeats nodes as it advances time, except where a node going silent is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |