bf2d171e8941bbff7cbbfba078a5ceb18c388f9e
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5
|
40f5c930c4 |
Say when a replica is running an edited profile, and check a source from the web too
CI / test (push) Successful in 1m43s
Two defects, both of the same kind: something the CLI knew and the rest of the system did not. A profile can be edited while a replica is running against the old one. The reconciler deliberately leaves it alone — churning a replica that is serving costs more than the imperfection — but there was no way to see it, so the cluster could disagree with its own configuration indefinitely and read as healthy. Profile now carries a fingerprint, a replica records the one it was placed with, and both the API and `ctl replicas` say "stale profile" when they differ. An empty fingerprint is not stale: it means a replica adopted from an older agent, which is unknown rather than wrong. Adding a model from the admin page skipped the source check `ctl model add` has done all along, so a typo or a mirror that is down became a replica failing on a node minutes later, with an error naming the node. The check moves out of the CLI into a package both use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d998ea7ef5 |
Let a model say what it is for
CI / test (push) Successful in 1m42s
A gateway serving nine models offered nine names and nothing to choose between them. /v1/models is the only view most clients ever have, and it listed ids. Models carry a description now: written when the model is added, shown on the dashboard and sent to clients in /v1/models. Written rather than derived. A registry offers a pipeline tag and a wall of README, and neither answers the question somebody picking from a list is asking — which of these should I send this request to. "General chat, strong Swedish; slower than the 8B" is not in any metadata field. It is optional, and absent means absent rather than an empty string in the response: a model nobody described should read as undescribed, not as described with nothing. On the dashboard it sits under its row rather than in a column. A sentence in a table cell either wraps every other column into a stripe or gets cut off. Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7fe59bd5fb |
Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
277add5ccb |
Keep runtime models, so an added model outlives a restart
CI / test (push) Successful in 1m25s
The catalog can change while the cluster runs; this is where a change goes so it is still there tomorrow. Models declared in configuration and models added at runtime are two sources on purpose: config-as-code stays declarative and under review, and the database holds what someone added to try it. Configuration wins every conflict, and the direction matters. A request for a reviewed production model must never be answered by something added from a laptop, so a stored model colliding with a declared name is refused when it is added. A declared model cannot be removed at runtime either — it would return on the next restart, leaving the cluster disagreeing with its own config file until then, which is a worse answer than refusing and saying where to edit. The whole catalog is built before anything is written. A model naming a missing fallback, or colliding, or malformed in any of the ways catalog.New refuses, is rejected rather than persisted and then skipped forever after — a model that looks added and never appears is the worst outcome available. Removal is checked the same way, because another model may fall back to the one being removed, and a fallback pointing at nothing is discovered during the outage it was meant to cover. At startup each stored model is validated on its own against what is already accepted, so one bad row is skipped and named rather than taking the cluster with it. The declared models are precisely the ones that must survive that. Catalog types now carry json tags mirroring their koanf ones, so a stored definition is the same shape as the YAML someone writes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
3081c6f69b |
Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
e9fe3ad379 |
Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
8a894af1ca |
Let a profile say which nodes it is for
CI / test (push) Successful in 1m35s
Agents have reported labels since the beginning and nothing read them. That
was survivable while every CPU profile was a 0.5B model that fits anywhere,
and stops being survivable with a 14B one: a CPU profile has no card to be
free, so eligibleNodes accepts every ready node, and placing Qwen2.5-Coder
14B on one of the 8 GB nodes would OOM a machine that reported itself
perfectly healthy.
Nothing in the inventory says how much memory a node has, and inferring it
would be guessing at the wrong layer — the amount that matters is the
model's working set, which the scheduler does not know either. So the
operator says which nodes a profile is for, with the labels that already
exist:
node_selector:
ram: 32g
A rejection names the label it did not find, because "no candidates were
considered" is not something anyone can act on and "does not carry ram=32g"
is. An absent selector constrains nothing, which is every profile that
exists today, and there is a test that says so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
72b03a1a1c |
Say which model is simulated instead of letting it look broken
CI / test (push) Successful in 1m31s
big-model runs on the sim engine, whose whole vocabulary is the Greek alphabet, cycled at a fixed rate until max_tokens. That is what it is for: with no H100s it is the only thing exercising the TP8 two-node path, the fabric invariant and rebuild-after-node-loss. The output is the point. What was wrong is that nothing said so. /v1/models returned it with owned_by "llmgw", identical to the real model, and the chat picker sorted purely by name — so "big-model" led the list, was preselected by the browser, and anyone who typed a question and pressed Send got Greek letters back with no explanation. Reported as the model spitting out crap, which is a fair reading of what it looks like from outside. Simulated is derived from the profiles rather than configured: an operator who adds a sim profile and forgets a flag would be shipping placeholder text under a real model's name. A model counts as simulated only when every profile is, because one real profile means the request can still come back as inference. The marker on /v1/models is owned_by "llmgw-sim", which is already what the sim engine calls itself on its own /v1/models — the gateway was flattening a distinction that existed one layer down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
501db3d3e8 |
Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same OpenAI API on CPU, so the whole path — placement, supervision, scraping, scheduling, admission — can be exercised end to end on ordinary machines. Three things had to give way for it. A profile may now omit its GPU class, and that is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it; for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. Placement spreads CPU replicas instead of bin packing them, since unlike GPUs there is no allocation to stop two of them sharing a machine's cores. Metric mappings are now per engine rather than per cluster. llama.cpp calls the same three numbers requests_processing, requests_deferred and kv_cache_usage_ratio, and a single global mapping would leave half a mixed fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that names its own Running metric is taken as a complete statement, so the fields it leaves blank stay blank rather than being filled with names that engine cannot have. The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits across local GPUs by a different mechanism than this adapter drives, and accepting either would produce a replica that starts and then serves from one machine while the catalog believes it has several. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
f0134a2f95 |
Phase 3 (2/3): catalog and reconciler
CI / test (push) Successful in 24s
The desired-versus-actual loop, with a Dispatcher interface where the agent stream will go. The loop's logic is where the interesting failures live, not the transport, so it is tested against a fake dispatcher and a synthetic clock. - catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and fallbacks resolve at load rather than at request time, because a typo in a fallback should stop a deployment instead of surfacing during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load rather than minutes into loading. - reconciler: one replica placed per pass, since placement depends on what is already allocated and several from one snapshot would be handed the same GPUs. Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves. - a failed shard fails the whole replica. The others are waiting on collectives that will never complete while holding GPUs, so every shard is stopped and the replica rebuilt, with a jittered exponential backoff carried on the model rather than on the replica that was just destroyed. - node expiry has two stages, Unreachable then Lost, deliberately far apart: a network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading. - scale-down drains before stopping, so in-flight requests finish. The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the reconciliation half end to end. The router continuing to serve throughout needs the EndpointSource wiring and the agent stream. Writing the tests caught a real property of the design worth recording: advancing the clock makes every node look silent, so placement correctly stops. The test harness now heartbeats nodes as it advances time, except where a node going silent is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |