707d774c76bb154dce8848539cc95cfe2befed85
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5
|
707d774c76 |
Change a model's replica count without editing a file on the control plane
The count is scaling.min, and until now the only way to change it was to edit server.yaml on the host and restart — or, for a model added at runtime, to remove it and add it back, which means having its whole definition to hand. `llmgw ctl scale <model> --min N` instead. Only the scaling block moves: everything else is read back from what was stored and written out untouched, so changing a number cannot quietly become a redefinition. The catalog is rebuilt and validated before anything is written, as in Add. A max below a min is refused with the stored definition untouched — persisting a rejected change would come back on the next restart as a catalog that does not build. The answer reports the scaling in force rather than the request. The catalog fills in a max that was left out, and a caller echoing its own request would print a number the cluster is not using. A model declared in configuration is refused, for the reason Remove already refuses one: the YAML is read on every start, so the new count would hold until the next restart and then silently revert. A cluster disagreeing with its own config file is worse than an error saying where to edit. The help says what --max does, because it is not what it looks like: a model with a minimum runs exactly that many and nothing scales it up under load, and an on-demand model loads one replica however high its max. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
a35e8796d2 |
Carry a download's progress to where somebody is looking at it
CI / test (push) Successful in 1m28s
The agent has always logged a percentage as it fetches weights, with a comment saying a replica pulling a hundred gigabytes and one that is wedged look identical from the control plane. They still did: the log is on the node, and the node is not where anyone is looking when a replica has been starting for ten minutes. Progress now rides on the shard status and appears under the replica in `ctl replicas` — a line rather than a column, because most replicas are not downloading anything and a column would be empty for all of them. It is deliberately not a state transition. Treating each chunk as one would re-mark the shard as starting on every report and reset the timer that decides when a slow start has become a failed one, so an hour-long download would keep a doomed replica alive indefinitely. It arrives on its own method for that reason, and a real transition clears it — otherwise a finished download sits in the listing at ninety-nine percent under a shard that has been serving for an hour. Watchers are keyed by source rather than by shard, because a fetch is single-flight: eight shards starting together produce one download and all eight are waiting on it. No percentage is invented when the server sent no length. One that stops moving is worse than a byte count that keeps rising, so the field is absent and the listing says "so far". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
c13c567cc5 |
Know where the weights are, without owning the storage
CI / test (push) Successful in 1m32s
Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
331de9b495 |
Add and remove models from the command line
CI / test (push) Successful in 1m30s
The last piece of not restarting a control plane to try a model: llmgw ctl model add -f qwen-coder-32b.yaml llmgw ctl model remove qwen-coder-32b The file is the same YAML a model takes in server.yaml, read by the same koanf parser through the same tags, so an entry moves between the two by being copied and a field that works in one cannot silently mean nothing in the other. Wired only when there is a store. A control plane without one says so rather than accepting a model that vanishes on the next restart, which is the kind of success that wastes an afternoon. At startup the catalog is rebuilt from configuration plus what was stored, so models added before a restart are still there after it — without that the whole feature would be a slower way to lose work. Every failure adding a model is the definition being wrong — a collision, a missing fallback, a profile that cannot be placed — so they are 400s. A 500 would send someone to the server's logs for a mistake in their own file. The actor comes from the caller and is kept with the row, so a catalog that has grown over months can say where each entry came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1c57a4e378 |
Let the catalog change while the cluster is running
CI / test (push) Successful in 1m29s
Adding a model was an operator procedure with downtime: edit a file on the control plane, restart it, wait out the settling window, watch replicas churn while adoption catches up. I did that eight times in a day, and one of the restarts cost two minutes of outage while stalled replicas timed out. None of it was necessary. The reconciler already compares desired against actual on every pass, so a new catalog is simply a different desired. A Catalog stays immutable — New is the only way to build one and it refuses an invalid one — and what changes is which immutable catalog is current. Readers take the pointer and finish their operation against it, so a swap mid-request is invisible rather than a race. The router is told rather than asked, the same shape as the endpoint list and for the same reason: a push happens once per change, a pull would happen once per request. The router still seeds from configuration as well as being pushed to. A router with a static endpoint list and no control plane is a supported mode and has nothing to push it a catalog; without seeding it would resolve no model at all. Which turned up a bug that was latent while catalogs were fixed. reconcileModel only ever visits models that are in the catalog, so a replica whose model had been removed was visited by nothing: it kept its GPUs, kept counting, and kept running an engine for a name no client could ask for. The endpoint publisher skipped such a replica with the comment "the reconciler will tear it down", which was not true. It is now, and the comment says what actually happens instead of what was assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9d8cd5e284 |
Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing, and they are the same thing seen from two distances. Idle tracking. The control plane knew what was placed and whether it was healthy, never whether anyone wanted it — so on a fleet with more models than cards there was no way to answer the first question anyone asks, which is what can I stop. The router is the only thing that knows, so it now reports it: a timestamp written in Acquire, where a request already meets an endpoint under a lock already held, and collected once a tick as a map. That shape is the point. Per-request notification would be a firehose carrying one bit — the newest timestamp wins — and this costs the same on four replicas as on four hundred. It travels through an interface in the direction opposite EndpointSink, for the same reason that one exists: the control plane decides what should run, the router knows whether anyone wants it, and neither should need to be in the other's process to say so. Unknown is kept distinct from idle throughout. A router that has just restarted knows nothing, and a zero timestamp would be the oldest idle replica in every comparison — so one restart would nominate an entire fleet for eviction. Absent from the map, "-" in the table, negative in the JSON, and IdleFor returns a second value rather than a duration anything could act on by mistake. ctl status. A per-node listing is right for four nodes and wrong for two hundred: how full am I, what is broken, what is holding cards nobody is using are aggregates, and reading them off a long table is a chore, which means it gets skipped. Capacity per GPU class with usage bars, free whole nodes per fabric — the number that decides whether a multi-node replica can land, since free cards on the wrong fabric are not capacity for a model that spans nodes — replica and model counts, models that cannot be placed, and what has gone unasked-for. Derived from the same state the tables show, so it cannot disagree with them. Unhealthy GPUs count as neither used nor free, because counting them free is how a cluster looks like it has room it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
7e65cdf5c9 |
Fix the adminapi tests the settling window broke
CI / test (push) Successful in 1m36s
|
||
|
Ilker ManapandClaude Opus 5
|
3081c6f69b |
Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1839832b8b |
Add a landing page and a read-only cluster dashboard
CI / test (push) Successful in 1m2s
Pasting the gateway's address into a browser returned 404, which looks like a broken endpoint rather than one that tells you how to use it. Two pages, and the split between them is a security boundary rather than a layout choice. The landing page is served without a key — it has to be, because the person reading it is the one who does not have one yet — so it contains no data at all: it says how to connect, not what is here. Model names stay behind /v1/models. It reconstructs the address from the forwarded headers so the copy-pasteable example names the address the client actually used rather than the internal one; those headers are reflected into displayed text and never used for a decision. The dashboard shows nodes, models, replicas, placement failures and usage, and lives on the admin listener only. That listener is loopback-bound and also serves pprof, which can dump the process heap — where API keys sit in plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is read-only: anything that mints a credential stays in `llmgw keys`, where it needs a shell rather than a browser session this project has no way to authenticate. Mounting the landing page uncovered a hole in the exemption list. An entry ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing page would have exempted every path there is — authentication still installed, still logging, and letting the whole API through. "/" is now exact, with a test enumerating what must and must not be exempt. Server-rendered templates and one stylesheet, embedded. The single-binary goal rules out a JavaScript toolchain and nothing on either page needs one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
e1eb1d4149 |
Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |