fa60b81f37e8d668cdd30565f89e6576dd91c402
11
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ilker ManapandClaude Opus 5
|
774e134ed1 |
Refuse a node with no room, and take back only what llmgw put there
CI / test (push) Successful in 1m39s
Two halves of the same problem: nothing stopped a model being placed on a node that could not hold it, and nothing ever removed the weights of a model that had been deleted from the catalog. Placement now has a disk check, shaped exactly like the memory one it sits beside, with one addition: it is skipped on a node that already holds the weights. A node holding the file needs no room for it, and without that the node best placed to run a model — because it already has it — is the one most likely to be rejected, since holding the file is what used the space. Reclaiming is the dangerous half, so the whole design is about what it will not do. llmgw deletes only files it fetched itself, and that is a calculation rather than a heuristic: the cache name is a hash of the source, so the set of wanted files can be computed from the catalog exactly. Anything without that name was put there by an operator — scp, a shared mount, a copy by hand — and is none of this system's business however unreferenced it looks. A model that took a week to copy must not be removable by a command someone ran to tidy up. The node checks every path again before unlinking it, and that duplication is the point. The control plane knows the catalog; the node knows its disk. A control plane that is wrong, newer than the agent, or tampered with must not be able to talk a node into deleting something, so the agent refuses anything outside its own cache directory or without the cache name — including a path that tries to traverse out of it. Listing is the default and deleting takes --collect. The summary says "asked to remove", because each node decides, and one that refuses a path it cannot prove it fetched is this working rather than failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
753f4e3265 |
Let a node say how much room it has left
CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
d07dddb1bb |
Say plainly whether a model is serving, and why not
CI / test (push) Successful in 1m28s
Reported after watching a model appear in ctl models before it could answer anything. Checked: clients are not told early — /v1/models listed it only once a replica was ready, and the chat picker reads the same source. The operator view showing it is deliberate, because a model coming up is exactly what someone running the cluster wants to see. But the report pointed at something real. That view distinguished "a replica is coming up" from "nothing could be placed" by a parenthetical — "0/1 ready (1 total)" against "0/1 ready" — and those need opposite responses: one means wait, the other means look at explain-placement. On four models the difference is findable. On forty it is invisible, and a model that will never start reads exactly like one that is thirty seconds away. SERVING now answers the question that was being asked of the replica counts — can a client use this right now — and STATUS says which of ready, starting, not placed or stopped is the case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
331de9b495 |
Add and remove models from the command line
CI / test (push) Successful in 1m30s
The last piece of not restarting a control plane to try a model: llmgw ctl model add -f qwen-coder-32b.yaml llmgw ctl model remove qwen-coder-32b The file is the same YAML a model takes in server.yaml, read by the same koanf parser through the same tags, so an entry moves between the two by being copied and a field that works in one cannot silently mean nothing in the other. Wired only when there is a store. A control plane without one says so rather than accepting a model that vanishes on the next restart, which is the kind of success that wastes an afternoon. At startup the catalog is rebuilt from configuration plus what was stored, so models added before a restart are still there after it — without that the whole feature would be a slower way to lose work. Every failure adding a model is the definition being wrong — a collision, a missing fallback, a profile that cannot be placed — so they are 400s. A 500 would send someone to the server's logs for a mistake in their own file. The actor comes from the caller and is kept with the row, so a catalog that has grown over months can say where each entry came from. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
9d8cd5e284 |
Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s
Two things a cluster with many nodes and many GPUs needs that were missing, and they are the same thing seen from two distances. Idle tracking. The control plane knew what was placed and whether it was healthy, never whether anyone wanted it — so on a fleet with more models than cards there was no way to answer the first question anyone asks, which is what can I stop. The router is the only thing that knows, so it now reports it: a timestamp written in Acquire, where a request already meets an endpoint under a lock already held, and collected once a tick as a map. That shape is the point. Per-request notification would be a firehose carrying one bit — the newest timestamp wins — and this costs the same on four replicas as on four hundred. It travels through an interface in the direction opposite EndpointSink, for the same reason that one exists: the control plane decides what should run, the router knows whether anyone wants it, and neither should need to be in the other's process to say so. Unknown is kept distinct from idle throughout. A router that has just restarted knows nothing, and a zero timestamp would be the oldest idle replica in every comparison — so one restart would nominate an entire fleet for eviction. Absent from the map, "-" in the table, negative in the JSON, and IdleFor returns a second value rather than a duration anything could act on by mistake. ctl status. A per-node listing is right for four nodes and wrong for two hundred: how full am I, what is broken, what is holding cards nobody is using are aggregates, and reading them off a long table is a chore, which means it gets skipped. Capacity per GPU class with usage bars, free whole nodes per fabric — the number that decides whether a multi-node replica can land, since free cards on the wrong fabric are not capacity for a model that spans nodes — replica and model counts, models that cannot be placed, and what has gone unasked-for. Derived from the same state the tables show, so it cannot disagree with them. Unhealthy GPUs count as neither used nor free, because counting them free is how a cluster looks like it has room it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
84896c0ad2 |
Add ctl tool-check, so an afternoon becomes one command
CI / test (push) Successful in 1m32s
The catalog now declares whether a model supports tools, which makes it a
claim. This is the claim that can be checked, and the one worth checking:
the gateway forwards a tools array to any model, and an engine that does
not return structured tool_calls looks exactly like a model that ignores
its tools. Every OpenAI-compatible client then behaves as though the model
is broken.
It separates the two diagnoses, which have the same symptom and different
fixes. The model ignoring its tools is fixed by choosing another model. The
model making the call and the engine handing it back as prose is fixed by
changing engine build, and no larger model will help — which is what
today's investigation established only after eliminating the gateway, the
model size, two chat templates and tool_choice in turn.
Against the live engine it says:
the model called the tool and the engine did not parse it.
{"name": "get_weather", "arguments": {"city": "Istanbul"}}
This is the engine, not the model.
engine-check asks whether an engine still accepts our flags; this asks
whether it behaves. Flags were the failure we had seen, so they were the
failure we checked for. Non-zero exit, so a rollout can be gated on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ilker ManapandClaude Opus 5
|
3081c6f69b |
Place on nodes that can actually hold the model
CI / test (push) Successful in 1m23s
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
5b425c32e5 |
Ship binaries with releases, and ask engines whether our flags still exist
CI / test (push) Successful in 1m31s
A release was a git tag. Installing llmgw on a node meant having a Go toolchain, and the onboarding script copied a binary from wherever it was run — which works and quietly makes whoever runs it the build server, with the fleet ending up on whatever was in their working tree. Tags now build static linux/amd64, linux/arm64 and darwin binaries and attach them. CGO_ENABLED=0 is the point: it has to run on a node whose libc is older than the builder's, and llmgw has no C dependency because modernc.org/sqlite is pure Go precisely so this stays true. Verified: "not a dynamic executable", runs on a node, reports its version. `ctl engine-check` closes the other half of the phase 4 gap. llmgw's launch arguments are read from each engine's documentation, and engines rename flags between versions; when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. It builds the widest command llmgw would ever produce — multi-node, since a flag that only appears there is the one nobody notices has gone — and checks each flag against the engine's own --help. No GPU: a CLI prints its help anywhere. Verified against llama.cpp b10991, all seven flags accepted. Flags are matched on a word boundary, so --port is not satisfied by --port-range. Without that, a renamed flag would sail through the check that exists to catch renamed flags. It also warns when the help text looks truncated, which is how the first run went: llama.cpp's help cut to 400 of its 725 lines reported five flags as missing that the running cluster uses. TestSidecarRunsBeforeEngine was flaky under a full run, and worse than flaky — its wait loop fell through on timeout, so a slow machine turned the ordering assertion into a no-op and the failure surfaced later as something else. It now asserts, with a deadline that suits a machine running every other package at the same time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
05bdc7d6d4 |
Add ctl bench, and fix two holes it walked into
`ctl bench` measures what a replica can actually serve, so weight, target_waiting and kv_ceiling stop being numbers somebody typed. Concurrency climbs until the SLO breaks and the last level that met it is the capacity: peak throughput arrives long after latency has become unusable, so throughput alone overstates what the hardware can serve. Queue depth and KV usage are read from the engine while it is still meeting the target, which is exactly what admission should treat as healthy. Prompts are unique per request by default. With one repeated prompt the engine's cache answers most of the prefill and the measurement describes the cache rather than the hardware; --shared-prefix asks that question deliberately, and it now produces byte-identical prompts, which the first version did not — it dropped the random header and still picked words at random, so the cache missed and the flag quietly measured the thing it exists to avoid. Running it against the cluster turned up two real defects: A replica stuck in Starting never timed out. Every other way out of that state needs the agent to report something, and when it never does the replica sits there holding GPUs and counting towards the model's replica count while serving nothing. Two of them had been like that for eight minutes past a two-minute startup timeout, and nothing would ever have moved them. They were stuck because adoption lost to a replica we had only placed. Replica ids come from the model name and a counter, so a control plane that restarts and places before its agents reconnect builds a different replica under the same id on a freshly allocated port; the agent's report of the engine that was actually serving was then discarded as already-known. The cluster pointed at port 20004 while the engine answered on 20000. The fleet is authoritative about what is running, so what we merely placed now loses to what an agent reports — unless our copy has a shard that has actually come up, in which case two things really are running and the stray is the one we did not ask for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Ilker ManapandClaude Opus 5
|
1e3cafdc86 |
Add a private CA and mutual TLS for the agent stream
A node's identity was whatever the agent said it was. With mutual TLS the control plane compares the client certificate's common name against the claimed node id and refuses the connection when they disagree, so identity becomes something a node proves rather than something it picks. A public authority cannot supply these certificates: Let's Encrypt and its peers stopped issuing the clientAuth usage, and Go verifies client chains against it. Hence internal/tlsconf/ca.go, a small authority whose key lives on one machine and whose root is trusted everywhere, and `llmgw ctl cert` over it so issuing is reproducible instead of remembered openssl. The server certificate's hosts matter more than they look: hostname verification reads subject alternative names and ignores the common name entirely, so a certificate without the address agents dial — bare IP included — is rejected no matter what its subject says. There is a test for exactly that, because it is the failure everyone hits once. requireIdentity stays wired to false until the configuration lands, so this commit changes no running behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 ilkerandClaude Opus 5
|
e1eb1d4149 |
Phase 3: ctl, and make draining a rolling move
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |