Two defects, both of the same kind: something the CLI knew and the rest of
the system did not.
A profile can be edited while a replica is running against the old one. The
reconciler deliberately leaves it alone — churning a replica that is serving
costs more than the imperfection — but there was no way to see it, so the
cluster could disagree with its own configuration indefinitely and read as
healthy. Profile now carries a fingerprint, a replica records the one it was
placed with, and both the API and `ctl replicas` say "stale profile" when
they differ. An empty fingerprint is not stale: it means a replica adopted
from an older agent, which is unknown rather than wrong.
Adding a model from the admin page skipped the source check `ctl model add`
has done all along, so a typo or a mirror that is down became a replica
failing on a node minutes later, with an error naming the node. The check
moves out of the CLI into a package both use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gateway serving nine models offered nine names and nothing to choose between
them. /v1/models is the only view most clients ever have, and it listed ids.
Models carry a description now: written when the model is added, shown on the
dashboard and sent to clients in /v1/models.
Written rather than derived. A registry offers a pipeline tag and a wall of
README, and neither answers the question somebody picking from a list is
asking — which of these should I send this request to. "General chat, strong
Swedish; slower than the 8B" is not in any metadata field.
It is optional, and absent means absent rather than an empty string in the
response: a model nobody described should read as undescribed, not as described
with nothing.
On the dashboard it sits under its row rather than in a column. A sentence in a
table cell either wraps every other column into a stripe or gets cut off.
Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com>
A node's registry holds an adapter for every engine whether or not its binary
exists, the agent never said which were installed, and placement carried an
Engine field it had never once read. So a vLLM replica was as happily placed on
a node with no vLLM: the node registered healthy, the shard failed minutes
later with "executable file not found" in the engine's own words, and the
reconciler tried again.
That is how fifteen gigabytes of safetensors reached this cluster's mirror
before anybody noticed it had no card and no vLLM to run them with.
Agents now probe what is installed and report it — the protocol has carried
engine_versions since it was written and nothing ever filled it in. Placement
filters on the key set, `ctl nodes` has a column, and the mirror page's serve
warning is a fact instead of a guess about cards, which would have waved
through a fleet of eight-GPU nodes that never had vLLM.
Probing resolves binaries and runs none of them. The first version asked each
engine for --version, which is fine for an engine and not fine for the wrapper
scripts people put in front of one: a script that shifts past flags it does not
recognise and then execs the server will start an inference server, at agent
startup, on every node. The agent's own test suite installs exactly such a
script and caught it. So presence is a path lookup, and the version is left to
`ctl engine-check`, which is run by somebody who meant to run the engine.
SGLang is the exception and has to execute, because a module has no path to
look up — but the command is llmgw's own one-line import rather than the
engine's launcher. Its exit code is the answer: python3 is on almost every
machine, and treating a failed import as "installed" reported SGLang on every
node in a fleet that had none, which is the failure being fixed, inverted.
Probed once, not per reconnect, and bounded by the context that registration
waits on: an agent that cannot reach the control plane is worse than one
reporting an incomplete list.
A node that reports no engines at all is not excluded anywhere — by placement,
or by the page. That is what every agent older than this reports, and emptying
a fleet on a control plane upgrade is a worse failure than the one being fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Quotas can be written on an organization, a team or a project, and nothing
could show what any of them had used: usage came back per key, per user and per
model only. A quota whose consumption cannot be seen is one nobody reviews
until the day it refuses somebody — and since the previous commit made a group
quota a shared pool, it is now a number several people are spending against
without being able to look at it.
`llmgw ctl usage --by team` sums every key beneath each group, with the budget
that level sets beside it.
Beside it, and only when that level sets one. A team bounded by its
organization's budget has none of its own, and printing the organization's
against each of its teams would read as though each had that much — the exact
misreading this work exists to remove.
Joined through api_keys rather than read from usage.project_id: that column
records where the key was when the request was served, while the quota is
written against the tree. They agree until a key is moved, and a report that
disagrees with the limit shown beside it is the useless one.
Three existing reports were fixed on the way, all failing on PostgreSQL and all
passing in tests, because the tests run on SQLite:
column "k.name" must appear in the GROUP BY clause (UsageSince)
column "us.email" must appear in the GROUP BY clause (UsageByUser)
column "usage.ts" must appear in the GROUP BY clause (UsageBuckets)
SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL
refuses. The first two were selecting names they had not grouped. The third
grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as
text and so saw a bare ts — it is grouped by ordinal now, the same expression by
construction. The dashboard's usage charts have been empty on the cluster this
whole time, with the reason only in the log.
All four queries were run against the cluster's own PostgreSQL before this was
committed, because passing on SQLite is what they were already doing.
The grouping level picks from a fixed table of levels rather than being
interpolated, so an unknown one is an error and not a fragment of SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.
`ctl bench --save` records it; `ctl capacity` reads it back.
Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.
The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.
The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.
Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.
The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.
A failed save is reported and not fatal: the run has already cost the GPU time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The count is scaling.min, and until now the only way to change it was to edit
server.yaml on the host and restart — or, for a model added at runtime, to
remove it and add it back, which means having its whole definition to hand.
`llmgw ctl scale <model> --min N` instead. Only the scaling block moves:
everything else is read back from what was stored and written out untouched, so
changing a number cannot quietly become a redefinition.
The catalog is rebuilt and validated before anything is written, as in Add. A
max below a min is refused with the stored definition untouched — persisting a
rejected change would come back on the next restart as a catalog that does not
build.
The answer reports the scaling in force rather than the request. The catalog
fills in a max that was left out, and a caller echoing its own request would
print a number the cluster is not using.
A model declared in configuration is refused, for the reason Remove already
refuses one: the YAML is read on every start, so the new count would hold until
the next restart and then silently revert. A cluster disagreeing with its own
config file is worse than an error saying where to edit.
The help says what --max does, because it is not what it looks like: a model
with a minimum runs exactly that many and nothing scales it up under load, and
an on-demand model loads one replica however high its max.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of the same problem: nothing stopped a model being placed on a node
that could not hold it, and nothing ever removed the weights of a model that
had been deleted from the catalog.
Placement now has a disk check, shaped exactly like the memory one it sits
beside, with one addition: it is skipped on a node that already holds the
weights. A node holding the file needs no room for it, and without that the
node best placed to run a model — because it already has it — is the one most
likely to be rejected, since holding the file is what used the space.
Reclaiming is the dangerous half, so the whole design is about what it will
not do. llmgw deletes only files it fetched itself, and that is a calculation
rather than a heuristic: the cache name is a hash of the source, so the set of
wanted files can be computed from the catalog exactly. Anything without that
name was put there by an operator — scp, a shared mount, a copy by hand — and
is none of this system's business however unreferenced it looks. A model that
took a week to copy must not be removable by a command someone ran to tidy up.
The node checks every path again before unlinking it, and that duplication is
the point. The control plane knows the catalog; the node knows its disk. A
control plane that is wrong, newer than the agent, or tampered with must not be
able to talk a node into deleting something, so the agent refuses anything
outside its own cache directory or without the cache name — including a path
that tries to traverse out of it.
Listing is the default and deleting takes --collect. The summary says "asked to
remove", because each node decides, and one that refuses a path it cannot prove
it fetched is this working rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space it returns in the same struct was thrown away. So the
only way to find out why a node was filling up was to ssh to it, which is what
this system exists to avoid.
Reported per filesystem rather than per configured directory. Free space
belongs to the disk, not the path: two model directories on one disk are one
pool of bytes, and counting them separately would let the control plane add
24 GB to 24 GB and place a 40 GB model on a node that has 24.
Shared filesystems are summed separately from local ones and left out of a
node's own total. Space on a filer belongs to everyone mounting it, so twenty
nodes would otherwise each report the same free terabyte, and a scheduler
would place twenty models into room for one.
Nothing reported means nothing is known, everywhere: an agent too old to send
storage and a node with no model directories both send nothing, and both must
read as unknown rather than as a full disk. `ctl nodes` and the dashboard show
a dash, the metric is absent rather than zero, and a missing directory — which
is what an unmounted filer looks like — is skipped rather than reported as
having no space.
Shown as free out of total, because the first is useless without the second:
8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root
partition, and only the second stops a model being placed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Storage will be different at every installation — Lustre, NFS over RDMA, an
image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes
it rather than managing it. An agent told which directories to look in
reports what it can see; how the files got there is nobody's business here,
and a node told nothing reports nothing and behaves exactly as every node
did before this existed.
That fact buys two things. Placement prefers a node that already holds the
model over one that must first move tens of gigabytes, as a preference and
never a filter: fetching them is a valid answer, and refusing to place would
turn a slow start into no start at all. And ctl status shows what each
fabric holds.
Per fabric because fabrics are not connected to one another. Weights in
ib-a are not weights in ib-b — the only path between them is the management
network — so a model seeded into one fabric leaves the rest of the cluster
unable to serve it without a slow copy, and nothing said so before now. One
shared filer counts the same as twenty local copies, because "can this
fabric serve it" is the question being asked.
Three distinctions the tests hold to, because each is a way to get this
quietly wrong. A node that was never asked to look must rank with one that
looked and has nothing, or enabling the feature on a single node would
drain every other. An empty directory is not weights, because a
half-finished download looks exactly like one and would send placement to a
node whose engine then fails to load. And local storage is never mistaken
for shared: a filer read as local costs an unnecessary copy, while local
read as shared would have the cluster believe a model is everywhere when it
is on one node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of not restarting a control plane to try a model:
llmgw ctl model add -f qwen-coder-32b.yaml
llmgw ctl model remove qwen-coder-32b
The file is the same YAML a model takes in server.yaml, read by the same
koanf parser through the same tags, so an entry moves between the two by
being copied and a field that works in one cannot silently mean nothing in
the other.
Wired only when there is a store. A control plane without one says so
rather than accepting a model that vanishes on the next restart, which is
the kind of success that wastes an afternoon. At startup the catalog is
rebuilt from configuration plus what was stored, so models added before a
restart are still there after it — without that the whole feature would be
a slower way to lose work.
Every failure adding a model is the definition being wrong — a collision, a
missing fallback, a profile that cannot be placed — so they are 400s. A 500
would send someone to the server's logs for a mistake in their own file.
The actor comes from the caller and is kept with the row, so a catalog that
has grown over months can say where each entry came from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adding a model was an operator procedure with downtime: edit a file on the
control plane, restart it, wait out the settling window, watch replicas
churn while adoption catches up. I did that eight times in a day, and one
of the restarts cost two minutes of outage while stalled replicas timed
out. None of it was necessary. The reconciler already compares desired
against actual on every pass, so a new catalog is simply a different
desired.
A Catalog stays immutable — New is the only way to build one and it refuses
an invalid one — and what changes is which immutable catalog is current.
Readers take the pointer and finish their operation against it, so a swap
mid-request is invisible rather than a race. The router is told rather than
asked, the same shape as the endpoint list and for the same reason: a push
happens once per change, a pull would happen once per request.
The router still seeds from configuration as well as being pushed to. A
router with a static endpoint list and no control plane is a supported mode
and has nothing to push it a catalog; without seeding it would resolve no
model at all.
Which turned up a bug that was latent while catalogs were fixed.
reconcileModel only ever visits models that are in the catalog, so a
replica whose model had been removed was visited by nothing: it kept its
GPUs, kept counting, and kept running an engine for a name no client could
ask for. The endpoint publisher skipped such a replica with the comment
"the reconciler will tear it down", which was not true. It is now, and the
comment says what actually happens instead of what was assumed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.
Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.
It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.
Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.
ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.
Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e05d5d4 was pushed with this suite failing. The full run was in front of me
and I printed "green" from an unconditional echo instead of reading it,
which is worse than not having looked.
The failure was real and not a surprise: explain-placement had no
rejections to show, because placement had not run yet. These tests build
the whole fleet before anything starts, which is exactly the state settling
waits for, so they opt out rather than simulate a startup they are not
about.
A negative SettleFor now means off, distinct from zero meaning "use the
default" — a test that wants no window must be able to say so without that
reading as "unset".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Placement had no idea how much memory a node has or a model needs. A CPU
profile has no card to be free, so eligibleNodes accepted every ready node:
Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed
just as happily on one of the 8 GB nodes, which reports itself perfectly
healthy right up until the OOM killer arrives.
node_selector was yesterday's stopgap and it is the operator restating by
hand something the system can know. Agents now report MemTotal, profiles
declare min_memory_gb, and placement filters on it, naming what was short —
"has 8 GB of memory, needs 24" is actionable where "no candidates were
considered" is not. The label selector stays, because "this node is for
coding models" is a real thing to want to say that memory does not express.
Two deliberate non-exclusions. A profile with no requirement fits anywhere,
which is every profile that exists today. And a node that reports no memory
is still usable: an agent older than the field says nothing, and letting a
control plane upgrade quietly empty half a fleet is a worse failure than the
one this prevents.
ctl nodes grows a MEM column showing "-" rather than 0 for a node that did
not say, because an old agent is not a small machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Pasting the gateway's address into a browser returned 404, which looks like a
broken endpoint rather than one that tells you how to use it.
Two pages, and the split between them is a security boundary rather than a
layout choice. The landing page is served without a key — it has to be,
because the person reading it is the one who does not have one yet — so it
contains no data at all: it says how to connect, not what is here. Model names
stay behind /v1/models. It reconstructs the address from the forwarded headers
so the copy-pasteable example names the address the client actually used
rather than the internal one; those headers are reflected into displayed text
and never used for a decision.
The dashboard shows nodes, models, replicas, placement failures and usage, and
lives on the admin listener only. That listener is loopback-bound and also
serves pprof, which can dump the process heap — where API keys sit in
plaintext — so reaching the page from elsewhere is an SSH tunnel's job. It is
read-only: anything that mints a credential stays in `llmgw keys`, where it
needs a shell rather than a browser session this project has no way to
authenticate.
Mounting the landing page uncovered a hole in the exemption list. An entry
ending in "/" was a prefix rule, and "/" ends in "/", so exempting the landing
page would have exempted every path there is — authentication still installed,
still logging, and letting the whole API through. "/" is now exact, with a
test enumerating what must and must not be exempt.
Server-rendered templates and one stylesheet, embedded. The single-binary goal
rules out a JavaScript toolchain and nothing on either page needs one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ctl talks to a management API on the admin listener, not the client-facing one.
The two have different audiences: one serves inference to whoever holds an API
key, the other lets a caller cordon nodes and read the cluster's shape.
- nodes surfaces unhealthy GPUs and why they are excluded. That is usually the
answer when a node looks idle but nothing will schedule onto it, and it was
invisible before.
- explain-placement prints every candidate considered and why it was rejected.
- replicas shows the shards a replica is made of, with per-shard errors.
- cordon, uncordon and drain are one endpoint with different targets, because
the difference between cordon and drain is exactly what an operator needs to
be sure of before running either.
Draining had to be fixed first to make ctl drain worth having. A drained replica
was stopped on the next pass, which cuts in-flight requests — the thing draining
exists to avoid. Now a node marked Draining has its replicas drained, the model
falls below its desired count, a replacement is placed elsewhere, and the
original is stopped only once the replacement is serving. A grace period bounds
it, because draining the only node that can host a model would otherwise never
finish and the node could never be taken out of service.
Verified against a two-node cluster of real binaries: draining moved a replica
to the surviving node and traffic kept returning 200 throughout.
ctl logs is deliberately absent. It needs a request/response pair the agent
protocol does not have, and the failure path already ships an engine log tail
with every status report — so the case that matters most is covered, and
reading a healthy shard's output on demand is not worth a protocol change yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
