The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.
Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.
So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.
There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no reference. Four narrative documents explain how to install and
operate llmgw, and the only way to learn that `lease_ttl` exists was to read
the one that happened to mention it — or the struct.
Written by hand it would have been wrong within a fortnight. Six settings
arrived in the last two weeks alone: store, servers, lease_ttl, the two
retention windows, min_disk_gb. A reference nobody maintains is worse than
none, because it is the same list with the setting you came to look up quietly
missing from it.
So it is derived from the structs, with the field comments as the descriptions
— which means the only way for it to be wrong is for the code to be wrong, and
it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys.
Two tests keep that true after everyone has forgotten this exists. One
regenerates and compares, so adding a setting without regenerating is a
failure rather than an omission — checked by adding one and watching it fail.
The other walks the tree for koanf tags and fails on a configuration file
that is in neither the source list nor the short list of structs an operator
never writes, because a list is a thing to forget.
The generator finds the repository from its own path rather than the working
directory, so `go generate ./...` produces the same document from anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five nodes fit in a table. Fifty do not, and this is built for people who
already have racks of GPUs — so the flags are how a question gets asked:
which nodes on this fabric, which of this GPU class, which are not ready,
which replicas have a shard on this machine.
Filtered here rather than in the API, so --json is narrowed the same way the
table is. A server-side filter would be two implementations of one predicate,
and the JSON one is the one nobody would notice had drifted.
An empty result says which case it is. A filter that matched nothing and an
empty fleet call for opposite next steps — one is a cluster to look at, the
other is a word to correct — and printing "no nodes" for both makes a mistyped
fabric name read as an outage.
A state matches on prefix because states are few and distinct and somebody
scanning a fleet types quickly. A pool matches either half, because it is
fabric/class and both are worth narrowing by. A replica is found from any of
its nodes, since one spanning two machines is on both.
--label takes a bare key as "carries this at all", which is how you ask for
every node with a role without caring which. The flag's help said so before
the parser did; the test caught them disagreeing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The store stops being a file on one machine, which is what stood between this
and more than one control plane.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three seconds to move leadership, six for every agent to arrive, and every
replica adopted rather than restarted. Also what it did not do: the public
address returned 502 while the standby served correctly on its own port,
because nothing in front of the two moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second half of leadership. A standby that refuses to place but still holds
an agent's stream is worse than useless: commands travel down the stream the
agent opened, so the leader's placements fail with "no agent connected" for a
node that is plainly connected.
So a standby refuses the stream, before reading the opening message, and says
why. And an agent takes a list rather than an address, trying each in turn — a
standby refusing, a stopped control plane refusing the connection and a crashed
one never answering are the same thing from a node, and the answer to all three
is somewhere else.
`server` and `servers` are both accepted, with `server` first when both are
given: every agent configuration written before there could be a second control
plane names one address, and an operator adding a standby expects the one
already working to stay preferred. With a single address the behaviour is
exactly what it was, including the backoff — which is skipped only while there
is another address that has not just been tried, because the delay exists to
stop hammering one control plane and the next one has not been hammered.
Removed a check that asked whether one field was empty, which is how the first
run of these tests failed: the list was populated and `server` was not, and an
agent configured entirely correctly was refused by its own validation.
The tests run a real agent against a real service over h2c. The first attempt
used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error
that said nothing about why — which is exactly what an operator would have seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These TRUNCATE every table before each test. Pointed at a deployment that is a
silent and complete data loss — every key, every user, every conversation.
Not hypothetical. I did it: an SSH tunnel to the production database had been
opened minutes earlier for a perfectly good reason, and the same DSN was then
handed to the test suite. Twenty-one users, nine API keys, eighteen sessions,
four hundred and sixty-one usage records and three runtime-added models, gone
between one command and the next. Recovered only because the pre-migration
SQLite file was still there.
The guard is the database name. A test database has to say so, because a check
that could be satisfied by accident would not have stopped this one — the DSN
was entirely correct for what it had been opened for, and catastrophic for
what it was then used for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.
A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.
The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.
A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.
Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.
The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.
What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keys are stored hashed, so a migration that drops them cannot recreate them:
every person and every script needs a new one issued, and the old ones cannot
even be looked up to say which they were. `llmgw keys move-store` copies
instead.
Found by running it against the real cluster, twice, after the gateway had
already been stopped.
The first order came from the order the migrations create tables in, and
users is created in a later migration than api_keys even though api_keys
references it — so the copy died three tables in. The order is now checked by
a test that derives the foreign keys from the migrations rather than restating
them, so a new reference fails there instead of during somebody's migration.
Checked by putting the wrong order back and watching it fail.
Then key_claims. Its sealed column is emptied the moment a claim is used, so
every claimed row holds an empty blob in a NOT NULL column. An empty blob
scans as a typed nil slice and a genuine NULL as an untyped one — a
distinction this side can see and pgx cannot, since it encodes any nil slice
as NULL. Restored before the insert, which is the difference between a copy
that completes and one that stops four tables in.
One transaction per table rather than one for the whole copy: a single
transaction holding every row of a large usage table is how a migration runs
out of memory on the machine least able to spare it. A failure therefore
leaves a partial copy, which is why the destination must have been empty —
starting again is dropping and recreating it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens
SQLite and behaves exactly as it did.
One set of queries rather than one per database. The SQL here is almost
entirely portable — ON CONFLICT upserts work in both, and the only construct
that genuinely differs is the placeholder — so two copies of every query would
buy nothing and drift, one query at a time, discovered in production.
The rewrite is a wrapper around the handle rather than a change at each call
site. There are sixty of those and sixty chances to forget one; this way the
only route to the driver goes through the rewrite. Making it a wrapper found
six places that reached past it, including all three manual transactions and
two prepared statements — every one of which would have worked on SQLite and
failed on PostgreSQL, on a path nobody exercises before deploying.
The rewriter skips quoted sections. No query here contains a question mark in
a literal today; this is so that adding one is a test failure rather than a
statement that silently binds the wrong argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap list said a fetch "reports no progress". It does — the agent logs a
percentage as it downloads, with a comment saying a replica pulling a hundred
gigabytes and one that is wedged look identical from the control plane.
That comment describes the real gap, which is narrower and more useful than
the one written down: the progress exists, it just never leaves the node, so
the operator looking at ctl or the dashboard still cannot tell those two
apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by answering "where can we see this live". `ctl weights` was offering a
file for deletion that I had deleted two days earlier, and the node's disk
figure was the one it had when its agent started. Discover() is called at
registration and nowhere else, so both were frozen.
Disk is a fact about a moment, not about a machine. Sampled once, the figure
is wrong on exactly the node worth watching — the one busy filling up — and it
stays wrong until that agent restarts. The heartbeat already refreshes GPU
memory for the same reason, with a comment about noticing a foreign process
"after registration"; disk belonged beside it and I did not put it there.
The stale list cut both ways. A model fetched after registration was invisible
until a restart, so placement could not prefer the node that had just spent ten
minutes downloading it, and the disk check could not see the space it used.
Storage goes in every heartbeat, being a few numbers per filesystem. The file
list goes only when a fingerprint of paths and sizes changes, with a flag
saying the list is meant — without that, an empty list would be ambiguous
between "this node now holds nothing" and "nothing to report", and the second
reading would empty a node's inventory five seconds after it registered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of the four items were already built. A model falls through to its
fallbacks on no capacity as well as on a spent budget, and the per-request
routing profile header exists — it is ignored under authentication on purpose,
because a header that picks a routing profile is a header that promotes a
client into the top priority band.
Checked against the code rather than against memory, which is how both got
onto the list in the first place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SetThreadRetention existed and nothing called it, so the feature was
unreachable: a project could never opt in and every request carrying a thread
was refused.
A command rather than a configuration key, because it is a decision about data
rather than about deployment. Someone has to choose to keep what a team types,
and choosing it should look like an act rather than like a default nobody
read.
--retain-hours is required. Without a number the command would create exactly
the thing the design avoids: a conversation store with no expiry, holding
everything a team ever typed because nobody chose one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The request path half. A caller sets x-llmgw-thread and sends only the new
turn; llmgw puts the stored history in front of it, sends that to whichever
model is serving, and stores the exchange afterwards.
A header rather than a body field because some SDKs drop fields they do not
know and all of them let a caller set a header.
Capturing the reply is opt-in and bounded. This is the hottest path in the
system, so a request without a thread accumulates nothing at all, and one with
a thread stops accumulating at a cap rather than growing to it — a gateway
that kept every reply in memory would be one long answer away from trouble. A
reply is truncated on a rune boundary rather than dropped: a shortened turn
replays as a shortened turn, while nothing at all replays as the assistant
having said nothing, and invalid UTF-8 would break the next turn's body for
good rather than for one turn.
The streamed path sets the captured reply on every exit, not only the one that
looks like the end. A stream can also finish with the client gone or the
engine stalled, and what the assistant said up to that point is still what it
said.
A thread on a project that keeps no conversations is refused. The e2e test
asserted that and got a cheerful 200: the request was served, nothing was
stored, and the caller would have discovered it only when the model had no
idea what they were referring to — which reads as the model being broken. It
now asks whether storage is on before doing anything else.
History is spliced after the model is finally settled, because a budget
fallback may have changed it and trimming is measured against the context of
the model that will actually serve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The part that makes a thread work across models: the history goes in front of
what the caller just said, because a client using a thread sends the new turn
rather than the conversation again.
Trimming has three rules, and each is there because the alternative fails
quietly. The oldest stored turns go first, so a conversation that outgrew the
model now serving it stays usable — after a budget fallback that model was not
the caller's choice, and a permanently unusable thread would be a strange way
to punish them for running out of budget. The caller's own messages are never
dropped: answering something they did not ask is worse than refusing. And a
system message is kept wherever it sits, because losing one silently changes
what the model is for while it carries on answering.
A model that declares no context length has nothing trimmed. Guessing a limit
for it would drop turns for no reason and the caller would never learn why the
model forgot.
The body is rebuilt through a map so every field the client sent survives.
Forwarding fields llmgw does not understand is how engine-specific options
reach an engine at all, and a thread must not be what starts dropping them.
An empty history returns the body byte for byte, so a request without a thread
takes exactly the path it took before this existed.
Not yet wired to the request path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage half of continuing a thread on a different model than it started
on. Nothing uses it yet.
Until now llmgw stored no message content anywhere: usage rows carry token
counts, costs and latencies, never text, and even its own chat page keeps the
conversation in a JavaScript variable that a refresh discards. That property
is worth more than the feature, so it is the default and stays the default.
A project stores nothing until someone turns it on for that project.
Retention is a number of hours rather than a boolean, because a thread store
with no expiry is a transcript of everything a team ever typed, kept forever
because nobody chose a number. Turning it on means saying for how long. The
window measures idleness rather than age — expiring a conversation someone is
still having would be a strange way to keep that promise.
Threads are scoped to a project and the scope is part of every lookup, not a
check afterwards. Reading a thread from another project and reading one that
does not exist return the same error: two errors would let a caller learn
which ids are real in projects they cannot read. Writing across projects is
refused for the same reason in the other direction — append, then read your
own reply back.
Deleting a thread takes its messages with it, which the test checks by
counting rows rather than by trusting the cascade. Someone has to be able to
say "forget that", and a retention window is no answer to a message sent by
mistake.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context check ran against the model the caller asked for. The budget
fallback then swapped in a different one and nothing looked again, so a prompt
sized for a 32k model could be sent to a 512-token one. Measured before the
fix: 10002 prompt tokens delivered to a model declaring 512.
The caller sees that as a truncated reply or an engine error naming a model
they never chose, which is a worse outcome than the refusal the fallback
exists to avoid — it fails later and explains less.
Candidates that cannot hold the prompt are now skipped, and a fallback further
down the list that can hold it is used instead. When none fit, the request is
refused as it was before the fallback existed.
The history is deliberately not trimmed to make it fit. Dropping a caller's
messages to stay under a limit they never asked about would be a silent loss,
inside a request that was already being quietly moved to another model.
The tests drive the real request path, because a unit test on the fallback
would have asserted the same wrong thing the code did. Both were checked
against the unfixed code and fail there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.
Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl weights --collect` answered "this control plane cannot ask nodes to
delete files" against a real cluster, with the whole suite green.
The controller is handed a dispatcherRef rather than the agent service —
an indirection that exists to break a construction cycle, and one that
satisfies the dispatcher interface by forwarding each method by hand. The
controller reaches past that interface by type assertion to find capabilities
beyond scheduling, and an assertion against dispatcherRef fails for anything
nobody remembered to forward.
Nothing could have caught it below the wiring. The dispatchers the tests use
do not implement the capability either, so its absence there is the expected
case and looks identical to this bug. The test therefore asserts on the
production types themselves: that the forwarding dispatcher carries the
capability, and that the service it forwards to has one to carry. Checked by
deleting the method and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of the same problem: nothing stopped a model being placed on a node
that could not hold it, and nothing ever removed the weights of a model that
had been deleted from the catalog.
Placement now has a disk check, shaped exactly like the memory one it sits
beside, with one addition: it is skipped on a node that already holds the
weights. A node holding the file needs no room for it, and without that the
node best placed to run a model — because it already has it — is the one most
likely to be rejected, since holding the file is what used the space.
Reclaiming is the dangerous half, so the whole design is about what it will
not do. llmgw deletes only files it fetched itself, and that is a calculation
rather than a heuristic: the cache name is a hash of the source, so the set of
wanted files can be computed from the catalog exactly. Anything without that
name was put there by an operator — scp, a shared mount, a copy by hand — and
is none of this system's business however unreferenced it looks. A model that
took a week to copy must not be removable by a command someone ran to tidy up.
The node checks every path again before unlinking it, and that duplication is
the point. The control plane knows the catalog; the node knows its disk. A
control plane that is wrong, newer than the agent, or tampered with must not be
able to talk a node into deleting something, so the agent refuses anything
outside its own cache directory or without the cache name — including a path
that tries to traverse out of it.
Listing is the default and deleting takes --collect. The summary says "asked to
remove", because each node decides, and one that refuses a path it cannot prove
it fetched is this working rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space it returns in the same struct was thrown away. So the
only way to find out why a node was filling up was to ssh to it, which is what
this system exists to avoid.
Reported per filesystem rather than per configured directory. Free space
belongs to the disk, not the path: two model directories on one disk are one
pool of bytes, and counting them separately would let the control plane add
24 GB to 24 GB and place a 40 GB model on a node that has 24.
Shared filesystems are summed separately from local ones and left out of a
node's own total. Space on a filer belongs to everyone mounting it, so twenty
nodes would otherwise each report the same free terabyte, and a scheduler
would place twenty models into room for one.
Nothing reported means nothing is known, everywhere: an agent too old to send
storage and a node with no model directories both send nothing, and both must
read as unknown rather than as a full disk. `ctl nodes` and the dashboard show
a dash, the metric is absent rather than zero, and a missing directory — which
is what an unmounted filer looks like — is skipped rather than reported as
having no space.
Shown as free out of total, because the first is useless without the second:
8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root
partition, and only the second stops a model being placed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It had one verdict for a failed attempt — "This is the engine, not the model.
Choosing a bigger model will not help" — and that verdict was wrong, in a way
that sends someone off rebuilding llama.cpp when the fix is a one-line catalog
change.
Measured on the cluster. The engine renders the tool prompt correctly: the
<tools> block and the instruction to wrap the call in <tool_call> are both in
the prompt, which I confirmed through the engine's own /apply-template. The
coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call
in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser,
which looks for <tool_call>, has nothing to find and leaves it in the content.
qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated
tool_calls array. So it was the model all along, and the advice against trying
another one was the most expensive sentence in the output.
A failed attempt is now two verdicts, told apart by whether the reply used the
wrapper its template asked for. Wrapped and unparsed is genuinely the engine's
parser. Unwrapped is the model, and the fix is one trained for the format
rather than a bigger one from the same family.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It said "there is no distribution mechanism yet" and then, two paragraphs
later, documented the distribution mechanism. The known-gaps list — the part
someone reads at 2am — still said "no cache, no fetch, no progress". Fetching
landed in this cycle; both sentences were written before it and left standing.
node-onboarding.md carried the same claim.
What is actually still missing is narrower and worth naming precisely: a fetch
reports no progress, so a first start on a large model looks like a hang, and
every node fetches its own copy, so eight nodes on a new model are eight
downloads from one origin.
Also: the dashboard now shows who is using the cluster and what they are
running, which the sentence describing it did not mention; and one sentence
about idle timeouts had lost the words that made it a sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
My first fix was wrong about why the release could not be created. Gitea
materialises a release row for every pushed tag, so the lookup should have
found one; the create that followed hit a release that already existed.
The lookup failed because of how it read the id. `sed 's/.*"id":\([0-9]*\).*/\1/'`
is greedy, so `.*"id":` matches the last occurrence in the document, not the
first — "id" appears again inside the nested author object — and it does not
match `"id": 13` with a space after the colon at all. Checked against this
instance: for v0.5.0, whose id is 12, the old pattern returns 1.
It now greps the first "id", tolerates the whitespace, and re-reads the
release after creating rather than parsing the create response, so a 409
resolves to the release that was already there instead of to an error.
`set -e` was hiding the empty case: a command substitution in an assignment
fails the script, and grep exits non-zero when it matches nothing, so the
"no release yet" branch could never be reached — the job would simply stop.
Both extractions are guarded, and both paths are exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first v0.6.0 run built every binary, including the NVML one, and then
stopped at "could not find or create the release". Two things were wrong.
`releases: write` is not enough to create one. Creating a release can create
its tag, which is a code-unit operation; `contents` is the scope that covers
code and releases together.
And the step threw away the answer. It piped the response through sed looking
for an id and, finding none, reported only that it had failed — so a refusal,
a malformed request and an unreachable server all produced the same line. It
now prints the status code and the body, which is how the above was diagnosed
at all: by reading a log, not by knowing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and
was the detail behind it — phases 2 and 3, written while they were in progress
and never folded in when the release was cut — so it claimed as unreleased
work that had been shipped for two days. It is now a section of v0.5.0. The
status table said v0.5.0 was planned, which its own release notes disprove.
The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the
headline and the rest is what a cluster needs before anyone but its operator
is let near it.
Two things would have stopped the release workflow publishing anything:
Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on
the instance's default mode, and under the restricted default it is read-only
on releases. The job now asks for `releases: write`, which is correct under
either default. An upload that is refused now fails the job — without `-f`,
curl treats a 403 body as a successful transfer, so the release would end up
with some of its files, or none, and a green tick either way.
`workflow_dispatch` is offered but the upload assumed a tag. Fired from a
branch it would have created a release named "main".
README's status paragraph still said authentication, budgets and the control
plane were missing. It now says what is actually left, including the part
worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on
a machine that has any of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It is the only page a stranger sees, and until they have a key it is the
whole product: a stack of headings and shell snippets said "internal tool
you probably should not be here".
Now it opens with what this is and a diagram of why it exists — one address,
many GPUs, fabrics that cannot see each other — then the base URL as a
headline rather than a line of prose with a colon in it, the route to a key
as three cards, and the client examples as tabs instead of a pile.
The tabs are radio buttons and a stylesheet. No JavaScript decides what is on
the page, so someone who arrives with scripting off still gets every example;
the only script is the copy buttons, which exist only where the clipboard API
does and so cannot leave a dead control behind.
The page says exactly what it said before, including that it lists no models
and why. The diagram names no model, no node and no count: fabrics are drawn
because llmgw keeps them apart, not because this install has two.
Tests cover the parts that are easy to break later: every example lives in
the page itself rather than being fetched per tab, the reflected base URL —
now in five places rather than one — still cannot write markup, and the
diagram carries a label for anyone not looking at it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bars were wrapped in <div class="chart">, and .chart was already the
fixed-height box the time-series charts are drawn in: 120px, with a border
and a background. A chart that grows a row at a time does not fit in a cage,
so the SVG overflowed it and landed on the table underneath.
Renamed to .ranked, which is a wrapper rather than the mark itself, and
capped at the design width so a wide window scales the box instead of
inflating the 12px labels.
The test reads the class back out of the markup and checks the stylesheet
does not cap it, because the collision was invisible in both files on their
own — each rule was reasonable, and only the shared name was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.
Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.
Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.
The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.
Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.
A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The accounting layer has always exempted a model that costs nothing, with a
comment saying it is "what lets a key with no budget left still be served a
cheap fallback". That described a caller nobody had written: the budget was
checked once, against the requested model, and a refusal was the end of it,
so the exemption could never fire.
A caller that has run out is now offered the first model in its fallbacks it
can still afford. The allowlist still applies — running out of money is not
a way to reach a model a key was never allowed — and a model with no
fallback is refused exactly as before, or a budget would stop meaning
anything. Only a spent budget falls back: a rate limit is answered by
waiting a second, and moving the request elsewhere would spend someone
else's capacity to avoid a delay nobody asked to avoid.
Which uncovered a header that has never worked. x-llmgw-served-model was set
in finish, and finish runs after the proxy has written the response — so the
one mechanism for telling a client that something else answered has been
announcing it to nobody since it was written. Nothing failed, because the
only thing asserting it was a comment. It is now set before dispatch, for
the capacity fallback as well as this one.
Finding that took bisecting a test I had written wrong: it polled for a 429
that never came, because the budget had run out and the fallback had served
the request — the feature working, reported as the feature failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics were exported and the only way to look at them was to stand up
Prometheus beside the binary. For a project whose whole shape is one static
binary with no dependencies, that is the wrong default: the common case is
somebody wanting to know how full their cluster is, and that should not cost
a monitoring stack.
/admin now shows capacity per GPU class with a usage bar, free whole nodes
per fabric — the number that decides whether a replica spanning nodes can
land, since free cards on the wrong fabric are not capacity for a model that
needs this one — and which replicas are holding hardware nobody is asking
for. Nodes gain their memory and how many model files they can see.
Taken from the same Summary that ctl status uses rather than derived in the
page, because two implementations of the same arithmetic drift apart and the
one on the screen is the one people would believe.
A faulty card counts as neither used nor free, matching the scheduler.
Counting it free is how a cluster looks like it has room it does not, and a
page that disagreed with placement about that would be worse than no page.
Prometheus stays supported and is now the answer to a narrower question:
history, alerting, several clusters at once. The dashboard shows what is
true now and keeps nothing, which is the honest division between them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The metrics existed and nothing could record them. They shared a listener
with pprof, the management API and the dashboard, and pprof can dump the
process heap where API keys sit in plaintext — so a fleet could be monitored
or keep its heap private, and not both. That is not a choice anyone should
have to make, and it is why every previous answer to "how do I scrape this"
was a tunnel.
metrics_listen serves metrics and health and nothing else, so it can be
reached from wherever Prometheus lives. Metrics stay on the admin listener
too: adding a door is not the same as moving one, and anything already
scraping keeps working.
While there: admin_listen defaulted to :9090, every interface, with a
comment beside it saying not to expose it. A default that contradicts its
own documentation is a footgun, so it is loopback now and someone who wants
otherwise says so.
The handler is nil until the registry is built, and says so rather than
serving an empty page. Ordering has already caught this exact area once —
a collector registered after AdminHandler published nothing at all, silently
— and an empty scrape target looks like a working one.
deploy/monitoring/prometheus.yml has a starting configuration and the alerts
worth having: a node that has stopped reporting, a model that should be
serving and is not, a card kept out of placement by a fault, and a fleet
holding models nobody is asking for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On-demand loading is four components agreeing: the router notices nothing is
serving, the control plane decides the model may be started, the reconciler
places it, and the endpoint list carries it back. Each half was tested
alone, and the wiring between them is where this session's real bugs have
been — a metrics collector registered after the registry was built, an
adoption path that never health-checked — neither of which failed a unit
test.
Three tests through the real request path: one request loads a model that
was not running and is answered by it, an idle one is stopped with the node
still heartbeating so it is eviction rather than node loss, and an unknown
model still gets the unknown-model error, because on-demand must not turn
every typo into "loading".
Writing them was itself the check. The first version heartbeated once and
then slept through an UnreachableAfter of 300 milliseconds, so the node went
quiet and nothing was placed — which looks exactly like the feature failing.
A wrong assertion too: with an on-demand-only cluster and nothing loaded,
naming what is served means naming nothing, and "no model is currently being
served" is the honest answer rather than a lost error message.
This matters more now that there is no cluster to deploy to. Live
verification found several things today that no unit test would have, and
these tests are what has to replace it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.
Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.
Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.
Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.
No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.
The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.
Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.
A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.
An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.
Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.
And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.
A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
42fd0aa deployed and published nothing. RegisterMetrics appends to a slice
that AdminHandler reads when it constructs the registry, and I called it
after — a comment on the function says "called before AdminHandler, because
the registry is built there", written by me, immediately above the code
that did not.
Nothing failed. The build was clean, the tests passed, the endpoint
returned 200 with fourteen kilobytes of metrics, and not one of them was a
node. Only grepping the live endpoint for what was supposed to be there
showed it.
So there is now a test that a registered collector reaches /metrics, which
is the assertion that was missing: everything around it was tested and the
wiring between them was not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The router published twenty-two metric families about requests, endpoints,
admission and scoring — everything about traffic and nothing about the
machines carrying it. There was no way to graph GPU usage, watch a node go
unhealthy, see a fabric fill up, or answer "was the cluster full at the
time", which is the first question anyone asks after an incident. The
agents exported nothing at all.
Exported by the control plane rather than by each agent. It already holds
every number: hardware arrives at registration and state on each heartbeat,
so this is a map read away. One scrape target for a fleet of hundreds, no
Prometheus configuration per node, no firewall rule per machine, and
nothing new running on the nodes. On the same /metrics as the traffic
figures, because correlating "the cluster was full" with "requests were
queueing" should not need two systems.
A collector rather than gauges kept current by a ticker: the numbers are
derived from state that already exists, so computing them when asked is
simpler than maintaining a copy that can drift, and nothing runs between
scrapes.
GPU utilisation is new, and is the number people actually watch. Memory
used says a model is loaded; this says whether it is doing anything, and a
fleet at ninety percent occupancy and two percent utilisation is a fleet
holding models nobody is asking for.
Adding it caught two things. utilization.gpu goes last in the nvidia-smi
query and is optional, because a driver too old to know it returns a
shorter row — requiring it made an llmgw upgrade silently stop discovering
GPUs on an older node, which is far worse than not knowing how busy a card
is. And I gave the new protobuf field number 14, which ecc_errors already
had; buf lint is why that was a moment rather than a corrupted wire format.
Absent is kept distinct from zero throughout, for the same reason as
everywhere else here: an unreported utilisation shown as a flat zero reads
as an idle fleet rather than a missing measurement, and a replica nothing
has reported would be the most idle thing in the cluster.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default arrangement is the simple one: every node keeps its own copy.
So a source naming a remote file is now fetched before the engine is
launched, rather than left to fail as a missing path minutes into a load.
Eight shards of a tensor-parallel replica start at the same instant and
want the same file; they produce one download. An interrupted fetch resumes
instead of starting over, which matters at forty gigabytes, and the
download is written beside its target and renamed only when complete — a
partial file under the model's name would be found by the next scan,
reported as present, and handed to an engine that fails on it.
The exotic arrangements need no code, which is the point. Weights on a
Lustre mount or an image exported over the fabric are simply present, and
present means nothing is fetched. The escape hatch from a distribution
mechanism is a distribution mechanism that checks first.
Wiring it up caught a regression in my own design. A bare model id —
"Qwen/Qwen2.5-7B-Instruct" — is a documented source that vLLM and SGLang
resolve themselves, and treating every non-remote source as a local path
made every existing vLLM profile fail. Only an absolute or explicitly
relative path is now checked as a file; anything else is passed through
untouched. Three agent tests failed on the same change for a better reason:
they used "/models/x", and the new check is exactly the one that catches
that.
Whole repositories of shards are deliberately not fetched here. vLLM and
SGLang do that well already, and a second snapshot downloader would be work
spent arriving back where we started.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Storage will be different at every installation — Lustre, NFS over RDMA, an
image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes
it rather than managing it. An agent told which directories to look in
reports what it can see; how the files got there is nobody's business here,
and a node told nothing reports nothing and behaves exactly as every node
did before this existed.
That fact buys two things. Placement prefers a node that already holds the
model over one that must first move tens of gigabytes, as a preference and
never a filter: fetching them is a valid answer, and refusing to place would
turn a slow start into no start at all. And ctl status shows what each
fabric holds.
Per fabric because fabrics are not connected to one another. Weights in
ib-a are not weights in ib-b — the only path between them is the management
network — so a model seeded into one fabric leaves the rest of the cluster
unable to serve it without a slow copy, and nothing said so before now. One
shared filer counts the same as twenty local copies, because "can this
fabric serve it" is the question being asked.
Three distinctions the tests hold to, because each is a way to get this
quietly wrong. A node that was never asked to look must rank with one that
looked and has nothing, or enabling the feature on a single node would
drain every other. An empty directory is not weights, because a
half-finished download looks exactly like one and would send placement to a
node whose engine then fails to load. And local storage is never mistaken
for shared: a filer read as local costs an unnecessary copy, while local
read as shared would have the cluster believe a model is everywhere when it
is on one node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reading the tool's own output against the real fleet: it chose q8_0 as the
best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF
quantisation that only llama.cpp serves, and from a file rather than a
repository — so following that entry loads full-precision weights, which
the same output had just listed as not fitting one card. The planner would
have caused the out-of-memory it exists to prevent, from a recommendation
that looked right.
Choosing a precision and choosing an engine turn out to be one decision, so
a Precision now carries the engine that serves it, the quantisation flag
where one is needed, and whether it lives in a file. A GGUF fit prints a
path and says to download it; a vLLM fit prints the repository and passes
--quantization so the engine does not quietly load bf16.
The arithmetic was right and the recommendation was wrong, which is the
failure mode worth having tests for: everything reported a number, and the
numbers agreed with each other.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
