The count is scaling.min, and until now the only way to change it was to edit
server.yaml on the host and restart — or, for a model added at runtime, to
remove it and add it back, which means having its whole definition to hand.
`llmgw ctl scale <model> --min N` instead. Only the scaling block moves:
everything else is read back from what was stored and written out untouched, so
changing a number cannot quietly become a redefinition.
The catalog is rebuilt and validated before anything is written, as in Add. A
max below a min is refused with the stored definition untouched — persisting a
rejected change would come back on the next restart as a catalog that does not
build.
The answer reports the scaling in force rather than the request. The catalog
fills in a max that was left out, and a caller echoing its own request would
print a number the cluster is not using.
A model declared in configuration is refused, for the reason Remove already
refuses one: the YAML is read on every start, so the new count would hold until
the next restart and then silently revert. A cluster disagreeing with its own
config file is worse than an error saying where to edit.
The help says what --max does, because it is not what it looks like: a model
with a minimum runs exactly that many and nothing scales it up under load, and
an on-demand model loads one replica however high its max.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Control plane mode was inferred from the config file alone: a model with
profiles meant "control plane", nothing meant "router only". So a deployment
that manages its catalog at runtime could not start at all — the file declares
no models, the process refuses for want of anything to serve, and the stored
models it would have loaded are never read. A cluster unmanageable at runtime
because it is managed at runtime.
control.enabled says so outright. AgentListen cannot carry the signal: it has a
default, so every deployment would silently become a control plane.
The inference stays, so nothing already working has to add a flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/admin/mirror, behind the same admin session as the other admin pages. What is
held, what is arriving with its byte count, what failed and why; start a fetch,
retry a failed one, remove a version.
Not mounted at all without a mirror configured. The right page for a deployment
with no mirror is no page, rather than an empty one pointing at nothing.
It reloads itself only while something is downloading. A reload at any other
moment takes with it whatever is half-typed into the fetch form, and there is
nothing to watch when nothing is moving.
A failed row offers Retry, not Remove. A fetch that failed left nothing on disk,
so the remove could only answer "the mirror holds nothing for this" — a button
whose single outcome is a confusing error. Starting the same download again is
what somebody looking at that row wants, and the mirror allows it: only a
download still running is refused.
An unreachable mirror is said outright instead of rendering as an empty table.
The two look nearly identical and mean opposite things: one is a mirror waiting
to be filled, the other is a machine to go and look at.
Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a
call site get swapped silently and are discovered by somebody clicking the
wrong link.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All() preferred the disk entry wholesale, and a directory listing has no
checksum in it. So the sha256 a fetch computed was reported while the bytes
were still arriving and vanished the moment the file landed — it disappeared
exactly when it became verifiable, which is the one moment it is for.
It is the field an operator compares against what upstream published. A mirror
is the single place in this system where a corrupted file is copied to every
node instead of being caught once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror has had a management API since it was built, and the only thing
driving it was curl on the mirror's own machine. This is the other half: a
client, so the operations belong to whatever has a reason to perform them.
The mirror's refusals travel as their own sentence rather than a status code.
Putting the reason in the body was deliberate on the server side, and a client
that kept only the number would undo it — "the mirror said 400" sends somebody
to the wrong machine, "the source URL answered 404" sends them to the right one.
NextProtos is cleared for the same reason the mirror's server clears it. That
configuration is shaped for the agent stream, which pins h2; a transport
carrying a pinned protocol it is not set up to speak negotiates h2 and then
talks HTTP/1.1 over it, and fails with an error mentioning neither ALPN nor
the mirror.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page described nginx and a curl loop, which was accurate when the mirror
was a directory somebody kept by hand. It is a subcommand now, so it says so:
the two listeners and why they are two, the API and what each endpoint answers,
and the certificates coming from the authority the fleet already has.
nginx stays, further down. It serves a directory perfectly well and anyone
already running it should not be told to install something to replace it.
The systemd unit spells out --base in a comment because it is the flag that
goes wrong quietly: it ends up in catalog entries, and a wrong one makes
`mirror unused` call files unreferenced while they are in use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test started a sidecar and an engine, each a shell appending its name to a
file, and demanded the sidecar's line come first. The supervisor spawns the
sidecar first, but spawning is not sequencing: without WaitForLeaderPort
nothing holds the engine back, so two shells race and the scheduler picks the
winner. It failed under full-suite load, and lately on its own.
The wait that is a real guarantee already has its own test, so the fix is to
check what this one is actually for — that both processes run under one shard,
that the shard's pid is the engine's rather than the sidecar's, and that their
output is distinguishable in a single log tail. Renamed to say so.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror holds the weights, so the mirror does the downloading. A control
plane that had to pull the bytes itself and push them here would carry every
model twice over the network, through the one machine least able to spare the
bandwidth.
Two listeners, and the split is the point. Weights stay on plain HTTP where
every node can reach them: that path is eight nodes pulling forty gigabytes,
and putting a client certificate in front of it would mean re-certifying every
agent's fetcher in order to change how models are administered. Management gets
its own port, which can be given the fleet's CA — the same tlsconf the agent
stream uses, so this is a configuration rather than a new idea.
A fetch answers 202, not 200. The bytes are not here yet, and a caller that
read it as "done" would point a catalog at a file still arriving, which every
node would then copy. Progress is asked for rather than waited on, because no
client between here and the operator will hold a connection open for an hour.
A failed fetch is remembered with its reason. Forgetting it would leave the
control plane waiting on a model that is never coming, with nothing to say why.
Status falls back to reading the disk, so a restart does not report a model
that is plainly there as unknown.
Removal takes the whole version: a model split across files is useless with one
of them missing, and half a version on disk looks present and will not load. It
refuses while a download is running, because the fetch would carry on writing
and the file would reappear, looking like the delete had silently failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mirror was a directory and a discipline written down in a README. This
makes the three operations that matter into commands, on the mirror host,
where `llmgw keys` already sets the precedent of operator tooling that works
on files rather than through the API.
The gateway still never writes to the mirror. That property was chosen
deliberately — a mirror writable by the thing reading from it is a way to
replace a model with one every node then fetches without complaint — and a
person running a command does not take it away. `serve` states it rather than
inheriting it, so a handler added later cannot quietly make the mirror
writable.
`add` downloads to a temporary name and renames, for the reason the agent's
fetcher does: a mirror is copied to every node, so a partial file here is a
partial file everywhere. It verifies the length the server promised, reports a
checksum — the mirror is the one place a corrupt file reaches every node
rather than being caught once — and refuses to replace an existing version,
which is the whole point of the layout.
`unused` refuses to answer when it cannot read the catalog. An empty catalog
and an unreachable one produce the same list, everything, and the obvious next
step after that list is deletion. It compares by path rather than by spelling,
because a base typed with a different scheme or a trailing slash would
otherwise report a file in use as unused.
`remove` checks the catalog first and says to retire the entry before deleting
the file, because a node that has to refetch a deleted file cannot, and that
failure arrives later, on the node, looking like something else.
Verified end to end: Range requests answer 206, which is what makes an
interrupted fetch on a node resume; writes answer 405; a traversal attempt
answers 404.
Note: internal/agent/supervisor's TestSidecarRunsBeforeEngine fails under full
suite load on main as well as here — it asserts that one shell's echo lands
before another's, which the supervisor does not promise. Everything else
passes. Left alone rather than fixed in a commit about mirrors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"Set up a local mirror" was advice. This is the nginx config, the directory
layout and the reason the layout has a date in it — a repository that changes
upstream changes what you are running, and the first sign is a prompt that
worked last week answering differently this week.
It says what the mirror is not, because both mistakes are easy. It is not a
caching proxy: nothing populates it on demand, deliberately, since a mirror
that reaches the internet at the least convenient moment fails in a way that
looks like llmgw being broken. And it is not needed at all if you already run
a shared filesystem — put the models there, use a path, and nothing is fetched.
It also says when a mirror stops being enough, and that llmgw is positioned to
distribute between nodes because it already knows which of them hold which
weights. That is not built, and the page says so rather than implying it is
coming.
Every behaviour it relies on was checked against the source: an http:// source
is fetched and a path is not, a killed download resumes, a partial file is
never visible under the model's name, and a shared filesystem is not counted
once per node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Neither had a page. "We have GPUs, now what" was left to the reader, and what
llmgw stores was scattered across four documents and a changelog — which is
the same as not being written down when somebody is doing a security review.
docs/data.md describes the behaviour and leaves the legal mapping to whoever
holds the obligation. Every claim in it was checked against the source, and it
names where to look for each: usage rows carry no text field, keys are stored
hashed, conversations are off until a named project turns them on and cannot be
turned on without a retention window.
It also states the egress exactly, which took reading every outbound call to
say honestly: nothing on the serving path leaves the cluster, and the two
things that can reach outside — fetching weights, and ctl plan reading a model's
architecture — are both started by an operator. Weight fetching points wherever
you tell it, including an internal mirror.
docs/choosing-models.md is the opinion llmgw itself does not have. Three jobs
rather than one model, with the third — embeddings — being the one that gets
forgotten and the one that decides whether the deployment can answer questions
about your company or only questions in general.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The choice is unchanged and now better argued: a deployment does not announce
which language its notes are in, so any stemmer picked here is the wrong one
somewhere — and applying the wrong one does not fail loudly, it quietly stops
finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.
Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.
So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.
There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The agent has always logged a percentage as it fetches weights, with a comment
saying a replica pulling a hundred gigabytes and one that is wedged look
identical from the control plane. They still did: the log is on the node, and
the node is not where anyone is looking when a replica has been starting for
ten minutes.
Progress now rides on the shard status and appears under the replica in `ctl
replicas` — a line rather than a column, because most replicas are not
downloading anything and a column would be empty for all of them.
It is deliberately not a state transition. Treating each chunk as one would
re-mark the shard as starting on every report and reset the timer that decides
when a slow start has become a failed one, so an hour-long download would keep
a doomed replica alive indefinitely. It arrives on its own method for that
reason, and a real transition clears it — otherwise a finished download sits in
the listing at ninety-nine percent under a shard that has been serving for an
hour.
Watchers are keyed by source rather than by shard, because a fetch is
single-flight: eight shards starting together produce one download and all
eight are waiting on it.
No percentage is invented when the server sent no length. One that stops moving
is worse than a byte count that keeps rising, so the field is absent and the
listing says "so far".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was no reference. Four narrative documents explain how to install and
operate llmgw, and the only way to learn that `lease_ttl` exists was to read
the one that happened to mention it — or the struct.
Written by hand it would have been wrong within a fortnight. Six settings
arrived in the last two weeks alone: store, servers, lease_ttl, the two
retention windows, min_disk_gb. A reference nobody maintains is worse than
none, because it is the same list with the setting you came to look up quietly
missing from it.
So it is derived from the structs, with the field comments as the descriptions
— which means the only way for it to be wrong is for the code to be wrong, and
it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys.
Two tests keep that true after everyone has forgotten this exists. One
regenerates and compares, so adding a setting without regenerating is a
failure rather than an omission — checked by adding one and watching it fail.
The other walks the tree for koanf tags and fails on a configuration file
that is in neither the source list nor the short list of structs an operator
never writes, because a list is a thing to forget.
The generator finds the repository from its own path rather than the working
directory, so `go generate ./...` produces the same document from anywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five nodes fit in a table. Fifty do not, and this is built for people who
already have racks of GPUs — so the flags are how a question gets asked:
which nodes on this fabric, which of this GPU class, which are not ready,
which replicas have a shard on this machine.
Filtered here rather than in the API, so --json is narrowed the same way the
table is. A server-side filter would be two implementations of one predicate,
and the JSON one is the one nobody would notice had drifted.
An empty result says which case it is. A filter that matched nothing and an
empty fleet call for opposite next steps — one is a cluster to look at, the
other is a word to correct — and printing "no nodes" for both makes a mistyped
fabric name read as an outage.
A state matches on prefix because states are few and distinct and somebody
scanning a fleet types quickly. A pool matches either half, because it is
fabric/class and both are worth narrowing by. A replica is found from any of
its nodes, since one spanning two machines is on both.
--label takes a bare key as "carries this at all", which is how you ask for
every node with a role without caring which. The flag's help said so before
the parser did; the test caught them disagreeing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The store stops being a file on one machine, which is what stood between this
and more than one control plane.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three seconds to move leadership, six for every agent to arrive, and every
replica adopted rather than restarted. Also what it did not do: the public
address returned 502 while the standby served correctly on its own port,
because nothing in front of the two moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second half of leadership. A standby that refuses to place but still holds
an agent's stream is worse than useless: commands travel down the stream the
agent opened, so the leader's placements fail with "no agent connected" for a
node that is plainly connected.
So a standby refuses the stream, before reading the opening message, and says
why. And an agent takes a list rather than an address, trying each in turn — a
standby refusing, a stopped control plane refusing the connection and a crashed
one never answering are the same thing from a node, and the answer to all three
is somewhere else.
`server` and `servers` are both accepted, with `server` first when both are
given: every agent configuration written before there could be a second control
plane names one address, and an operator adding a standby expects the one
already working to stay preferred. With a single address the behaviour is
exactly what it was, including the backoff — which is skipped only while there
is another address that has not just been tried, because the delay exists to
stop hammering one control plane and the next one has not been hammered.
Removed a check that asked whether one field was empty, which is how the first
run of these tests failed: the list was populated and `server` was not, and an
agent configured entirely correctly was refused by its own validation.
The tests run a real agent against a real service over h2c. The first attempt
used a plain HTTP server, and the refusal arrived as an HTTP/2 framing error
that said nothing about why — which is exactly what an operator would have seen.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
These TRUNCATE every table before each test. Pointed at a deployment that is a
silent and complete data loss — every key, every user, every conversation.
Not hypothetical. I did it: an SSH tunnel to the production database had been
opened minutes earlier for a perfectly good reason, and the same DSN was then
handed to the test suite. Twenty-one users, nine API keys, eighteen sessions,
four hundred and sixty-one usage records and three runtime-added models, gone
between one command and the next. Recovered only because the pre-migration
SQLite file was still there.
The guard is the database name. A test database has to say so, because a check
that could be satisfied by accident would not have stopped this one — the DSN
was entirely correct for what it had been opened for, and catastrophic for
what it was then used for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.
A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.
The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.
A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.
Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.
The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.
What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section leads with why SQLite is still the default rather than with how to
leave it: one file, no server, and a request path that barely touches it. Two
things change that, and both are recent — conversations and notes read and
write on every turn, and a second control plane is impossible while the store
is a file on one machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keys are stored hashed, so a migration that drops them cannot recreate them:
every person and every script needs a new one issued, and the old ones cannot
even be looked up to say which they were. `llmgw keys move-store` copies
instead.
Found by running it against the real cluster, twice, after the gateway had
already been stopped.
The first order came from the order the migrations create tables in, and
users is created in a later migration than api_keys even though api_keys
references it — so the copy died three tables in. The order is now checked by
a test that derives the foreign keys from the migrations rather than restating
them, so a new reference fails there instead of during somebody's migration.
Checked by putting the wrong order back and watching it fail.
Then key_claims. Its sealed column is emptied the moment a claim is used, so
every claimed row holds an empty blob in a NOT NULL column. An empty blob
scans as a typed nil slice and a genuine NULL as an untyped one — a
distinction this side can see and pgx cannot, since it encodes any nil slice
as NULL. Restored before the insert, which is the difference between a copy
that completes and one that stops four tables in.
One transaction per table rather than one for the whole copy: a single
transaction holding every row of a large usage table is how a migration runs
out of memory on the machine least able to spare it. A failure therefore
leaves a partial copy, which is why the destination must have been empty —
starting again is dropping and recreating it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Groundwork for PostgreSQL, with no PostgreSQL in it yet: the store still opens
SQLite and behaves exactly as it did.
One set of queries rather than one per database. The SQL here is almost
entirely portable — ON CONFLICT upserts work in both, and the only construct
that genuinely differs is the placeholder — so two copies of every query would
buy nothing and drift, one query at a time, discovered in production.
The rewrite is a wrapper around the handle rather than a change at each call
site. There are sixty of those and sixty chances to forget one; this way the
only route to the driver goes through the rewrite. Making it a wrapper found
six places that reached past it, including all three manual transactions and
two prepared statements — every one of which would have worked on SQLite and
failed on PostgreSQL, on a path nobody exercises before deploying.
The rewriter skips quoted sections. No query here contains a question mark in
a literal today; this is so that adding one is a test failure rather than a
statement that silently binds the wrong argument.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gap list said a fetch "reports no progress". It does — the agent logs a
percentage as it downloads, with a comment saying a replica pulling a hundred
gigabytes and one that is wedged look identical from the control plane.
That comment describes the real gap, which is narrower and more useful than
the one written down: the progress exists, it just never leaves the node, so
the operator looking at ctl or the dashboard still cannot tell those two
apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen commits since v0.6.0, and not a phase of the roadmap — which is why
the table leaves the phase column empty rather than inventing one. What landed
is what a cluster needs to be lived with rather than demonstrated: somewhere
to keep a conversation, somewhere to keep a note, and an honest answer about
the disk underneath both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by answering "where can we see this live". `ctl weights` was offering a
file for deletion that I had deleted two days earlier, and the node's disk
figure was the one it had when its agent started. Discover() is called at
registration and nowhere else, so both were frozen.
Disk is a fact about a moment, not about a machine. Sampled once, the figure
is wrong on exactly the node worth watching — the one busy filling up — and it
stays wrong until that agent restarts. The heartbeat already refreshes GPU
memory for the same reason, with a comment about noticing a foreign process
"after registration"; disk belonged beside it and I did not put it there.
The stale list cut both ways. A model fetched after registration was invisible
until a restart, so placement could not prefer the node that had just spent ten
minutes downloading it, and the disk check could not see the space it used.
Storage goes in every heartbeat, being a few numbers per filesystem. The file
list goes only when a fingerprint of paths and sizes changes, with a flag
saying the list is meant — without that, an empty list would be ambiguous
between "this node now holds nothing" and "nothing to report", and the second
reading would empty a node's inventory five seconds after it registered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of the four items were already built. A model falls through to its
fallbacks on no capacity as well as on a spent budget, and the per-request
routing profile header exists — it is ignored under authentication on purpose,
because a header that picks a routing profile is a header that promotes a
client into the top priority band.
Checked against the code rather than against memory, which is how both got
onto the list in the first place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The section opens on the default rather than on the feature, because the
default is the part someone deploying this needs to be sure of: nothing anyone
types is stored until a project asks, and the command to ask is an act rather
than a setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SetThreadRetention existed and nothing called it, so the feature was
unreachable: a project could never opt in and every request carrying a thread
was refused.
A command rather than a configuration key, because it is a decision about data
rather than about deployment. Someone has to choose to keep what a team types,
and choosing it should look like an act rather than like a default nobody
read.
--retain-hours is required. Without a number the command would create exactly
the thing the design avoids: a conversation store with no expiry, holding
everything a team ever typed because nobody chose one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of keeping the promise that storing makes.
DELETE /v1/threads/{id} is a client endpoint, not an admin one: the person who
said the thing should be able to unsay it, and making them ask an operator
would mean the operator reads it first. Deleting a thread in another project
answers exactly as deleting one that never existed does — two answers would
let a caller map which ids are real in projects they cannot read, and this one
would do it by trying to destroy them.
And something now runs the pruner. Without it "we keep conversations for 24
hours" was a comment rather than a promise: the rows would accumulate and the
number a project chose would describe nothing. Hourly, and once at startup,
because a gateway that has been down for a week should not wait another hour
before honouring a window that expired while it was off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The request path half. A caller sets x-llmgw-thread and sends only the new
turn; llmgw puts the stored history in front of it, sends that to whichever
model is serving, and stores the exchange afterwards.
A header rather than a body field because some SDKs drop fields they do not
know and all of them let a caller set a header.
Capturing the reply is opt-in and bounded. This is the hottest path in the
system, so a request without a thread accumulates nothing at all, and one with
a thread stops accumulating at a cap rather than growing to it — a gateway
that kept every reply in memory would be one long answer away from trouble. A
reply is truncated on a rune boundary rather than dropped: a shortened turn
replays as a shortened turn, while nothing at all replays as the assistant
having said nothing, and invalid UTF-8 would break the next turn's body for
good rather than for one turn.
The streamed path sets the captured reply on every exit, not only the one that
looks like the end. A stream can also finish with the client gone or the
engine stalled, and what the assistant said up to that point is still what it
said.
A thread on a project that keeps no conversations is refused. The e2e test
asserted that and got a cheerful 200: the request was served, nothing was
stored, and the caller would have discovered it only when the model had no
idea what they were referring to — which reads as the model being broken. It
now asks whether storage is on before doing anything else.
History is spliced after the model is finally settled, because a budget
fallback may have changed it and trimming is measured against the context of
the model that will actually serve.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The part that makes a thread work across models: the history goes in front of
what the caller just said, because a client using a thread sends the new turn
rather than the conversation again.
Trimming has three rules, and each is there because the alternative fails
quietly. The oldest stored turns go first, so a conversation that outgrew the
model now serving it stays usable — after a budget fallback that model was not
the caller's choice, and a permanently unusable thread would be a strange way
to punish them for running out of budget. The caller's own messages are never
dropped: answering something they did not ask is worse than refusing. And a
system message is kept wherever it sits, because losing one silently changes
what the model is for while it carries on answering.
A model that declares no context length has nothing trimmed. Guessing a limit
for it would drop turns for no reason and the caller would never learn why the
model forgot.
The body is rebuilt through a map so every field the client sent survives.
Forwarding fields llmgw does not understand is how engine-specific options
reach an engine at all, and a thread must not be what starts dropping them.
An empty history returns the body byte for byte, so a request without a thread
takes exactly the path it took before this existed.
Not yet wired to the request path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage half of continuing a thread on a different model than it started
on. Nothing uses it yet.
Until now llmgw stored no message content anywhere: usage rows carry token
counts, costs and latencies, never text, and even its own chat page keeps the
conversation in a JavaScript variable that a refresh discards. That property
is worth more than the feature, so it is the default and stays the default.
A project stores nothing until someone turns it on for that project.
Retention is a number of hours rather than a boolean, because a thread store
with no expiry is a transcript of everything a team ever typed, kept forever
because nobody chose a number. Turning it on means saying for how long. The
window measures idleness rather than age — expiring a conversation someone is
still having would be a strange way to keep that promise.
Threads are scoped to a project and the scope is part of every lookup, not a
check afterwards. Reading a thread from another project and reading one that
does not exist return the same error: two errors would let a caller learn
which ids are real in projects they cannot read. Writing across projects is
refused for the same reason in the other direction — append, then read your
own reply back.
Deleting a thread takes its messages with it, which the test checks by
counting rows rather than by trusting the cascade. Someone has to be able to
say "forget that", and a retention window is no answer to a message sent by
mistake.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context check ran against the model the caller asked for. The budget
fallback then swapped in a different one and nothing looked again, so a prompt
sized for a 32k model could be sent to a 512-token one. Measured before the
fix: 10002 prompt tokens delivered to a model declaring 512.
The caller sees that as a truncated reply or an engine error naming a model
they never chose, which is a worse outcome than the refusal the fallback
exists to avoid — it fails later and explains less.
Candidates that cannot hold the prompt are now skipped, and a fallback further
down the list that can hold it is used instead. When none fit, the request is
refused as it was before the fallback existed.
The history is deliberately not trimmed to make it fit. Dropping a caller's
messages to stay under a limit they never asked about would be a silent loss,
inside a request that was already being quietly moved to another model.
The tests drive the real request path, because a unit test on the fallback
would have asserted the same wrong thing the code did. Both were checked
against the unfixed code and fail there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A command that deletes data needs its limits written down more than its
syntax. The section says plainly that llmgw only takes back files it fetched
itself, and then says the uncomfortable half out loud: that this means it will
not reclaim the storage most likely to be filling your disk, because weights
you placed yourself are yours to remove.
Also records the dash in the DISK column as "not asked" rather than "no
space", which is the reading that matters when a node will not schedule.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl weights --collect` answered "this control plane cannot ask nodes to
delete files" against a real cluster, with the whole suite green.
The controller is handed a dispatcherRef rather than the agent service —
an indirection that exists to break a construction cycle, and one that
satisfies the dispatcher interface by forwarding each method by hand. The
controller reaches past that interface by type assertion to find capabilities
beyond scheduling, and an assertion against dispatcherRef fails for anything
nobody remembered to forward.
Nothing could have caught it below the wiring. The dispatchers the tests use
do not implement the capability either, so its absence there is the expected
case and looks identical to this bug. The test therefore asserts on the
production types themselves: that the forwarding dispatcher carries the
capability, and that the service it forwards to has one to carry. Checked by
deleting the method and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two halves of the same problem: nothing stopped a model being placed on a node
that could not hold it, and nothing ever removed the weights of a model that
had been deleted from the catalog.
Placement now has a disk check, shaped exactly like the memory one it sits
beside, with one addition: it is skipped on a node that already holds the
weights. A node holding the file needs no room for it, and without that the
node best placed to run a model — because it already has it — is the one most
likely to be rejected, since holding the file is what used the space.
Reclaiming is the dangerous half, so the whole design is about what it will
not do. llmgw deletes only files it fetched itself, and that is a calculation
rather than a heuristic: the cache name is a hash of the source, so the set of
wanted files can be computed from the catalog exactly. Anything without that
name was put there by an operator — scp, a shared mount, a copy by hand — and
is none of this system's business however unreferenced it looks. A model that
took a week to copy must not be removable by a command someone ran to tidy up.
The node checks every path again before unlinking it, and that duplication is
the point. The control plane knows the catalog; the node knows its disk. A
control plane that is wrong, newer than the agent, or tampered with must not be
able to talk a node into deleting something, so the agent refuses anything
outside its own cache directory or without the cache name — including a path
that tries to traverse out of it.
Listing is the default and deleting takes --collect. The summary says "asked to
remove", because each node decides, and one that refuses a path it cannot prove
it fetched is this working rather than failing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space it returns in the same struct was thrown away. So the
only way to find out why a node was filling up was to ssh to it, which is what
this system exists to avoid.
Reported per filesystem rather than per configured directory. Free space
belongs to the disk, not the path: two model directories on one disk are one
pool of bytes, and counting them separately would let the control plane add
24 GB to 24 GB and place a 40 GB model on a node that has 24.
Shared filesystems are summed separately from local ones and left out of a
node's own total. Space on a filer belongs to everyone mounting it, so twenty
nodes would otherwise each report the same free terabyte, and a scheduler
would place twenty models into room for one.
Nothing reported means nothing is known, everywhere: an agent too old to send
storage and a node with no model directories both send nothing, and both must
read as unknown rather than as a full disk. `ctl nodes` and the dashboard show
a dash, the metric is absent rather than zero, and a missing directory — which
is what an unmounted filer looks like — is skipped rather than reported as
having no space.
Shown as free out of total, because the first is useless without the second:
8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root
partition, and only the second stops a model being placed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It had one verdict for a failed attempt — "This is the engine, not the model.
Choosing a bigger model will not help" — and that verdict was wrong, in a way
that sends someone off rebuilding llama.cpp when the fix is a one-line catalog
change.
Measured on the cluster. The engine renders the tool prompt correctly: the
<tools> block and the instruction to wrap the call in <tool_call> are both in
the prompt, which I confirmed through the engine's own /apply-template. The
coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call
in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser,
which looks for <tool_call>, has nothing to find and leaves it in the content.
qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated
tool_calls array. So it was the model all along, and the advice against trying
another one was the most expensive sentence in the output.
A failed attempt is now two verdicts, told apart by whether the reply used the
wrapper its template asked for. Wrapped and unparsed is genuinely the engine's
parser. Unwrapped is the model, and the fix is one trained for the format
rather than a bigger one from the same family.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It said "there is no distribution mechanism yet" and then, two paragraphs
later, documented the distribution mechanism. The known-gaps list — the part
someone reads at 2am — still said "no cache, no fetch, no progress". Fetching
landed in this cycle; both sentences were written before it and left standing.
node-onboarding.md carried the same claim.
What is actually still missing is narrower and worth naming precisely: a fetch
reports no progress, so a first start on a large model looks like a hang, and
every node fetches its own copy, so eight nodes on a new model are eight
downloads from one origin.
Also: the dashboard now shows who is using the cluster and what they are
running, which the sentence describing it did not mention; and one sentence
about idle timeouts had lost the words that made it a sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
My first fix was wrong about why the release could not be created. Gitea
materialises a release row for every pushed tag, so the lookup should have
found one; the create that followed hit a release that already existed.
The lookup failed because of how it read the id. `sed 's/.*"id":\([0-9]*\).*/\1/'`
is greedy, so `.*"id":` matches the last occurrence in the document, not the
first — "id" appears again inside the nested author object — and it does not
match `"id": 13` with a space after the colon at all. Checked against this
instance: for v0.5.0, whose id is 12, the old pattern returns 1.
It now greps the first "id", tolerates the whitespace, and re-reads the
release after creating rather than parsing the create response, so a 409
resolves to the release that was already there instead of to an error.
`set -e` was hiding the empty case: a command substitution in an assignment
fails the script, and grep exits non-zero when it matches nothing, so the
"no release yet" branch could never be reached — the job would simply stop.
Both extractions are guarded, and both paths are exercised.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first v0.6.0 run built every binary, including the NVML one, and then
stopped at "could not find or create the release". Two things were wrong.
`releases: write` is not enough to create one. Creating a release can create
its tag, which is a code-unit operation; `contents` is the scope that covers
code and releases together.
And the step threw away the answer. It piped the response through sed looking
for an id and, finding none, reported only that it had failed — so a refusal,
a malformed request and an unreachable server all produced the same line. It
now prints the status code and the body, which is how the above was diagnosed
at all: by reading a log, not by knowing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and
was the detail behind it — phases 2 and 3, written while they were in progress
and never folded in when the release was cut — so it claimed as unreleased
work that had been shipped for two days. It is now a section of v0.5.0. The
status table said v0.5.0 was planned, which its own release notes disprove.
The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the
headline and the rest is what a cluster needs before anyone but its operator
is let near it.
Two things would have stopped the release workflow publishing anything:
Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on
the instance's default mode, and under the restricted default it is read-only
on releases. The job now asks for `releases: write`, which is correct under
either default. An upload that is refused now fails the job — without `-f`,
curl treats a 403 body as a successful transfer, so the release would end up
with some of its files, or none, and a green tick either way.
`workflow_dispatch` is offered but the upload assumed a tag. Fired from a
branch it would have created a release named "main".
README's status paragraph still said authentication, budgets and the control
plane were missing. It now says what is actually left, including the part
worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on
a machine that has any of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It is the only page a stranger sees, and until they have a key it is the
whole product: a stack of headings and shell snippets said "internal tool
you probably should not be here".
Now it opens with what this is and a diagram of why it exists — one address,
many GPUs, fabrics that cannot see each other — then the base URL as a
headline rather than a line of prose with a colon in it, the route to a key
as three cards, and the client examples as tabs instead of a pile.
The tabs are radio buttons and a stylesheet. No JavaScript decides what is on
the page, so someone who arrives with scripting off still gets every example;
the only script is the copy buttons, which exist only where the clipboard API
does and so cannot leave a dead control behind.
The page says exactly what it said before, including that it lists no models
and why. The diagram names no model, no node and no count: fabrics are drawn
because llmgw keeps them apart, not because this install has two.
Tests cover the parts that are easy to break later: every example lives in
the page itself rather than being fetched per tab, the reflected base URL —
now in five places rather than one — still cannot write markup, and the
diagram carries a label for anyone not looking at it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bars were wrapped in <div class="chart">, and .chart was already the
fixed-height box the time-series charts are drawn in: 120px, with a border
and a background. A chart that grows a row at a time does not fit in a cage,
so the SVG overflowed it and landed on the table underneath.
Renamed to .ranked, which is a wrapper rather than the mark itself, and
capped at the design width so a wide window scales the box instead of
inflating the 12px labels.
The test reads the class back out of the markup and checks the stylesheet
does not cap it, because the collision was invisible in both files on their
own — each rule was reasonable, and only the shared name was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard could say what the cluster was made of and not who was
spending it. Usage was per key, which is the right unit for revoking
something and the wrong one for the question actually being asked: one
person holds a laptop key, a CI key and a notebook key, and adding three
rows up by eye is a chore that gets skipped.
Per person now, joined through the keys they own, and per model — counted
by the model that answered rather than the one requested, since those differ
whenever a fallback serves and what actually ran is the question. Headline
figures are stat tiles with a sparkline: a single current value is a number,
and plotting one is how a chart ends up with a single bar in it. The
sparkline says only rising, falling or flat, which is the one thing the
number cannot.
Ranked horizontal bars, because the labels are email addresses and model
names and the label is the point — the question is which person, not what
shape the distribution has. Rotated or truncated axis labels say the chart
matters more than reading it. Trimmed where drawn and complete on hover, so
the plot keeps its geometry and a reader can still find out whose row it is.
The palette is validated rather than chosen: every adjacent pair clears the
colour-vision separation threshold, and the dark values are the same hues
re-stepped for the dark surface instead of an automatic flip. Run through a
validator, because that is the only way anyone can honestly claim a palette
is colourblind-safe.
Which turned up a bug in the existing chart code. cssVar is an allowlist of
four names that silently returns --accent for anything else, and callers
wrap it in var() themselves — so every new series would have rendered in one
colour with accent-coloured text. Caught by a test written from the rule
that chart text wears text tokens, never the series colour.
A key with no owner is labelled "operator-issued keys" rather than unknown:
it predates accounts and is doing its job, and "unknown" would imply
something is wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
