A gateway serving nine models offered nine names and nothing to choose between
them. /v1/models is the only view most clients ever have, and it listed ids.
Models carry a description now: written when the model is added, shown on the
dashboard and sent to clients in /v1/models.
Written rather than derived. A registry offers a pipeline tag and a wall of
README, and neither answers the question somebody picking from a list is
asking — which of these should I send this request to. "General chat, strong
Swedish; slower than the 8B" is not in any metadata field.
It is optional, and absent means absent rather than an empty string in the
response: a model nobody described should read as undescribed, not as described
with nothing.
On the dashboard it sits under its row rather than in a column. A sentence in a
table cell either wraps every other column into a stripe or gets cut off.
Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com>
Serving a model from the mirror page left "Needs RAM (GB)" optional and empty.
Empty writes no constraint, and a CPU profile has no card to be short of, so
every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB
machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never
told what the model needed.
The mirror knows the size of the weights, so the field arrives filled in:
weights plus half again for the KV pool, plus two for the engine. The half is
there because the cache grows with the same things the weights do, layers and
heads. It is an estimate and the field is still editable — what it is not is
blank.
A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every
model to the one large machine would be a different way of getting placement
wrong.
An unknown size fills in nothing rather than the floor of two, which would
exclude no node while reading as though somebody had measured it.
And when the estimate is larger than any node in the fleet, the page says so,
with the largest node's memory. Better read there than discovered by a node
being OOM-killed twenty minutes into a load.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fetching a model put it on the mirror and left the operator to hand-write a
catalogue entry and run `ctl model add` on the control plane — for a source the
page was already displaying, with an engine decided by the file extension it
had just shown them.
A present row now offers to serve it. The form opens on that row with the name,
the mirror's own URL and the engine filled in, and asks for the two things
nobody can infer: the context pool and how many replicas.
What it refuses is the point.
A version of several files is refused with the reason. A catalogue entry names
one source, so an entry for a sharded model is an entry for one shard, which
loads as a broken model rather than failing to load. Nothing said so before —
the mirror happily holds a four-shard model that no catalogue entry can
describe.
Safetensors on a fleet with no GPU is warned about before anything is added.
This is the failure we hit today: fifteen gigabytes of shards arrived, and only
then did looking at a node show no card, no vLLM and no SGLang. The control
plane is not told which engines a node has installed, so the check is on cards,
which it does know — a warning rather than a refusal, because a card can be
added tomorrow.
A name already declared in configuration is refused here too, matching what the
model service would say anyway, but before the form is filled in rather than
after.
Without a store there is no serve action at all: a model added at runtime would
not survive a restart, and offering it then refusing is worse than not offering.
The mirror page's dependencies became a struct. Three are optional interfaces
and two are the same shape, which at a call site is a swap nobody notices until
a page shows the wrong thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The explorer listed every weight file with its own button. For a repository of
alternatives that is right — q4_k_m or q8_0, you want one — and for a sharded
model it is fifteen clicks to fetch one thing, with nothing stopping you from
losing count and leaving a version with fourteen of its parts.
Files are grouped into sets now. A run of "-00002-of-00004" is one row and one
button; everything else stays its own row.
Not "fetch all the weights", which would be wrong half the time: two safetensors
files are halves of one thing and useless apart, two GGUF files are usually
alternatives and fetching both spends gigabytes on a copy nobody asked for. The
set is the unit that is whole.
A repository can hold both the split and the joined copy of the same
quantisation — the real Qwen coder repos do — so a shard set keeps the -of-
notation in its name rather than collapsing onto the name of the whole file
beside it.
A set publishes no single checksum. Several files have several, and showing one
of them invites comparing it against the wrong thing.
The form posts every source in the set, and a failure part way says how many
already started: a set that fails on its fourth shard has three arriving, and
reading that as "nothing happened" sends somebody to fetch them all again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirroring anything meant knowing a repository existed, knowing which of its
thirty files were the weights, and assembling a resolve URL by hand. The mirror
page had a text box for that URL and no way to find one.
It now searches Hugging Face and lists a repository's weight files with their
sizes and published checksums, each with a button that fills the fetch form.
Only weight files. A repository is mostly a README and a tokenizer, and listing
those buries the four entries somebody came for.
The size shown is the LFS size, not the plain one. For an LFS-stored file —
every weights file — the plain field is the size of the pointer, so the other
one reports a gigabyte model as 133 bytes, and that is the number somebody
would then plan disk against.
The source URL is pinned to the commit the listing came from rather than to a
branch, so what the mirror fetches is what was on screen. A repository
re-uploaded under the same branch otherwise leaves two different sets of
weights with identical provenance and no way to tell them apart. The version
defaults to that revision for the same reason.
The registry's own failures are reported on their own line. It is somebody
else's server on the other side of the internet, and it being down says nothing
about the mirror — which is what the page is for and has to keep working
regardless. With no registry configured there is no search box at all, rather
than one that can only time out.
Gating decodes from a bool or a string, because the field is documented as the
first and is the second in practice: strict decoding fails the whole response,
so one gated model would hide the other nineteen in a page of results.
And a bug this made obvious: downloads were keyed by model and version, so the
second shard of a sharded model was refused as "already fetching" until the
first had finished. A fifteen-shard model had to be pulled one file at a time,
each waiting on the last. Keyed by file now — which is what the single-flight
was ever protecting, since the race is two writers on one .part. A version made
of several files reports as the worst of them: it is not present until every
shard is, and one shard failing makes the version unusable however many others
arrived.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/admin/mirror, behind the same admin session as the other admin pages. What is
held, what is arriving with its byte count, what failed and why; start a fetch,
retry a failed one, remove a version.
Not mounted at all without a mirror configured. The right page for a deployment
with no mirror is no page, rather than an empty one pointing at nothing.
It reloads itself only while something is downloading. A reload at any other
moment takes with it whatever is half-typed into the fetch form, and there is
nothing to watch when nothing is moving.
A failed row offers Retry, not Remove. A fetch that failed left nothing on disk,
so the remove could only answer "the mirror holds nothing for this" — a button
whose single outcome is a confusing error. Starting the same download again is
what somebody looking at that row wants, and the mirror allows it: only a
download still running is refused.
An unreachable mirror is said outright instead of rendering as an empty table.
The two look nearly identical and mean opposite things: one is a mirror waiting
to be filled, the other is a machine to go and look at.
Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a
call site get swapped silently and are discovered by somebody clicking the
wrong link.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
