Commit Graph
4 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5 d998ea7ef5 Let a model say what it is for
CI / test (push) Successful in 1m42s
A gateway serving nine models offered nine names and nothing to choose between
them. /v1/models is the only view most clients ever have, and it listed ids.

Models carry a description now: written when the model is added, shown on the
dashboard and sent to clients in /v1/models.

Written rather than derived. A registry offers a pipeline tag and a wall of
README, and neither answers the question somebody picking from a list is
asking — which of these should I send this request to. "General chat, strong
Swedish; slower than the 8B" is not in any metadata field.

It is optional, and absent means absent rather than an empty string in the
response: a model nobody described should read as undescribed, not as described
with nothing.

On the dashboard it sits under its row rather than in a column. A sentence in a
table cell either wraps every other column into a stripe or gets cut off.

Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com>
2026-09-27 16:51:09 +02:00
Ilker ManapandClaude Opus 5 e0ea4f6be9 Fill in the memory a model needs, instead of leaving the field blank
CI / test (push) Successful in 1m41s
Serving a model from the mirror page left "Needs RAM (GB)" optional and empty.
Empty writes no constraint, and a CPU profile has no card to be short of, so
every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB
machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never
told what the model needed.

The mirror knows the size of the weights, so the field arrives filled in:
weights plus half again for the KV pool, plus two for the engine. The half is
there because the cache grows with the same things the weights do, layers and
heads. It is an estimate and the field is still editable — what it is not is
blank.

A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every
model to the one large machine would be a different way of getting placement
wrong.

An unknown size fills in nothing rather than the floor of two, which would
exclude no node while reading as though somebody had measured it.

And when the estimate is larger than any node in the fleet, the page says so,
with the largest node's memory. Better read there than discovered by a node
being OOM-killed twenty minutes into a load.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 16:38:49 +02:00
Ilker ManapandClaude Opus 5 2acd794022 Stop placing engines on nodes that do not have them
CI / test (push) Successful in 1m42s
A node's registry holds an adapter for every engine whether or not its binary
exists, the agent never said which were installed, and placement carried an
Engine field it had never once read. So a vLLM replica was as happily placed on
a node with no vLLM: the node registered healthy, the shard failed minutes
later with "executable file not found" in the engine's own words, and the
reconciler tried again.

That is how fifteen gigabytes of safetensors reached this cluster's mirror
before anybody noticed it had no card and no vLLM to run them with.

Agents now probe what is installed and report it — the protocol has carried
engine_versions since it was written and nothing ever filled it in. Placement
filters on the key set, `ctl nodes` has a column, and the mirror page's serve
warning is a fact instead of a guess about cards, which would have waved
through a fleet of eight-GPU nodes that never had vLLM.

Probing resolves binaries and runs none of them. The first version asked each
engine for --version, which is fine for an engine and not fine for the wrapper
scripts people put in front of one: a script that shifts past flags it does not
recognise and then execs the server will start an inference server, at agent
startup, on every node. The agent's own test suite installs exactly such a
script and caught it. So presence is a path lookup, and the version is left to
`ctl engine-check`, which is run by somebody who meant to run the engine.

SGLang is the exception and has to execute, because a module has no path to
look up — but the command is llmgw's own one-line import rather than the
engine's launcher. Its exit code is the answer: python3 is on almost every
machine, and treating a failed import as "installed" reported SGLang on every
node in a fleet that had none, which is the failure being fixed, inverted.

Probed once, not per reconnect, and bounded by the context that registration
waits on: an agent that cannot reach the control plane is worse than one
reporting an incomplete list.

A node that reports no engines at all is not excluded anywhere — by placement,
or by the page. That is what every agent older than this reports, and emptying
a fleet on a control plane upgrade is a worse failure than the one being fixed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 16:11:55 +02:00
Ilker ManapandClaude Opus 5 e034862b79 Serve a mirrored model from the list, instead of writing YAML for it
CI / test (push) Successful in 1m35s
Fetching a model put it on the mirror and left the operator to hand-write a
catalogue entry and run `ctl model add` on the control plane — for a source the
page was already displaying, with an engine decided by the file extension it
had just shown them.

A present row now offers to serve it. The form opens on that row with the name,
the mirror's own URL and the engine filled in, and asks for the two things
nobody can infer: the context pool and how many replicas.

What it refuses is the point.

A version of several files is refused with the reason. A catalogue entry names
one source, so an entry for a sharded model is an entry for one shard, which
loads as a broken model rather than failing to load. Nothing said so before —
the mirror happily holds a four-shard model that no catalogue entry can
describe.

Safetensors on a fleet with no GPU is warned about before anything is added.
This is the failure we hit today: fifteen gigabytes of shards arrived, and only
then did looking at a node show no card, no vLLM and no SGLang. The control
plane is not told which engines a node has installed, so the check is on cards,
which it does know — a warning rather than a refusal, because a card can be
added tomorrow.

A name already declared in configuration is refused here too, matching what the
model service would say anyway, but before the form is filled in rather than
after.

Without a store there is no serve action at all: a model added at runtime would
not survive a restart, and offering it then refusing is worse than not offering.

The mirror page's dependencies became a struct. Three are optional interfaces
and two are the same shape, which at a call site is a swap nobody notices until
a page shows the wrong thing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 15:54:20 +02:00