tp on a llama.cpp profile was a width nothing placed. gpus_per_node is
now the count: placement reserves that many cards on one node and the
engine splits layers across them. tp, pp and nodes are refused with
what to write instead.
Refs #2 (item 8).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Candidates tried on the way to finding no room were listed as whose replicas
a placement would stop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It said "stopping a drained replica without a replacement after=5m0s" when
it had been given the 30-second preemption grace.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Idle times are unknown after a control plane restart, which left the choice
between a 1-core and a 4-core replica to map order.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Candidates were added in order until the model fitted, and every one added
was stopped, including those tried first that freed room on the wrong node.
A set that works is now pruned to the replicas it needs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stored ctl bench results reach the reconciler: requests per replica past
the measured SLO concurrency scale a model up, a replica is taken back only
if the rest stay under 70% of theirs, and among profiles of equal priority
the most measured tokens per GPU or core is tried first. ctl model target
sizes a model for a throughput against the room there is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model has an owner (org, team or project). Quotas guarantee a scope
GPUs, cores and memory and may limit it; what a scope holds is summed from
its models' replicas and those beneath it. A replica within its owner's
guarantee is not taken for another scope; what is held beyond a guarantee
is borrowed and taken back for a scope within its own. A limit stops a
model before it starts, and headroom stops there too. ctl quota and ctl
model owner set them; metrics and a CapacityAtLimit alert report them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A profile's cpu_cores is reserved out of its node's CPUs by placement, as
memory is, and becomes the engine's thread count. Agents report their CPU
count in registration and heartbeats. New CPU models must declare memory;
model views show reserved against measured memory with warnings, and
ctl model resources sets both for a stored model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Held tab's model name opens a card: each replica with its node, state,
age, CPU and memory, the room for more found by dry-running the scheduler
on a copy of the inventory, and a form for the always-running and
under-load replica counts. An always-running count placement cannot meet
is refused with the reason. Headroom is also on the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.
Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.
Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.
reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's max never did anything: desiredFor returned the minimum, or one
for an on-demand model someone was asking for, whatever the queue. Six small
CPU nodes run out of slots quickly, and a model with room to grow stayed at
its minimum while requests waited.
The router already scrapes each engine's running and waiting requests. The
control plane now reads them per model on every tick, alongside when each
replica was last used, and the reconciler keeps how many replicas load has
added above a model's base. One or more requests waiting per ready replica,
held for a minute, adds one; ten minutes with nothing waiting removes one of
those again. It never goes above max or below the base, moves one step at a
time, and does not add while the last addition is still placing or starting
— on this fleet a replica is minutes of weights and gigabytes of memory, and
a spike should not leave a pile of replicas behind it. Decisions are logged
with the load behind them.
Nothing live changes until a model's max is raised above its min.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.
`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:
- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
--file, leaving out vision projectors and importance matrices, and naming
what the repository does have when nothing matches; gated repositories are
refused;
- the mirror: fetched at the repository's revision, with progress, and each
file's sha256 compared with what the registry published — the mirror is
the one place a corrupted download is caught once rather than loaded on
every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
the answer and tokens per second.
ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.
The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every upgrade so far has been a script of mine copying a binary to each
machine over ssh, restarting it and polling until it came back — which is
the job, but not something the product could do, and which needed ssh into
every node.
`sudo llmgw ctl upgrade VERSION` on the control plane does it. This machine
first: the release for its platform, checked against the release's
checksums, the old binary kept beside it, llmgw-server restarted, and a wait
until every node that was ready has reconnected. Then each node in turn: the
control plane tells its agent the version over the stream it already holds;
the agent fetches and checks the release itself, swaps it in with a rename,
answers, and exits with status 75 so systemd starts the new binary. Engines
keep running, as for any agent restart, and are adopted again.
A node is done when it is back on the new version, ready, with as many ready
replicas as before; one that is not within the timeout stops the run before
the next node, so a bad release reaches one machine. Nodes already on the
version are skipped, so running it again resumes.
The fetch-and-verify is one package now, shared with onboarding's idea of
what a release version looks like.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.
Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.
Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.
After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.
A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.
Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was no way to restart a running replica. A running engine picks up an
engine key, an edited profile or a new engine version only when it starts,
so after v0.10.3 the nine replicas already running stayed without a key —
reachable from any node — with nothing to do about it short of draining
nodes, which on a small CPU fleet has nowhere to move a model to.
`llmgw ctl restart REPLICA... | --model M | --all` queues replicas to be
replaced. A replica being replaced keeps serving but stops counting toward
its model, so the reconciler places a replacement beside it; once that is
ready the old one is drained and stopped as a node drain would. Replicas of
one model go one at a time, so a model never needs the memory of every
replica twice. Where no second copy fits anywhere, the old replica is
stopped after the drain grace period and the new one starts in its place,
and the log says so, since that is the one case with an outage.
`ctl replicas` shows "(restart queued)" and "(restart replacing)" while it
happens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin listener has no login. Reaching it is meant to mean holding the
control plane, so it binds to loopback and an operator reaches it over an
ssh tunnel — and so does every page open in that operator's browser. The
API decoded JSON whatever the Content-Type said, so a page could POST a
text/plain model definition, no preflight needed. A model's profile env
reached an engine the agent runs as root, unfiltered: LD_PRELOAD in a
definition was code execution on every node. A page that rebound its own
name to 127.0.0.1 could also read pprof's heap, which holds API keys.
The admin listener now answers only when Host is an address or localhost,
refuses requests a browser marks as cross-site (Origin, Sec-Fetch-Site),
and takes changes only as application/json, which no page can send across
origins without a preflight nobody grants. ctl sends that on every change,
bodiless DELETEs included; curl and ctl are otherwise unaffected.
Profile env that changes what runs rather than how — the dynamic loader's
LD_*, PATH, Python's PYTHON* start-up variables, BASH_ENV and their like —
is refused when a model is defined, and dropped again by the engine adapter
before the process starts, whatever sent the profile. None of the live
catalog's models sets any.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
doctor checks a node alone, and every check can pass while two nodes still
cannot reach each other. ctl fabric-test A B asks B's agent for an
ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs
them, one rail at a time, and grades each against its link's rate. It goes
through the agents' streams, so there is no SSH between nodes; the agent
installs perftest the first time, and names the RoCE v2 GID itself.
The result is read from perftest's table by its heading, not a column
number. Agents now list the requests they answer beyond the usual
(Register.features), and the control plane refuses a test on one too old to
answer instead of waiting minutes for it. The forwarding dispatcher carries
the new capability, and its test says so: a missing forward would have made
the command quietly unavailable in production.
Not yet run over real InfiniBand or RoCE; this cluster has neither.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane of this release asks every node for vLLM. An agent of the
last one does not know how to install it, and answered "unsupported", so a
cluster half way through its upgrade showed every node as failing. Agents
now say which engines they can install (Register.installable_engines), and
are asked for no others; an agent that says nothing is asked for llama.cpp
alone, as it always was.
doctor reports the CPU's vector extensions, and warns when AVX2 is missing,
with the fix for a VM: the test node for node add was one, on QEMU's default
CPU model, where llama.cpp runs several times slower and nothing else looks
wrong. node add shows it, since it runs doctor.
The changelog's unreleased section was two releases' worth of sections one
after the other, with two lists of fixes and the upgrade notes in the
middle. It is one entry now: largest changes first, the fixes together, and
every upgrade note at the end, with vLLM's default install and how to keep
your own.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its
GPUs reach each other over NVLink whatever node they are in. Each GPU
now reports its fabric from nvidia-smi -q - state, status, ClusterUUID,
CliqueId, bandwidth - at registration and once a minute after, since a
fabric manager restarting drops every card out at once; and each node
reports nvidia-imex and its channels, without which the domain's GPUs
cannot share memory across nodes. A single machine's NVSwitch, which
reports a cluster of zeros, is no domain.
A replica spanning nodes is placed inside one domain first, on nodes
whose every card is joined and whose IMEX is up, and its shards are
told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback.
/admin/nvlink draws each domain as a rack - the NVLink Switch band on
top, light running down a backplane, trays of GPU chips that glow when
joined, pulse red when not, turn amber on degraded bandwidth - and
lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink
matrix, each card's NUMA node and nearest adapter, and its domain and
IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics
llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane asks one node on each physical fabric to run
ibnetdiscover every five minutes and parses the text itself, so a
format the parser has not met is a control plane upgrade, not one on
every node. From the map it works out the switch tiers (leaf, spine,
core, by distance from the adapters), which adapter is which node's,
which cables trained below the rate the rest of their kind run at, and
whether the rails line up: every node's Nth adapter on its own leaf.
A page per fabric (/admin/fabrics) draws it: switches in their tiers,
adapters fanned out under their leaves, cables coloured by rate and
bundled, degraded ones marching red with a marker, and the cluster's
own adapters live - down ports glowing red, and particles running each
cable as fast as its adapter is sending and receiving, from the
counters. It zooms, pans, fades what a hovered switch or adapter does
not touch, opens a panel of detail on a click, colours by rail, finds a
node, and refreshes every fifteen seconds. The data is also at
/api/v1/fabrics/{id}/topology.
Placement reads the map: a replica spanning nodes takes nodes of one
leaf together when it can, and nodes whose cable trained short last.
Static assets are named by their content, as the stylesheet already
was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A fabric's identity was its master subnet manager, which changes on a
failover: for the minute its ports took to agree, one fabric read as
two, and a name covering it as two separate fabrics. Each agent now
also lists every subnet manager on a fabric (saquery SMIR, through one
port per master, once a minute); managers listed together are one
fabric, named by the lowest GUID among them, so the identity survives
the master changing.
The same list says what a fabric can survive. The cluster page's new
subnet manager table, /api/v1/fabrics, doctor and metrics
(llmgw_fabric_master_sm, llmgw_fabric_subnet_managers,
llmgw_fabric_problems) show each fabric's master, priority and
standbys, and say when a fabric has no standby, its ports disagree on
the master, ports are up with no subnet manager (INIT), or the master
changed in the last hour. Every change of master a port reports is
kept, and a fabric's last day of them is listed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A fabric was what agent.yaml called it. With several independent
InfiniBand fabrics at a site, two given one name would have had a
replica spread across both, starting and then hanging on its first
collective. Each active InfiniBand port now reports the master subnet
manager it sees (sminfo, asked once a minute), which every port on one
fabric shares and ports on separate fabrics cannot; a RoCE port reports
the IP network of its RoCE v2 address. Ports also carry their node and
port GUIDs.
Placement divides a declared fabric by that physical identity and forms
a replica on one part only; a node whose adapters for one name are on
two fabrics is refused, and a node that cannot say is kept apart from
those that can. /api/v1/fabrics and the cluster page list each physical
fabric with its subnet manager, and say when one name covers two
fabrics, two names one fabric, an adapter is up but undeclared, or a
node cannot be checked. doctor reports each port's subnet manager, and
an agent on a node with adapters installs infiniband-diags when sminfo
is missing.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GPUs were chosen by growing one card's NVLink peers, which took cards
each joined to that one but not to each other, and knew nothing of the
fabric. Every set of the right size is now scored - a node's handful of
cards makes that cheap - by NVLink pairs within it, then for a
multi-node shard by how near each card is to one of the fabric's
adapters (PIX before SYS), then by NUMA nodes spanned, with the lowest
indices breaking ties. A node without topology still gets the lowest
indices.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.
The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release binary is built without cgo, so without NVML, and NVML was
the only source of NVLink peers: on every NVLink machine the portable
agent reported none, and tensor-parallel groups were chosen by GPU index.
Discovery now reads `nvidia-smi topo -m` after either path, for the
NVLink peers (kept from NVML where it gave them), each card's NUMA node,
and how far each network adapter is from it - PIX, PXB, PHB, NODE or
SYS - which travels to the control plane as nic_paths for placement to
use. Both the NICn-with-legend layout of newer drivers and the
adapter-named columns of older ones are read.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nodes are CPU machines, and nothing measured how busy one was: the
heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the
total. The agent now reads /proc between heartbeats - CPU across all
cores, load averages, memory used and available, and each engine's
process group's cores and resident memory - and sends it with the
heartbeat. Nothing new runs on the nodes and Prometheus needs no new
target: the control plane exports it with the rest of the fleet, as
llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load,
llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes,
llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more
than a minute old is left out rather than repeated.
The cluster page's node table shows CPU and RAM as bars, and each node
links to a page of its own: current CPU, RAM, load and disk, the engines
on it with their cores and memory, and an hour of CPU, RAM, load,
per-model cores and memory and queue depth from Prometheus. It refreshes
in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.
internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.
Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.
Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every mirrored version was in use on the cluster, so the serve link — hidden for a version already served — appeared nowhere, and changing a model between always on and on demand needed the CLI. Each model using a file now shows how it runs and offers always on, on demand (15 minutes idle) or stopped, through the model service's Scale; models declared in server.yaml show their mode and where to change it. The serve form asks the same question.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each held file took two table rows — seven columns and its address
beneath — so one sharded model filled the screen. A version is now one
line: model and version, the file or "N files", total size, and the
catalog models that use it. The address is behind a "copy address"
button (a set's is its directory, trailing slash included) and the
checksum is in the file's tooltip.
The model service reports which models name each source, so a file in
use shows who uses it and is not offered for serving a second time;
removing one warns that the catalog uses it. Downloads and failures
stay one line per file, since each has its own progress or retry. The
header counts versions, not files, and the nav calls /admin/users
"people" everywhere.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Seeing what a node did meant ssh through a jump host, to a machine whose
address nobody remembers, to read a journal only root may open. When an
engine install failed on a new node today, the reason was one line — and it
was four steps away from the page that said the install had failed.
Each node's row on the cluster page now links to its logs: the agent's own,
and every engine running there, chosen from the list the node itself sends.
The page can follow, refreshing every five seconds, and is on both the admin
listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the
same.
They travel over the stream the agent already holds, so they need nothing
the control plane does not already have: a request goes down, the lines come
back, and a reply that does not come in ten seconds is an agent too old to
answer. The agent keeps its last 2000 lines in memory as well as writing them
wherever it did before; the engines' last few hundred lines were already kept
and sent only when one crashed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node ran whatever llama-server somebody had once unpacked on it. Onboarding
checked an engine only when told its path, installed none, and a node with
nothing on it registered as ready. The one time that went wrong it went wrong
invisibly: the binary was there, executable, and could not start for want of
libgomp1, while the node reported healthy.
The control plane now says which engines every node has, at which version —
`engines:` in server.yaml, defaulting to the llama.cpp build this release was
checked against — and each agent converges on it. What only the node can
know, the node works out: the build for its processor and driver (CUDA with
its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages
that build needs from its own package sources, and whether the result runs.
Every archive is checked against the sha256 GitHub publishes for it and
unpacked refusing any path or link that leaves the install directory. ldd
names a missing library outright; running `--version` proves the rest. Only
then is a directory marked installed, and it is checked again every ten
minutes, because an unrelated upgrade can remove a library.
An engine is reported only once installed and seen to run, or when the
agent's configuration names it on purpose — never because something happened
to be on PATH. The agent now says when it has probed, so an empty list means
"has none" rather than "too old to say": a node rebuilt from nothing is no
longer a candidate for every model at the moment it can run none of them.
Placement's refusal carries the node's own words, "installing: downloading
llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the
cluster page, along with a warning for a CPU without AVX2 — the VM CPU type
that makes everything five times slower and nothing look wrong.
Agents also report their OS, architecture, vector extensions and NVIDIA
driver. Onboarding installs llmgw itself from the CI release for the node's
architecture, checked against the release's checksums, rather than copying
whatever binary the operator's machine had.
vLLM and SGLang are not installed yet: they are Python environments of
several gigabytes, not an archive, and still have to be named in the agent's
configuration.
Also fixes a race on the weights fingerprint, which two heartbeat loops touch
during a failover.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every agent file on the cluster named its own node, its own address, the
path to llama-server and two weight directories — different on every node,
kept true by hand, and not true: node05 carried "ram: 32g" beside a measured
31 GB, looked for weights in an empty directory while fetching them into
another, and the other four never looked in theirs at all, so four nodes
reported no disk. Growing a node's RAM would have gone unnoticed, because
memory was read once, at registration.
The agent now finds these out. Its id is the hostname, which is also the
certificate's common name. Its address is the one on the route to the
control plane, which the kernel picks for a UDP dial without sending
anything. llama-server is looked for on PATH and then where release archives
are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather
than a failed discovery — which also cut discovery short before InfiniBand
on every CPU node not told to skip GPUs. Fetched weights land in
/var/lib/llmgw-weights by default, and wherever they land is always looked
in.
Memory is read on every heartbeat, and installed engines are probed again
each minute and sent when they change, so a VM given more RAM or a node
given vLLM becomes a candidate without anyone restarting its agent. The
control plane logs both changes, since "the node now qualifies" is the
answer to why a model placed an hour after it was asked for.
What is written still wins. The agent file shrinks to the control plane's
addresses and the TLS files, the same on every node; onboarding writes only
the overrides it was given. What cannot be found stays configuration: which
fabric an adapter is on, because OpenSM's default prefix is the same on all
of them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The stale-profile flag could not fire where it matters. The control plane
keeps no record of a running replica across its own restart — it rebuilds
them from what the agents report — and adoption left the fingerprint empty,
so every replica read as unknown afterwards. That is precisely the moment
the catalog is most likely to have been edited: change the configuration,
restart the server, and nothing would ever say the fleet disagrees with it.
The fingerprint now travels with StartShard, the agent keeps it beside the
pool and fabric it already echoes back, and adoption takes the replica's
version from the agent rather than from the catalog it is about to be
compared against. An agent too old to report it sends nothing, which still
reads as unknown rather than as matching.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defects, both of the same kind: something the CLI knew and the rest of
the system did not.
A profile can be edited while a replica is running against the old one. The
reconciler deliberately leaves it alone — churning a replica that is serving
costs more than the imperfection — but there was no way to see it, so the
cluster could disagree with its own configuration indefinitely and read as
healthy. Profile now carries a fingerprint, a replica records the one it was
placed with, and both the API and `ctl replicas` say "stale profile" when
they differ. An empty fingerprint is not stale: it means a replica adopted
from an older agent, which is unknown rather than wrong.
Adding a model from the admin page skipped the source check `ctl model add`
has done all along, so a typo or a mirror that is down became a replica
failing on a node minutes later, with an error naming the node. The check
moves out of the CLI into a package both use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Placement compared a profile's memory requirement against the node's total, so
a 31 GB machine accepted two models that each declared 15: each was asked about
on its own, and nothing recorded that the first had claimed anything. GPUs were
never handed out twice this way — they are tracked as claimed indices — and
memory was, because the allocation carried no figure for it.
An allocation records what its profile asked for, and eligibility subtracts
what a node already owes before comparing. The rejection names both numbers,
because "needs 15" beside a 31 GB machine reads as a bug in the scheduler
rather than a full node.
Declared, not measured. A profile that says 15 is treated as holding 15 whether
the process has touched that much yet or not — the alternative is deciding
against what a model has grown into so far, which is wrong every time one is
still loading.
Zero is nobody said, and claims nothing. Counting it would let one undeclared
model make a whole node look full.
Adoption after a restart re-reads the figure from the catalog rather than from
the agent: what a profile asked for is not something a running process reports,
and a control plane that forgot every claim on restart would put a second large
model onto a node already full of the first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gateway serving nine models offered nine names and nothing to choose between
them. /v1/models is the only view most clients ever have, and it listed ids.
Models carry a description now: written when the model is added, shown on the
dashboard and sent to clients in /v1/models.
Written rather than derived. A registry offers a pipeline tag and a wall of
README, and neither answers the question somebody picking from a list is
asking — which of these should I send this request to. "General chat, strong
Swedish; slower than the 8B" is not in any metadata field.
It is optional, and absent means absent rather than an empty string in the
response: a model nobody described should read as undescribed, not as described
with nothing.
On the dashboard it sits under its row rather than in a column. A sentence in a
table cell either wraps every other column into a stripe or gets cut off.
Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com>
A node's registry holds an adapter for every engine whether or not its binary
exists, the agent never said which were installed, and placement carried an
Engine field it had never once read. So a vLLM replica was as happily placed on
a node with no vLLM: the node registered healthy, the shard failed minutes
later with "executable file not found" in the engine's own words, and the
reconciler tried again.
That is how fifteen gigabytes of safetensors reached this cluster's mirror
before anybody noticed it had no card and no vLLM to run them with.
Agents now probe what is installed and report it — the protocol has carried
engine_versions since it was written and nothing ever filled it in. Placement
filters on the key set, `ctl nodes` has a column, and the mirror page's serve
warning is a fact instead of a guess about cards, which would have waved
through a fleet of eight-GPU nodes that never had vLLM.
Probing resolves binaries and runs none of them. The first version asked each
engine for --version, which is fine for an engine and not fine for the wrapper
scripts people put in front of one: a script that shifts past flags it does not
recognise and then execs the server will start an inference server, at agent
startup, on every node. The agent's own test suite installs exactly such a
script and caught it. So presence is a path lookup, and the version is left to
`ctl engine-check`, which is run by somebody who meant to run the engine.
SGLang is the exception and has to execute, because a module has no path to
look up — but the command is llmgw's own one-line import rather than the
engine's launcher. Its exit code is the answer: python3 is on almost every
machine, and treating a failed import as "installed" reported SGLang on every
node in a fleet that had none, which is the failure being fixed, inverted.
Probed once, not per reconnect, and bounded by the context that registration
waits on: an agent that cannot reach the control plane is worse than one
reporting an incomplete list.
A node that reports no engines at all is not excluded anywhere — by placement,
or by the page. That is what every agent older than this reports, and emptying
a fleet on a control plane upgrade is a worse failure than the one being fixed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Quotas can be written on an organization, a team or a project, and nothing
could show what any of them had used: usage came back per key, per user and per
model only. A quota whose consumption cannot be seen is one nobody reviews
until the day it refuses somebody — and since the previous commit made a group
quota a shared pool, it is now a number several people are spending against
without being able to look at it.
`llmgw ctl usage --by team` sums every key beneath each group, with the budget
that level sets beside it.
Beside it, and only when that level sets one. A team bounded by its
organization's budget has none of its own, and printing the organization's
against each of its teams would read as though each had that much — the exact
misreading this work exists to remove.
Joined through api_keys rather than read from usage.project_id: that column
records where the key was when the request was served, while the quota is
written against the tree. They agree until a key is moved, and a report that
disagrees with the limit shown beside it is the useless one.
Three existing reports were fixed on the way, all failing on PostgreSQL and all
passing in tests, because the tests run on SQLite:
column "k.name" must appear in the GROUP BY clause (UsageSince)
column "us.email" must appear in the GROUP BY clause (UsageByUser)
column "usage.ts" must appear in the GROUP BY clause (UsageBuckets)
SQLite picks an arbitrary row for an ungrouped column and carries on; PostgreSQL
refuses. The first two were selecting names they had not grouped. The third
grouped "ts / ?" while selecting "(ts / ?) * ?", which PostgreSQL matches as
text and so saw a bare ts — it is grouped by ordinal now, the same expression by
construction. The dashboard's usage charts have been empty on the cluster this
whole time, with the reason only in the log.
All four queries were run against the cluster's own PostgreSQL before this was
committed, because passing on SQLite is what they were already doing.
The grouping level picks from a fixed table of levels rather than being
interpolated, so an unknown one is an error and not a fragment of SQL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
