Accounts, API keys and Organizations views read the store and change it by
running the users, keys and org commands in-process, so their checks and
messages are the ones shown. A Mirror view lists what the mirror holds and
which models use it, downloads into it and removes from it. Models gain
adding from a definition file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Enter on a row of Models, Nodes or Replicas opens what can be done to it,
as forms and questions over the admin API. A Quotas view sets and clears
capacity quotas. Do gains serving a model, upgrading the cluster,
restarting every replica and a backup.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The summary's GPU, fabric and idle tables were written to standard output
regardless of the writer they were given, so on a cluster with a GPU the
console printed the GPU table across its full screen every three seconds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The console gains live views of nodes, GPUs, models, replicas, usage and
agent logs, and a settings tree over every key of server.yaml (agent.yaml on
a node), generated from the configuration structs: values with their
defaults and doc comments, edited in dialogs, saved only once the service's
own loader accepts them, comments kept. The generator now tells blocks apart
by package, which brings the TLS, engine and supervisor settings back into
the reference.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The setup screens can reach a database machine by a join code instead of
ssh: they listen on 7444 and show a short code; llmgw join dials out,
checks the setup by fingerprint, admits itself with the code's secret, and
runs what the setup sends until it is done. A joined machine is one more
onboard.Remote, so checks and the Patroni install run over it unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
With nothing installed, sudo llmgw asks for one machine or three, takes how
to reach the other two (ssh agent, key or password, and a sudo password used
only for the install), checks every machine first, shows the plan and its
command line, and installs. On a control plane it opens a console with the
cluster's state and the next steps. Machines are reached with Go's ssh so a
password works; the store's etcd and access rules use this machine's routed
address rather than a client-facing name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as
running on the reference cluster, and the address that follows the primary.
Backups read such an address as libpq host and port lists and prefer a
standby, rather than resolving the list as one name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Requests and tokens by tenant, the shares failing and refused by a limit,
first-token p95, budgets spent this period and the day's heaviest users,
filtered by a tenant chooser. Installed beside the cluster dashboard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An organization's members, limits and use are a panel of the people page,
where they can also be changed, so the administrator's menu loses its
Organizations entry and /org redirects there. Managers keep /org.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A team's members, quota and spend were on three pages that linked to each
other. They are three tabs of one page now, each at its old address, and
choosing anything in the tree opens its limits, capacity and use beside it,
editable in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A card with something else on it when the agent started stayed out of
placement after that process let go of it, until the agent restarted. The
agent now sends what processes other than its own engines hold on each card,
and the control plane replaces the registration verdict with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page draws every card coloured by utilisation, memory or
temperature, and lists replicas whose every card is under 5% busy.
llmgw_gpu_assigned says which replica, model and owner hold each card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tp on a llama.cpp GPU profile is read as gpus_per_node rather than
refused, and a Hugging Face repository page as a vLLM source as the
repository id, so models written that way are placed after the upgrade
instead of dropped from the catalog. Engines adopted after an upgrade
are checked for computing on their GPUs too.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The explorer fetches a safetensors set with its config and tokenizer;
the mirror gives out a version's manifest only once nothing is
downloading and every shard its index names is there, with each file's
sha256, which nodes check; mirror add takes several URLs; a repository
page is refused as a source.
Refs #2 (item 9).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A taken address on PostgreSQL fell through to a generic error, because
the store looked for SQLite's "UNIQUE". It checks code 23505 too. The
done page tells someone whose address had an account to sign in or ask
for a reset, without saying which case it was; other failures give a
logged reference.
Refs #2 (item 6).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A RoCE port without a v2 GID and a missing ib_uverbs failed doctor, and
node add, whatever fabric used them. They are now failures only for a
declared fabric and warnings otherwise; node add passes its --fabric
list to doctor so those are checked as declared.
Refs #2 (item 5).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A restart with no room beside it waited the drain grace period before
stopping the old replica, and stopped it even when the replacement
would fit nowhere. It now stops at once when the replica alone blocks
its replacement, gives the restart up (saying why) when nothing would
fit, and ctl restart --in-place stops first on request.
Refs #2 (item 3).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.
Refs #2 (items 4 and 7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The form offered no placement and named every GGUF profile cpu. It now
lists the fleet's GPU classes (the freest chosen) and the CPU, a GPU
count and parallel requests, names the profile from the placement, shows
what the model will run as, and refuses the CPU while GPUs are free
unless that is ticked.
Refs #2 (item 1).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
tp on a llama.cpp profile was a width nothing placed. gpus_per_node is
now the count: placement reserves that many cards on one node and the
engine splits layers across them. tp, pp and nodes are refused with
what to write instead.
Refs #2 (item 8).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llama.cpp whose CUDA backend cannot load runs on the CPU without an
error. The CUDA build is now asked for its devices when installed, a
shard on GPUs must hold memory on each before it is ready, and
engine-check reports the devices. Shards get CUDA_DEVICE_ORDER=PCI_BUS_ID
so CUDA numbers GPUs as placement does.
Refs #2 (item 2).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
--lora with --lora-init-without-apply leaves an adapter's scale at 1,
and llama-server applies it at that scale to any request without a
"lora" list of its own: every plain request for the model was served
with its adapters on. Found live on qwen2.5-coder-7b-instruct. Each
adapter is now loaded with --lora-scaled FILE:0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ed25519 over checksums.txt: llmgw release keygen makes the pair, CI signs
with the RELEASE_SIGNING_KEY secret when it is set, and release.Install
refuses an unsigned or wrongly signed release when the build embeds a
trusted key (internal/release/trusted.pub, empty for now) or
LLMGW_RELEASE_KEY names one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Candidates tried on the way to finding no room were listed as whose replicas
a placement would stop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
