server add gives another machine this llmgw, a certificate, the
configuration and store address, lets it into the store through Patroni, and
starts it; --vip keeps a floating address on the leader with keepalived,
following a new /leader endpoint. Nodes learn every control plane from the
leader and notice a dead one by HTTP/2 ping instead of after the kernel's
fifteen minutes. The setup, console, node and init screens are drawn full
screen in one frame: title bar, sidebar, bordered panel, key bar, dialogs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Charts, totals and budgets summed the request table on every view. Usage is
now also summed in the writing transaction into five-minute, hourly and daily
tables per key, person, project and model, with latency and first-token
histograms. Charts read the coarsest tier their slices allow; totals and
budgets compose rows, slices, hours and days, and stay exact. Request records
can be given a retention without moving any total.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
--lora with --lora-init-without-apply leaves an adapter's scale at 1,
and llama-server applies it at that scale to any request without a
"lora" list of its own: every plain request for the model was served
with its adapters on. Found live on qwen2.5-coder-7b-instruct. Each
adapter is now loaded with --lora-scaled FILE:0.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's adapters are fetched by each node with its weights and loaded
unapplied at start; a request naming an adapter reaches the model's
replicas with llama.cpp's per-request lora field. Adapter names share the
model namespace, are listed, and follow allowlists on their own. The
router's config seed and pushed catalog now build the name map in one place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The pool-priority filter kept the highest-priority pool however full it
was. It now takes the first pool, by priority, that is not saturated by
admission's own measure, and the preferred one when all are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A total bound across bands (admission.queue_capacity_total, default half of
all bands' capacity together); reached, a request is not refused while a
lower band has requests queued, and the newest of the lowest band gets a 503
with Retry-After instead.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CPU replicas start on half the machine's CPUs, not all of them: two busy
llama.cpp servers each spinning on all 8 vCPUs fell to 0.8 tok/s apiece,
where at 4 threads they keep 5.5. llamacpp.threads sets it per node.
Inference requests can be traced over OTLP/HTTP (tracing.endpoint), with
admission, schedule and upstream spans and traceparent sent to the engine.
Security: onboarding no longer puts the cluster root in the system trust
store and removes it where it was; the mirror's mutual-TLS API can be
limited to named certificates (--api-client); init.log no longer keeps the
administrator's password; a warning when auth is off on a public listen;
the recorded actor is sanitised and sudo-aware; Ray's dashboard is off, and
the remaining engine gaps are documented.
reconciler.go and agent.go are split by topic, the status mapping of each
failure is tested, and the plan's checklist is brought up to date.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's max never did anything: desiredFor returned the minimum, or one
for an on-demand model someone was asking for, whatever the queue. Six small
CPU nodes run out of slots quickly, and a model with room to grow stayed at
its minimum while requests waited.
The router already scrapes each engine's running and waiting requests. The
control plane now reads them per model on every tick, alongside when each
replica was last used, and the reconciler keeps how many replicas load has
added above a model's base. One or more requests waiting per ready replica,
held for a minute, adds one; ten minutes with nothing waiting removes one of
those again. It never goes above max or below the base, moves one step at a
time, and does not add while the last addition is still placing or starting
— on this fleet a replica is minutes of weights and gigabytes of memory, and
a spike should not leave a pile of replicas behind it. Decisions are logged
with the load behind them.
Nothing live changes until a model's max is raised above its min.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
SDKs and editors pace themselves from OpenAI's x-ratelimit-* headers, and
the gateway sent none: only Retry-After, and only on the refusal they would
have avoided. A limit can be set at several levels of the tenancy tree, each
counted in its own pool, so the headers report the one with least left —
the one that will refuse first — after this request's own charge, on
refusals and successes alike. The windows slide, so the reset figure is the
next minute boundary, when a whole window has moved on. A limit that is not
set is not advertised.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
"Why was this request slow?" had no answer. Nothing tied a request to the
replica that served it, the time admission held it went out in a response
header and was never kept, and the only per-request log lines were for
disconnects, early stream ends and upstream retries.
Every request now has an id: the client's X-Request-Id when it is a plain
string of reasonable length, otherwise one the gateway makes. It is set on
the response before anything else, so a request refused as malformed can be
named too, and forwarded to the engine. One access log line per request
carries it with the model asked for and served, the replica, status, time in
the queue, time to the first token, total time, tokens, and the key,
project and user.
Usage records gain the id, the replica, queue time and time to first token,
tried against a restored copy of the live store before being shipped: 591
rows kept, columns and index in place. `llmgw ctl request ID` reads one back
through the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A review of v0.10.4-rc.4, and a look at the live network, found:
- Alertmanager listening on every interface: from any node its API took
silences, with no login. Bound to loopback on the monitoring host, and the
monitoring README now installs it that way; Prometheus beside it is its only
client, and one Alertmanager needs no cluster port.
- Login and signup unbounded in memory. Each attempt runs argon2 with 64 MiB,
and nothing limited how many ran at once; the per-address throttle kept one
entry per address ever tried. Hashing now takes one of a few slots and
waits for one otherwise. The throttle expires entries whose window has
passed when it fills, and if it is still full it refuses new addresses:
evicting live ones instead would let a flood of made-up addresses reset the
throttle on the address actually being guessed.
- Metric labels taken from the request. A refused request counted the model
name it asked for, so any key holder could mint a series per request.
Requests are counted under the catalog's name, or "unknown".
- Two in code written this week. A SQLite backup made its snapshot with the
default mode in the listable backup directory, a full copy of the store
readable while the backup ran; it is now made in a private directory, and
the unit sets UMask=0077. And archive links were checked by name only, so
"a -> ." then "a/a/b -> ../.." passed and a file written through them landed
above the install directory, as root. Archives now unpack through os.Root,
and a link may only point at a file beside it — which every llama.cpp
b10991 archive, CPU and CUDA, was checked to follow.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CI's test job has failed intermittently — two of three runs on some days —
while every run here passed, including six repetitions on one CPU under the
race detector with eight packages at once. The difference is a shared,
loaded Linux runner, and the tests most exposed to it are the ones that
sleep a fixed time and then assert:
- admission: idle admission bounded at 50ms of wall clock against a 1ms
tick, and a queued request checked for after a 50ms sleep;
- scrape: a 10ms scrape timeout, which on a slow runner reports the engine
unreachable and then the violation a second time, in a test that counts
one;
- supervisor: log lines, three lines of output and a removed state file
each checked after a fixed sleep, and an exited worker bounded at 1.5s;
- agent: a heartbeat checked for after four intervals.
Each now waits for its condition up to a deadline of seconds, and the two
wall-clock bounds are set against what they exist to distinguish — the
admission queue's tick raised to ten seconds, the exited worker bounded
below the five-second grace it must not wait out — rather than against how
fast this laptop is.
Without the CI logs this cannot be proven to be the failure; these are the
tests that fail when the machine is slow, and they no longer do.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.
After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.
A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.
Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The admin listener has no login. Reaching it is meant to mean holding the
control plane, so it binds to loopback and an operator reaches it over an
ssh tunnel — and so does every page open in that operator's browser. The
API decoded JSON whatever the Content-Type said, so a page could POST a
text/plain model definition, no preflight needed. A model's profile env
reached an engine the agent runs as root, unfiltered: LD_PRELOAD in a
definition was code execution on every node. A page that rebound its own
name to 127.0.0.1 could also read pprof's heap, which holds API keys.
The admin listener now answers only when Host is an address or localhost,
refuses requests a browser marks as cross-site (Origin, Sec-Fetch-Site),
and takes changes only as application/json, which no page can send across
origins without a preflight nobody grants. ctl sends that on every change,
bodiless DELETEs included; curl and ctl are otherwise unaffected.
Profile env that changes what runs rather than how — the dynamic loader's
LD_*, PATH, Python's PYTHON* start-up variables, BASH_ENV and their like —
is refused when a model is defined, and dropped again by the engine adapter
before the process starts, whatever sent the profile. None of the live
catalog's models sets any.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The website was a file built by hand outside the repository. It is now a
template in internal/web: every gateway serves it at /about, without a key,
linked from the landing page's footer, and the published site is rendered
from the same template.
The fonts are vendored (Latin subsets, OFL) so the page sends nobody's
browser to Google. The proxy configurations it shows are copies of
deploy/proxy, kept equal by a test. A development build says so instead of
linking to release files that do not exist.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The account page makes a key for Continue, Codex or Cline and shows
what the tool needs with the key, this gateway's address and a served
model in it: Continue's assistant file, written to its own place under
~/.continue/assistants so the person's config.yaml is not touched;
Codex's provider and profile, added to ~/.codex/config.toml without
changing its default model; and for Cline, whose settings live in VS
Code's secret storage, the four fields to paste. Continue and Codex get
a one-line command whose script, served without a session, holds no
secret - the key comes from the environment, never a URL - keeps a copy
of whatever it replaces, and leaves the rest of the file as it was.
POST /v1/responses translates the Responses API, which Codex speaks, to
chat completions and back: instructions, messages, function tools and
Codex's free-form custom tools, their calls and outputs, and usage, as a
response object or as its stream of typed events. It goes down the same
path as any request - quotas, admission, routing, usage. Server-side
tools and stored responses have no equivalent and are ignored.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
While a reply streams the Send button is Stop, which aborts the request
- and with it the one to the engine - keeping what had arrived. Under
each reply: its tokens, from the engine's own count (the page's turn now
asks for include_usage), tokens a second, time to the first token and in
all. A settings fold takes a system prompt, sent before every turn, and
a temperature, which the gateway holds to 0-2 whatever the page sends.
The conversation, the prompt and the temperature are kept in the
browser's storage, so a reload puts them back - the model's turns
rendered again by the server as when they came - and "New conversation"
clears them. The gateway still keeps no conversation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its
GPUs reach each other over NVLink whatever node they are in. Each GPU
now reports its fabric from nvidia-smi -q - state, status, ClusterUUID,
CliqueId, bandwidth - at registration and once a minute after, since a
fabric manager restarting drops every card out at once; and each node
reports nvidia-imex and its channels, without which the domain's GPUs
cannot share memory across nodes. A single machine's NVSwitch, which
reports a cluster of zeros, is no domain.
A replica spanning nodes is placed inside one domain first, on nodes
whose every card is joined and whose IMEX is up, and its shards are
told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback.
/admin/nvlink draws each domain as a rack - the NVLink Switch band on
top, light running down a backplane, trays of GPU chips that glow when
joined, pulse red when not, turn amber on degraded bandwidth - and
lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink
matrix, each card's NUMA node and nearest adapter, and its domain and
IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics
llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The control plane asks one node on each physical fabric to run
ibnetdiscover every five minutes and parses the text itself, so a
format the parser has not met is a control plane upgrade, not one on
every node. From the map it works out the switch tiers (leaf, spine,
core, by distance from the adapters), which adapter is which node's,
which cables trained below the rate the rest of their kind run at, and
whether the rails line up: every node's Nth adapter on its own leaf.
A page per fabric (/admin/fabrics) draws it: switches in their tiers,
adapters fanned out under their leaves, cables coloured by rate and
bundled, degraded ones marching red with a marker, and the cluster's
own adapters live - down ports glowing red, and particles running each
cable as fast as its adapter is sending and receiving, from the
counters. It zooms, pans, fades what a hovered switch or adapter does
not touch, opens a panel of detail on a click, colours by rail, finds a
node, and refreshes every fifteen seconds. The data is also at
/api/v1/fabrics/{id}/topology.
Placement reads the map: a replica spanning nodes takes nodes of one
leaf together when it can, and nodes whose cable trained short last.
Static assets are named by their content, as the stylesheet already
was.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nodes are CPU machines, and nothing measured how busy one was: the
heartbeat carried GPU memory and disk, and llmgw_node_memory_bytes is the
total. The agent now reads /proc between heartbeats - CPU across all
cores, load averages, memory used and available, and each engine's
process group's cores and resident memory - and sends it with the
heartbeat. Nothing new runs on the nodes and Prometheus needs no new
target: the control plane exports it with the rest of the fleet, as
llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load,
llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes,
llmgw_shard_cpu_cores and llmgw_shard_memory_bytes. A measurement more
than a minute old is left out rather than repeated.
The cluster page's node table shows CPU and RAM as bars, and each node
links to a page of its own: current CPU, RAM, load and disk, the engines
on it with their cores and memory, and an hour of CPU, RAM, load,
per-model cores and memory and queue depth from Prometheus. It refreshes
in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A reply streams as text as before; once it ends, the server renders it
with the model card's rules - raw HTML dropped, dangerous links emptied,
images as links - and the page places that. Headings, lists, tables and
code blocks read as they were written.
Maths outside code is set aside before the markdown, so its underscores
and asterisks are not taken for emphasis, and marked for the page to
typeset. KaTeX no longer searches the reply for delimiters, which took
"$5 and $10" for a formula. A reply that cannot be rendered stays
plain text, with the old search.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A message to a model that was not running waited 45 seconds and came
back as "the model is loading; retry shortly". The page now asks for
the model with /chat/wake, shows where it stands from
/chat/model-status - waiting for a node, downloading with bytes and a
percentage, starting the engine - and sends the message when it answers.
Also passes the nodes' llama.cpp build to the pages, which the mirror's
checks for loadable weights needed and were never given.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A member of an organization can be its manager (users join --org X
--role manager, or the star on the people page). /org shows a manager
their organization: usage over time by team, project or person, its
teams and projects with limits and 30-day spend, and its members, each
once with every place they belong. Read-only; administrators see every
organization. Only what was spent in the organization's projects is
shown — UsageBucketsIn filters by where a request was spent.
Found on the way:
- Usage by org, team and project, and the quotas page's spend, joined
through api_keys and so dropped every chat-page request, which has no
key row. They now read usage.project_id; keys no longer move between
projects, so the two agree.
- /org was not exempt from API-key auth and answered browsers with a
JSON error. A router test now drives the session pages through the
middleware; it fails without the fix.
Membership queries, org-scoped usage and the reworked joins pass the
PostgreSQL store tests on 16.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A person was in exactly one project, chosen at approval, and could not
use the gateway before that. People belong to several teams and to
none, so:
- memberships: a person is a member of any number of organizations,
teams and projects, or of none; a member of a team or organization
may use every project in it. users join/leave, and the people page
adds and takes out from either side. Approval no longer needs a
project.
- personal keys: a key may have an owner and no project. /profile asks
what a new key spends against; keys create --for does the same.
auth.personal sets default limits for personal use or turns it off.
- limits for people: --scope user holds a person across every key,
project and the chat page; --per-member on an org, team or project
gives each member their own share there. Resolution folds the tree,
each per-member limit for the key's owner, and the owner's own, each
in its own pool; the quotas page shows and sets both.
- the chat page asks what to spend against and checks it per message.
- usage records who spent it, so a person's budget survives a restart,
personal use and chat included; restore no longer drops rows with no
project.
Leaving a project revokes the keys made there unless another membership
still reaches it. A personal key keeps no threads or notes, which are
per project, and is refused rather than dropped into the anonymous pool.
Migration 0010 turns each person's project into a membership, adds
usage.user_id and backfills it, and makes api_keys.project_id nullable.
SQLite cannot drop NOT NULL, so the table is rebuilt; that needs foreign
keys off, which only works outside a transaction, so migrations marked
for it run on one connection with foreign_key_check before commit.
Tested on a populated SQLite store with claims, and on a copy of the
cluster's PostgreSQL store: 5 people became members, 17 keys and 3
claims intact, 145 of 577 usage rows and all 69 chat rows attributed.
The PostgreSQL store tests, including new membership cases, pass on 16.
users move is gone; join and leave replace it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Seeing what a node did meant ssh through a jump host, to a machine whose
address nobody remembers, to read a journal only root may open. When an
engine install failed on a new node today, the reason was one line — and it
was four steps away from the page that said the install had failed.
Each node's row on the cluster page now links to its logs: the agent's own,
and every engine running there, chosen from the list the node itself sends.
The page can follow, refreshing every five seconds, and is on both the admin
listener and /admin. `llmgw ctl logs NODE [--source SHARD] [-n N]` reads the
same.
They travel over the stream the agent already holds, so they need nothing
the control plane does not already have: a request goes down, the lines come
back, and a reply that does not come in ten seconds is an agent too old to
answer. The agent keeps its last 2000 lines in memory as well as writing them
wherever it did before; the engines' last few hundred lines were already kept
and sent only when one crashed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The picker listed what had an endpoint, and a model at min 0 has none until
somebody asks for it. On this page a message is the asking, so an on-demand
model could never be started from here: mistral-nemo-12b-instruct sat in the
catalog, served by nothing, and absent from the one place someone went to
try it.
Such a model is now offered, marked "not loaded", with a note saying the
first message fetches the weights and starts the engine, and that until it
is up the reply is "the model is loading". A model stopped outright — max 0 —
is still left out, because nothing would load it and offering it would only
produce an error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Usage could be read as a total — per key, per user, per model, per group —
and the cluster's own charts covered the fleet's hardware. What nobody could
see was when a group used it. "Research doubled its traffic three weeks ago"
is not a question a column of totals can be asked, and it is the question
somebody asks before a quota conversation.
/admin/usage draws requests, tokens and spend over a chosen window, grouped
by any level of the tenancy tree or by person. Person is not a level of that
tree — one person holds keys in several projects — so the grouping is its
own type rather than store.Scope with a fourth value nothing else could use.
Three things learned from the charts already here. Buckets are gap-filled
per group, because a line drawn straight across an absence claims traffic
that did not happen, and because the chart sums its tail by timestamp and
can only do that if every series shares a clock. Series are ranked by total
rather than by peak — a counter's question is "who used the most", and
ranking by peak would colour the team that made one burst over the team that
ran steadily all week. And the spend chart is replaced by a sentence when no
model has a price, rather than drawn flat along the bottom.
The bucket query groups by ordinal and was planned against the cluster's
PostgreSQL before being believed, with a test alongside the others that run
there: three reports have shipped working on SQLite and failing on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The page imposes its own ceiling on a turn, so a playground request cannot
occupy a replica for ten minutes. It sent that ceiling regardless of the
model, and a model declaring a smaller max_output then answered every turn
with "max_tokens is 2048, but this model allows at most 1024" — the gateway
refusing its own page, over a number the person never chose and has no way
to change.
The ceiling is the page's to impose on itself and the model's limit is not
negotiable, so it sends the smaller of the two.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An endpoint advertises every name a client may address it by — the model's
own and all of its aliases — because that is what the router matches a
request against. The picker took those strings as they came, so one model
appeared as several: "coder", "coder-7b" and "qwen2.5-coder-7b-instruct"
were three choices, and choosing between them is a question with no answer.
Each name is now resolved through the catalog and the model listed once,
under its catalog name — the one that also appears in the model list, the
usage table and the logs.
/v1/models still lists the aliases separately, which is deliberate: there a
client is discovering which ids it may send, and every alias is one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The price table is written against the catalog. The name reaching the
accountant was the one the caller typed, so every request made by an alias
missed the table and fell to the default price — the same request charged
two different rates depending on which of a model's names the client
happened to be given.
With no default price that is zero, which is silent. With one it is a wrong
number, which is worse: a budget spent at the rate of whatever the default
says rather than at the model's own.
The router already resolves the alias, two lines above, to decide whether
the key is allowed the model at all. It now passes the resolved name to the
accountant as well, in the ordinary path and in the budget fallback.
The test charges a real request through an aliased name and compares the
figure with arithmetic it states itself, because the first version of it
asserted "more than the fallback" and passed with the bug in place.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A team was given a budget of a million, the quota page showed it, and after
sixty requests it read zero percent used. Not because the counting was wrong:
cost comes from a price table, no model had a price, and every request
therefore cost nothing. The number was a limit in appearance and bound on
nothing.
An empty price table is the ordinary case — self-hosted models have no
per-token price, which the schema already says. Showing somebody a budget
without mentioning it is not.
The page says so, and says it precisely: rate and token limits are unaffected,
because they count requests and tokens rather than money. A warning that read
as "no quota works here" would be worse than the silence it replaces — those
two were enforcing perfectly well, as twenty served and ten refused had just
shown.
AnythingPriced is on the accountant rather than the price table, because what a
caller is asking is whether the budgets it is about to display mean anything,
and the accountant is what would enforce them. A model priced at zero is not
priced; a fallback price prices everything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard charted what the cluster had been billed for — requests, tokens,
spend, all from llmgw's own store — and nothing about whether it was healthy.
Saturation, queue depth, cache occupancy and replica counts were numbers in a
table at the instant the page loaded, with no way to see that one of them had
been climbing for an hour.
Four line charts now, read back from wherever the metrics were scraped to.
Read back rather than kept twice. llmgw publishes these already; what a gauge
cannot do is remember, and a ring buffer in this process would be a rival copy
of the same numbers with its own retention, lost on every restart and
disagreeing with the one an operator has on screen elsewhere. monitoring.prometheus
is empty by default and the panels are then absent rather than empty.
One axis per chart, always. Two measures of different scale belong in two
charts: a second y-axis lets any two lines be made to cross wherever the author
likes and the reader cannot tell it happened.
Four colours, handed out in fixed order, never cycled. A fifth series is summed
into "other" and drawn in grey — dropping it would make a chart that does not
add up, and inventing a hue for it would put two lines on the page that nobody
can tell apart. The fold is by timestamp, not by position: the tail's series do
not share sample times, and adding them by index slides points sideways onto
each other's clock.
A legend for every chart with more than one line, because colour alone is not
an encoding a reader can rely on.
A flat zero is drawn along the bottom rather than left blank: nothing happening
is a real answer, and an empty frame reads as a broken chart. A single sample
is not a line and says so instead of putting a dot in the corner.
One failed query fails all four panels. Three drawn and one blank invites
reading the blank one as "nothing is happening there".
NaN is dropped rather than plotted. Prometheus sends it as a string for a
sample it has no value for, Go's ParseFloat accepts it happily, and one NaN
coordinate is an SVG path that does not draw at all — taking the rest of the
line with it.
Axis maxima are rounded to a number a person would pick. An axis topping out at
0.8347 makes every gridline unreadable, and the shape is what a chart is for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A gateway serving nine models offered nine names and nothing to choose between
them. /v1/models is the only view most clients ever have, and it listed ids.
Models carry a description now: written when the model is added, shown on the
dashboard and sent to clients in /v1/models.
Written rather than derived. A registry offers a pipeline tag and a wall of
README, and neither answers the question somebody picking from a list is
asking — which of these should I send this request to. "General chat, strong
Swedish; slower than the 8B" is not in any metadata field.
It is optional, and absent means absent rather than an empty string in the
response: a model nobody described should read as undescribed, not as described
with nothing.
On the dashboard it sits under its row rather than in a column. A sentence in a
table cell either wraps every other column into a stripe or gets cut off.
Co-Authored-By: Claude Opus 5 (1M context) <moreply@anthropic.com>
Fetching a model put it on the mirror and left the operator to hand-write a
catalogue entry and run `ctl model add` on the control plane — for a source the
page was already displaying, with an engine decided by the file extension it
had just shown them.
A present row now offers to serve it. The form opens on that row with the name,
the mirror's own URL and the engine filled in, and asks for the two things
nobody can infer: the context pool and how many replicas.
What it refuses is the point.
A version of several files is refused with the reason. A catalogue entry names
one source, so an entry for a sharded model is an entry for one shard, which
loads as a broken model rather than failing to load. Nothing said so before —
the mirror happily holds a four-shard model that no catalogue entry can
describe.
Safetensors on a fleet with no GPU is warned about before anything is added.
This is the failure we hit today: fifteen gigabytes of shards arrived, and only
then did looking at a node show no card, no vLLM and no SGLang. The control
plane is not told which engines a node has installed, so the check is on cards,
which it does know — a warning rather than a refusal, because a card can be
added tomorrow.
A name already declared in configuration is refused here too, matching what the
model service would say anyway, but before the form is filled in rather than
after.
Without a store there is no serve action at all: a model added at runtime would
not survive a restart, and offering it then refusing is worse than not offering.
The mirror page's dependencies became a struct. Three are optional interfaces
and two are the same shape, which at a call site is a swap nobody notices until
a page shows the wrong thing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mirroring anything meant knowing a repository existed, knowing which of its
thirty files were the weights, and assembling a resolve URL by hand. The mirror
page had a text box for that URL and no way to find one.
It now searches Hugging Face and lists a repository's weight files with their
sizes and published checksums, each with a button that fills the fetch form.
Only weight files. A repository is mostly a README and a tokenizer, and listing
those buries the four entries somebody came for.
The size shown is the LFS size, not the plain one. For an LFS-stored file —
every weights file — the plain field is the size of the pointer, so the other
one reports a gigabyte model as 133 bytes, and that is the number somebody
would then plan disk against.
The source URL is pinned to the commit the listing came from rather than to a
branch, so what the mirror fetches is what was on screen. A repository
re-uploaded under the same branch otherwise leaves two different sets of
weights with identical provenance and no way to tell them apart. The version
defaults to that revision for the same reason.
The registry's own failures are reported on their own line. It is somebody
else's server on the other side of the internet, and it being down says nothing
about the mirror — which is what the page is for and has to keep working
regardless. With no registry configured there is no search box at all, rather
than one that can only time out.
Gating decodes from a bool or a string, because the field is documented as the
first and is the second in practice: strict decoding fails the whole response,
so one gated model would hide the other nineteen in a page of results.
And a bug this made obvious: downloads were keyed by model and version, so the
second shard of a sharded model was refused as "already fetching" until the
first had finished. A fifteen-shard model had to be pulled one file at a time,
each waiting on the last. Keyed by file now — which is what the single-flight
was ever protecting, since the race is two writers on one .part. A version made
of several files reports as the worst of them: it is not present until every
shard is, and one shard failing makes the version unusable however many others
arrived.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A quota could be enforced and reported, and set nowhere but a shell. Nothing in
the browser showed one at all — not what a team may spend, not what it has
spent, not that it had no limit. In practice that means quotas nobody sets.
/admin/quotas lists the whole tree, whether or not a group has a limit: the one
without a quota is the one most likely to need one, so it cannot be the row the
page leaves out. Spend over the last thirty days sits beside the budget, with
the fraction used.
Setting a quota here is a change of position. It was kept to the shell because
raising a budget is a financial act and a stolen admin session should not
perform one. That argument holds for exactly one row on this page: resolution
takes the most restrictive value along the chain, so raising a team's or a
project's limit cannot exceed what the organization allows. Only an
organization's own limit is bounded by nothing. Every change is logged with who
made it and what it was before.
Per-key limits stay on the CLI. A key belongs to a person and is already
bounded by its project; letting an admin session reach individual keys would
put every caller's throughput one click away, and the page refuses a key scope
rather than rendering it and failing later.
A limit an ancestor already caps lower is marked as written-but-unreachable.
Resolution means a project given more than its team is not a bigger budget, it
is a number nobody will reach — and without saying so the page would show it as
though it applied, with the first sign being a refusal that disagrees with the
screen.
The group is chosen, not typed. Ids became random in the previous commit so two
organizations could both have a "platform" team, which also made them
impossible to remember and a typo impossible to notice: it would write a quota
onto nothing and report success.
A blank field is no limit of that kind, not zero — zero is a limit of nothing,
which refuses every request. Clearing removes the row rather than writing nulls
across it: both resolve the same, only one tells a reader nobody has set a limit
here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two things that were only ever confusing, and one of them stopped a command
working on the cluster.
The store type was named SQLite and kept the name when PostgreSQL was added, so
OpenPostgres returned a *SQLite and all seventy-odd methods on the production
store were spelled after the database it was not running on. Nothing broke;
people did. Earlier today I added three methods to *SQLite for a table that on
this cluster only ever lives in PostgreSQL, and the name did not once look
wrong. It is a field, which is all it ever was.
`llmgw users` and `llmgw keys` defaulted --store to a SQLite path. On a
deployment whose store is PostgreSQL every one of them failed with "unable to
open database file" while the right answer sat in auth.store in the config file
beside them; the conclusion an operator draws is that the tool is broken, and
they are not far off. They now ask the server's own configuration before
falling back, and the error names which of the three sources chose the address
— the failure was never the path, it was which one picked it.
The single-machine default still applies last, so a deployment with no config
file keeps working with no flags.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/admin/mirror, behind the same admin session as the other admin pages. What is
held, what is arriving with its byte count, what failed and why; start a fetch,
retry a failed one, remove a version.
Not mounted at all without a mirror configured. The right page for a deployment
with no mirror is no page, rather than an empty one pointing at nothing.
It reloads itself only while something is downloading. A reload at any other
moment takes with it whatever is half-typed into the fetch form, and there is
nothing to watch when nothing is moving.
A failed row offers Retry, not Remove. A fetch that failed left nothing on disk,
so the remove could only answer "the mirror holds nothing for this" — a button
whose single outcome is a confusing error. Starting the same download again is
what somebody looking at that row wants, and the mirror allows it: only a
download still running is refused.
An unreachable mirror is said outright instead of rendering as an empty table.
The two look nearly identical and mean opposite things: one is a mirror waiting
to be filled, the other is a machine to go and look at.
Dashboard takes NavExtras rather than a second bool. Two adjacent booleans at a
call site get swapped silently and are discovered by somebody clicking the
wrong link.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The choice is unchanged and now better argued: a deployment does not announce
which language its notes are in, so any stemmer picked here is the wrong one
somewhere — and applying the wrong one does not fail loudly, it quietly stops
finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.
Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.
So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.
There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A path is still SQLite and a postgres:// URL is PostgreSQL, so every existing
configuration keeps working untouched and moving is one line changed rather
than a new setting to find. An address of the form @/path/to/file reads the
real one out of that file, which is how a DSN carrying a password stays out of
a configuration that gets copied, diffed and pasted into issues.
The driver is pgx, which is pure Go. That is not a preference: the whole binary
is built with CGO_ENABLED=0 so it runs on a node whose libc is older than the
builder's, and a cgo driver would take that away from every deployment,
including the ones still on SQLite.
Tested against a real PostgreSQL, which found two things nothing else could.
The schema_migrations table is created in Go rather than in a migration file,
so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in
PostgreSQL, and a millisecond timestamp does not fit, so the very first
migration failed to record that it had run. And the memory migration uses an
FTS5 virtual table with three triggers keeping it in step, which has no
counterpart: PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget.
That leaves exactly one query that is not the same sentence in both dialects —
the note search, MATCH against @@ — and it is written out twice rather than
hidden behind a shared spelling that would mean something subtly different in
each. Everything else is one set of queries.
The text search uses the 'simple' configuration rather than English. These are
an agent's own notes, in whatever language it chose, and stemming English
rules over Turkish would quietly stop finding things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
