deniz/llmgw
1
0 Fork 0
forked from ilker/llmgw
Code Pull Requests Activity
Files
a8bb417618d269f2c2f35d901a82ae2cfefda01e
llmgw/CHANGELOG.md
T
Ilker ManapandClaude Opus 5.5 a8bb417618 The guardrails' judge has its own command, and its trouble is shown 
llmgw ctl guard judge status lists each judge model the policies ask:
in the catalog, as a judge, replicas ready, its last failure to answer, and
what is wrong in words. enable serves the missing one through model serve -
Llama Guard 3 1B on CPUs, role guard, under the name the policies give it -
with --replace for a model of that name that is not a judge; the console has
it under Do. GET /api/v1/guard/judges, and the summary's judges, put the
problem on ctl status and the console's Overview.

A Llama Guard file served from the mirror page is added as a judge, under
the name the policies ask, rather than as a model clients use the policies
never find. ctl says plainly when the API answers with a page.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-06 10:47:37 +02:00

3785 lines
209 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Added
- **`llmgw ctl guard judge status` and `enable`**: each judge model the
policies ask, whether it is in the catalog, as a judge, with replicas
ready and answering — and serving the missing one, Llama Guard 3 1B on
CPUs under the name the policies give it, with `--replace` for a model of
that name that is not a judge. No more `server init` to put a judge back.
Also **Do → Serve the guardrails' judge** in the console.
- **A missing or broken judge is said where it is looked for**: `llmgw ctl
status` and the console's Overview list each judge with a problem — not in
the catalog, not a judge, no replica ready, or failing to answer — and
what that does to requests (refused or unjudged).
### Fixed
- **A Llama Guard file served from the mirror page is the judge**: it was
added as a model clients use, under whatever name the form had, so the
policies never found it and refused everything they judged
(`guard_unavailable`). It is now added as a judge, under the name the
policies ask.
- **`ctl` says so when the API answers with a page**: a command a control
plane does not know — the leader on an older release, through a standby —
failed with "invalid character '<'"; it now says which and why.
## v0.12.0 — 2026-10-06
A minor version, because where the configuration lives changes: everything in
server.yaml that is not one machine's own is now the cluster's settings, kept
in the store, the same on every control plane and in force within seconds of
a change. Nothing needs doing by hand to upgrade. The first control plane to
start on v0.12.0 brings its server.yaml's cluster keys into the store, with
the baseline guard policy when the file had none. From then on, change them
with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
ctl upgrade` does, the leader last): until the leader runs v0.12.0,
`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
the firewall switch.
### Changed
- **The cluster's settings live in the store, and every change is in force
within seconds.** Everything in server.yaml that is not one machine's own
(guardrails, limits and prices, routing, engines, notifications, the
mirror's policy, declared models, offline, federation, facility budgets) is
a versioned document in the store, the same on every control plane.
`llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
console's Settings screen edits it. A change is checked, kept with who made
it and why, and applied: guardrails live, anything else by every control
plane starting again on it, standbys first and the leader last. A change to
a machine's own keys in its server.yaml is noticed and applied the same way.
No "restart now?" any more, and no second copy to keep in step. The first
start against a store brings in the file's cluster keys, with the baseline
guard policy when the file had none, so an upgraded cluster gets its
guardrails; after that, those keys in a server.yaml are ignored, with a
warning.
### Added
- **llmgw's firewall can be turned off and on for the whole cluster**:
`llmgw ctl config set firewall.enabled false`. Every node's agent takes its
table out within seconds, and each control plane's new
`llmgw-firewall-sync.path` does the same there; `true` puts the rules back
as they were. `firewall_networks` changes reach the nodes the same way.
- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
configuration, the authority, secrets, state, users — after listing it and
asking for the machine's name; `--models`, `--store` and `--all` take
weights, the store's database here, backups and the binary too. Also
**Do → Wipe this machine** in the console.
### Fixed
- **The mirror is given an address the nodes reach**: `server init` and
`mirror init` took the machine's first address for the mirror's `--base`,
which on a machine with Docker could be 172.17.0.1 — a node then asked its
own bridge for every model and was refused. They now take the source of
the default route, else the first address not on a container's or a
bridge's interface. And `node add` has the node reach the mirror's files
before it is added: one that cannot fails the checks, naming the address
and what to change.
- **A standby that becomes the leader reads the store again**: models added
or removed, engine builds chosen and domains held through the leader before
it, rather than placing on what it read when it started.
- **A node on another network joins**: `node add` set up the node's firewall
but left the control plane's to the networks it was installed with, so a node
on another subnet was installed and then never heard from (lost). It now lets
the node in on the control plane it runs on — its subnet, or its address
when a router is between them — and says so; a network already let in
changes nothing.
## v0.11.4 — 2026-10-05
Two control planes, run in anger on the reference cluster.
### Added
- **The console sets up a second control plane**: Do → Add a control plane
asks for the machine and a floating address and runs `server add`.
- **Who leads, everywhere.** `ctl status` and the console's Overview have a
`CONTROL` line: which control plane leads, which stand by, one that does not
answer, and whether the floating address is on the leader.
- **A standby's console is the leader's.** Its admin API passes `/api/` to the
leader over the agent port, with the cluster's certificates both ways, so
`sudo llmgw` and `llmgw ctl` show the same cluster on either machine. Control
plane certificates are now issued for client use too; `server add` reissues
older ones.
- **`ctl upgrade` in waves.** Nodes are upgraded one, then two, four… up to
`--parallel` (20) at once, so hundreds take minutes; a wave that does not
come back stops it. The other control plane is upgraded first, over ssh.
- `server add` says when the mirror's `--api-allow` would refuse the new
control plane.
- **Guardrails in the console**: a view of the policies and what each checks,
whether their judge models are serving, where they are attached (attach
and detach from it), and what they did in a window. `GET
/api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
what each policy does.
- **A standby's portal is the leader's.** Its client port passes everything
but `/health`, `/ready` and `/leader` to the leader, and its admin port the
dashboard as well as the API; a proxy left pointing at a standby showed an
empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
- `ctl model serve --tools` says in the catalog that a model makes tool
calls, for agents and editors.
### Fixed
- **Continue's agent did nothing.** The configuration the portal writes told
Continue every model uses tools, and nothing of its context: a model whose
calls came back as text stalled the agent, and prompts over a CPU
replica's 4096 tokens were refused. It now says `tool_use` only for a
model whose catalog says tools, and the context and longest answer the
gateway allows.
- `ctl model serve` gave llama.cpp four slots whatever was asked: a
`--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
refused what the gateway admitted. Slots are now what the pool holds of
the context. The longest answer grows with the context (a quarter, from
1024 to 8192) rather than being 1024 for every model.
- `ctl upgrade` upgrades the leader last wherever it is run; run on the
standby, it upgraded the leader first and the leadership moved twice.
- **Failover took 40 seconds; it takes a few.** A stopping control plane held
every agent's stream for the thirty seconds of a graceful shutdown; it now
gives up its lease first and lets the agents go at once.
- A control plane that lost its lease while running kept its agents, leaving
the new leader with no nodes. It now lets them go.
- An agent's reconnect backoff carried over from one outage to the next, so
each failover made the next one slower; it starts again after a session that
worked.
- The `CONTROL` line named no floating address.
## v0.11.2 — 2026-10-05
### Routing by what each vLLM replica's cache holds
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
publishing its KV cache events (`--kv-events-config`) to a port on
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
computed outside Python. The events never leave the node.
- The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new `KVEvents` agent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed.
- The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes.
- For a model with `kv_events`, the router asks one of its replicas to
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (`router.prefix`) or to load.
- The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
though it published nothing.
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
- Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
### Federating clusters: another llmgw cluster as a provider
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
makes another llmgw gateway a peer. It is checked first (its `/ready`,
timed, and its `/v1/models` with the key) and kept in the store (migration
0029, `federation_peers`) with the models it offers.
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
follow the peer's own list every `federation.interval` (10s). A model the
peer drops goes, unless a model here still falls back to it: then it is
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
The peer's aliases and its own peers' models are not taken.
- A peer whose `/ready` fails, which refuses the key, or which does not answer
is taken out of routing until it answers again. `/ready` now counts only a
gateway's own endpoints, not its peers'.
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
overflow can go to a peer's model.
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
have passed through (each cluster's id is kept in the store). A request that
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
request came from is not sent it.
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
org=,team=,project=` (ids); off by default.
- In-cluster residency and `no_store` keep requests off peers, by name and as
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
as the replica.
- `ctl federation list|remove|sync`, a Federation view in the console,
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
API `/api/v1/federation`, and a "Federating clusters" section in
docs/install.md. New `federation:` block in server.yaml (`name`,
`interval`).
- A response header the gateway already set (X-Request-Id,
x-llmgw-served-model) is no longer repeated from the upstream's.
### Facility-aware placement: racks, PDUs, holds and power caps
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
is kept in the store (migration 0031, `node_labels`), wins over the agent's
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
A node with no location places as before.
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter.
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
restarts (migration 0031, `facility_holds`).
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
profile's `power_watts`, or its cards at the power limit they report
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks.
- **Seen everywhere it matters.** `ctl explain-placement` names the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
- Docs: `docs/facility-aware-placement.md` says what is built and where it
departs from the design; `docs/install.md` and `docs/console.md`.
### SGLang measured, checked and benchmarked as itself
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
SGLang was started without `--enable-metrics`, so it served no `/metrics`
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
prompt and generation token counters and `sglang:num_requests_total`, as
0.5.21 publishes them.
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
`--help` and checked SGLang's flags against Python's. It now asks
`python -m sglang.launch_server --help`, in the environment SGLang was
installed into (or `--bin`).
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
abbreviation of `--tp-size`; it is given `--tp-size`.
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
key given for `--target` — a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at `--metrics`' address (or given
with the new `--metrics-key`); metrics that cannot be read — refused, or in
another engine's names than `--engine` — fail the run before any load,
saying why; and a step whose metrics went unread shows `-` and the reason
rather than zeros.
### Reusing answers
- **A response cache, per project, off until asked for.** `llmgw keys cache
--project P --ttl 1h` has the same request — same project, endpoint,
model by its catalog name, and body with `stream`, `stream_options`,
`user`, `metadata`, `store` and `service_tier` left out and its keys in
any order — answered from the gateway's memory instead of the model, for
the TTL. `--off` stops it; without either it says how it stands. The
console has it as **Cache answers…** on a project.
- **Only answers that come out the same are reused:** chat and completions
asked at temperature 0 or with a seed, and embeddings, rerank and score.
`--any` reuses sampled answers too. Only a whole 200 from the model asked
for is kept — not an error, a fallback's or a canary's answer, one a
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
- **Streams both ways:** a streamed answer is kept put back together into
one completion (tool calls and reasoning included), and a kept answer is
streamed as chunks — with the usage chunk if the client asked — to a
client that streams. `/v1/messages` and `/v1/responses` get it too.
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
- **Policies still apply:** limits and the input guardrails run before the
cache is looked in, the output policy judges a reused answer as it would
a new one, and a request under `no_store` or continuing a stored
conversation is never served from it nor kept in it. A project's answers
are never looked up by another project's keys.
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
marked `cache_hit` on the usage record (migration 0027), and shown by
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
together in each gateway process, least recently used dropped first;
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
`llmgw_cache_bytes` and `llmgw_cache_entries`.
### Offline sites and Kubernetes
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
collects what a cluster would otherwise download — llmgw's release for
each `--arch` with its signed checksums, every llama.cpp build for those
architectures (or the `--variant` ones) with the checksums GitHub
publishes, the starter model and guardrails' judge, and any `--model
name/version=hf:owner/repo/file` pinned to the registry's commit and its
checksum — each checked as it arrives, into a directory or one `.tar`
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
bundle. Run again into the same directory, it downloads only what is
missing. `llmgw bundle show --verify` reads one back.
- **`llmgw bundle import`** checks every file against the manifest and
llmgw's release against the project's signing key, and only then puts them
into the mirror — releases and engines under `_llmgw/`, models where the
mirror keeps models, with their checksums — so a damaged bundle leaves the
mirror as it was; a different file under a version already there is
refused. On a control plane it also records the engines' checksums in
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
does that alone, for a mirror on another machine.
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
outside: nodes install llama.cpp from the mirror, each file held to the
checksum the control plane sends them over the agent's authenticated
connection — never one from the mirror — and `releases` defaults to the
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
from it; no registry is asked about newer weights. A release may now come
over plain HTTP when the build trusts a signing key, since the signature
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
be unsupported on an offline node rather than attempted.
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
fields on the setup form: installs a control plane offline — the bundle
into the mirror, the checksums trusted, `offline:` written to server.yaml,
the starter model and judge served from the bundle or skipped when it does
not carry them.
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
it lists everything that does, and what an offline cluster changes.
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
port's certificate and the admin token from Secrets, the API and metrics
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
and ServiceMonitor, `config` laid into server.yaml, and the offline
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
### The admin API described, and a Terraform provider
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
and query parameters, and the schema of each request and answer with the
meaning of each field, in
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
every control plane at `GET /api/v1/openapi.json` (with the token, like the
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
in it. Generated by `go generate ./internal/docgen` from a route table in
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
the fields' comments as descriptions; a test fails when an endpoint is
served and not in the table, another when the committed document is stale.
Checked with openapi-spec-validator and openapi-generator-cli's validate.
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
what a request must carry — an address in `Host`, `application/json` on
every change, even a `DELETE` — and some calls with `curl`.
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
force, with the defaults the catalog filled in. The listing has only its
state and its profiles' names.
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
own so the gateway does not carry its dependencies. `llmgw_model` (a model
added at runtime: scaling, class, owner and adapters change in place, the
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
against the OpenAPI document, endpoint by endpoint and field by field, so a
change to the admin API that breaks it fails its tests; CI runs them.
Built and installed by hand for now: see its
[README](terraform-provider-llmgw/README.md).
### Tracing beyond the router
- **The control plane traces what it does to the fleet.** A reconcile pass
that does something is a `reconcile` trace — passes with nothing to do are
not traced at all — with what it did beneath it: `scale up` with `place`
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
candidate turned down and why, as `ctl explain-placement` would print it),
`scale down`, `drain replica` and `stop replica` each with its reason,
`evict`, `preempt` with its victims, `reap replica` with the error and
when the model is tried again, `restart replica`, `restart refused`,
`node unreachable` and `node lost`.
- **A replica's start is one trace, down to the node.** `replica start`
stays open from placement until the replica serves, fails or is stopped,
with each shard's progress as an event and how long it took. The command
to each node is a `send start` span whose context travels in the command
(a new `trace` field on the agent protocol's `ControlMessage`), and the
agent continues the same trace: `start shard`, with `fetch weights`
(local, cached, downloaded, or downloaded by another shard; a `download`
with its bytes and where a resume began, the source's credentials and
query left out), `launch engine` (and `wait for rendezvous` on a worker
of a multi-node replica), and `wait for engine` with how often the engine
was asked and its last answer. "Why did qwen take four minutes to start"
is that one trace.
- **Stops, drains and adoptions on a node** are in the control plane's
trace too: `stop shard` and `stop engine`, saying whether the engine had
to be killed; `drain shard`; and `adopt` when a registering node reports
engines already running, with how many nobody claimed. An engine install
is an `install engine` trace on the node.
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
`serve model` and `update model`, with a span per step — registry,
mirror, check, catalog, placement, smoke test, replicas — and from the
placement step a link to the replica's own start.
- **The agent has a `tracing` block** in `agent.yaml`, the same as
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
when unset). Point both at one collector. A node follows the control
plane's sampling for every start it is sent. Traces from a node carry its
id as `service.instance.id`, and a control plane's its host name.
- With no endpoint nothing changes: every span is the no-op one, and a
command carries no trace context.
### Quality suites, and new weights held to one
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
and scores its answers: per case, then the cases weighed into one score, 0
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
and one way to score it — `exact`, `contains`, `regex`, `number` within
a `tolerance` (the last number in the answer), `json_schema` (the answer
read as JSON, in a code fence or a sentence or not, held to a schema), or
`judge`: a rubric another model on the cluster grades 0 to 1. A
reasoning model's `<think>` is not part of its answer.
- **Run on the control plane**, through the router as its own requests,
as a replay is: past admission, quotas, accounting and the guardrails.
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
`deploy/eval/`, sized for small models. `--suites` lists them.
- **Kept** in the store (migration 0028): every run with its model, the
repository and revision of the weights that answered, the suite's name
and hash, its score and each case's answer and reason.
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
`model promote` take `--eval SUITE --min-score N`, or a model carries one
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
A promotion asks the canary first and leaves it in place below the score;
an update runs the new weights first as a canary given none of the
model's requests, asks it, and promotes it or takes it out again, so the
old weights serve until the new have passed. The job's `eval` step says
the score; `--skip-eval` moves once without the model's suite.
- **The console** has **Ask it a quality suite…** (the score counted in as it
comes), **Its quality runs** and **Hold its updates to a quality suite…**
on a model's menu.
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
handler cut off at 64 KiB, by the audit log's reading of it.
### MIG instances as GPUs of their own class
- **A card in MIG mode is placed as its instances.** Discovery reports a card
in MIG mode as its MIG instances instead of the whole card, each a device of
its own with its UUID (`MIG-…`), its own memory, and a class named from the
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
any other, and placement puts a replica on a free instance of it. Both
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
instances; `docs/install.md` has a MIG section on making them.
- **One instance a replica.** A process opens only one MIG device, and
instances have no NVLink or peer-to-peer between them, so a profile on a
MIG class asking for more than one GPU, tp or pp above 1, or more than one
node is refused when the model is added, with that reason; placement
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
for a model one holds and never several together.
- **Engines are given the instance by UUID.** Placement numbers card 0's
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
shows the device as `0/mig2`.
- **Memory per instance, no made-up utilisation.** Each instance's memory,
and the processes holding it, are its own — read per GPU and compute
instance from nvidia-smi's report, so a foreign process on one instance
keeps that instance out of placement and not its neighbours. Temperature,
power and wear are the card's. No driver reports utilisation per instance,
so none is published for one: the console says `n/a on MIG`, and
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
- **A card in MIG mode with nothing carved is kept out of placement**, with
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
is on.
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
of a card in MIG mode with no instances, of GPU instances with no compute
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
command for each.
- `ctl nodes` counts a node's devices by class when it has MIG instances or
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
GPU list shows instances under their card with their own memory and the
card's heat and power, and a node's page lists its classes with counts.
`ctl status` and the plan count each device under its own class, where a
node's cards were all put under its first class before.
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
### Engines and the mirror
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
installed it: name its interpreter in the node's `agent.yaml`
(`engines.trtllm.python`) and the node offers it while that interpreter
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
`--pp_size` and served name, through a few lines of its own that make it
demand the gateway's key on every path but `/health` (trtllm-serve has no
key option), turn on the Prometheus endpoint and iteration statistics, and
keep usage reports to NVIDIA off. The router reads it at
`/prometheus/metrics` — `trtllm_num_requests_running`,
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
GPU; one node per replica. Profiles may not give it an options file,
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
engine-check trtllm --bin <python>` checks an installed one. Written
against TensorRT-LLM's documented command line, server and metrics and
tested with stand-ins for them; not yet run on a datacenter GPU.
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
model's projector beside it, each checked against the sha256 Ollama named
its file by. With no name it lists Ollama's models; it finds Ollama's
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
service) or takes `--ollama-dir`. The version is the tag and the start of
the digest, since a tag moves in Ollama. It prints a model definition to
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
view does the same with **I**, for the mirror beside the control plane.
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
mirror machine into it the way a download is kept: whole or not at all, the
checksum beside it, a version never replaced, a `.gguf` checked to be one,
and `--sha256` checked when given.
### The portal in Turkish
- **The web portal speaks English and Turkish** — the landing page, sign-up
and sign-in, the account page and its editor setup, the claim page, chat,
the about page, and every administrators' page (cluster, nodes, NVLink and
fabrics, people, limits, usage, organizations, mirror, guardrails), and
the docs portal around its documents, which stay English. Flash
messages and form errors built by the server are translated too.
- **A switcher at the foot of every page** (English · Türkçe) remembers the
choice in a cookie; without one the browser's Accept-Language decides, and
without that the new `landing.language` (`en` or `tr`, empty is English).
An unknown code stops the server starting. `<html lang>` and
Content-Language follow the page.
- **Figures and dates in the page's language**: a Turkish page writes
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
Turkish. Form values, CSS and SVG coordinates are unchanged.
- **Ready for more languages**: the catalogue is JSON under
`internal/web/locales/<code>/`, and a language is a directory. Tests refuse
a key a page uses that some language lacks, a translation with different
arguments from English, and any template text left in English — so a new
page cannot quietly ship English-only.
- Messages that come from the control plane and other packages (node
warnings, engine errors, store refusals) are still shown as written, in
English; the API, CLI and console are English.
### Documentation
- **A docs portal at /docs.** The manual the binary carries — the pages
`llmgw docs` prints and the console's Manual shows, the configuration
reference among them — rendered on the client port in the portal's own
look: a sidebar of the documents in reading order, the open one's sections
under it, a link on every heading, and links between documents kept inside
the portal (links to files that are not in the binary, such as `deploy/`,
go to the repository). No script on the page, so the Content-Security-Policy
has nothing to allow. Public by default, like the landing page; set
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
the page carries the usual menu, which now has a **Docs** link.
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
needed — in Python, TypeScript, Go and curl: the base URL and key,
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
leads with it, and the landing page and the account page link to it.
- The console's Manual and the portal read one list of documents in one
order (`llmgw.Manual`), so the two cannot drift.
### Administration
- **An audit log.** Every change made through the admin API (`llmgw ctl`,
the console) or the pages under /admin is recorded: who, when, through
what, the fields it was given with passwords, keys and tokens taken out,
and how it ended — refusals included, in words. So are a person's own new
keys, revoked keys and password changes. Read it with `llmgw ctl audit`
(`--since`, `--actor`, `--contains`, `--detail`), `GET /api/v1/audit`, or
the console's **Audit** view. It moves with the store.
- **Staff roles** besides administrator: `viewer` looks at every
administration page and changes nothing, `operator` also runs the cluster
from them, `billing` sees usage and quotas. Set with `llmgw users role` or
the console's **Staff role…**.
- **Single sign-on through OpenID Connect** (`auth.oidc`). A "Sign in
with …" button sends people to the identity provider and back, with PKCE,
state and nonce. The ID token is verified, and the account is found by
the verified email address, or made when `create_accounts` is on. Groups
give staff roles (`roles`, set at every sign-in) and places in the tree
(`memberships`). `no_passwords` leaves the provider as the only way in,
except for administrators.
- **Provisioning through SCIM 2.0** (`auth.scim.token_file`, `/scim/v2`).
- **Users:** the identity provider makes, suspends and reinstates
accounts — the PATCH forms Entra ID and Okta send are both read.
Deleting a user suspends the account.
- **Groups:** joining and leaving the groups `auth.oidc` maps gives and
takes away staff roles and places in the tree.
- **Key lifecycle.**
- **Lifetime limits:** `auth.max_key_ttl` limits how long any key may
live, and `llmgw keys ttl team:Acme/ml 720h` sets shorter limits for an
organization, team or project. A key is made within the shortest that
applies, and one asked for with no expiry is given that longest — from
the CLI, the profile page and the editor connect page alike.
- **Rotation:** `llmgw keys rotate <id> --overlap 48h` issues a
replacement where the old key was, and the old one stops after the
overlap.
- **Listing:** `keys list --all` shows each key's kind — service (a
project's), member, personal — with service keys first, and when it
expires.
- **Usage export.** `llmgw ctl usage export --from 2026-09-01 --to
2026-10-01 --by project,model` writes CSV by hour, day or month and any of
organization, team, project, user, key and model.
- **Data:** read from the hourly and daily sums, so it reaches back past
the request records' retention.
- **Format:** cost is given in micro-units and units, and names that
look like a spreadsheet formula are quoted.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### Forgetting a person, and requests that stay in the cluster
- **`llmgw users forget`** erases a person for a request to be forgotten:
their usage kept under a random pseudonym so totals stay true, their kept
requests, personal files, batches and limits deleted, their name taken out
of the audit log, the account removed.
- **`llmgw keys residency --project P --in-cluster`**: no model served by an
outside provider for the project's keys, asked for, as a fallback or as a
canary.
- Both are on the console's menus.
### Secrets in Vault
- **Secrets from Vault or the environment.** Every setting naming a secret's
file also takes `env:VARIABLE` or `vault:mount/path#field` (HashiCorp
Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN).
- **The store's own secrets sealed** under `secrets.master_key`: Vault's
transit engine (`vault-transit:KEY`), or a 32-byte key read from a
reference. A store, or a backup of it, then does not hold what engine keys
are derived from. Values from before are sealed as they are next read.
### Scaling by the clock
- **`scaling.schedule`**: windows of the week with their own `min` and
`max` — more replicas through working hours, an on-demand model kept
loaded while people are at work — in `control.timezone`.
- **`scaling.prewarm`** opens a window early, so its replicas have loaded
when it starts.
### Token-level timing, and admission by SLO
- **Per-token metrics:** `llmgw_tpot_seconds`, `llmgw_inter_token_seconds`
and `llmgw_output_tokens_per_second` by model and pool, and TPOT and
tokens a second by tenant.
- **A model's `slo` counts in admission:** past its `ttft_p99` or
`tpot_p99` (the recent p99), it is treated as saturated, so the lower bands
wait. `llmgw_model_slo_seconds` and the `llmgw-slo` alerts make the
promise visible in Prometheus.
### Restoring a backup, and copies off the machine
- **`llmgw server restore ARCHIVE`** puts a backup back: checks it first
(whole, opens with `--identity`, not newer than this llmgw, where the store
goes), says so with `--dry-run`, then moves `/etc/llmgw` aside, restores it
with modes and owners, restores the store (SQLite, or PostgreSQL with
`pg_restore --clean`), and starts the server. A drill is in
docs/install.md.
- **`backup_copy_to`** copies each archive to a directory or over ssh as it
is written, keeping as many there as here; refused without
`backup_recipients`.
- Archives now record each file's owner by name, so a restore on another
machine gives the llmgw user its files back.
### Anthropic's Messages API
- **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat
completions and back, streamed or not: text, images, tools and their
results, the system prompt, stop sequences. Claude Code and Anthropic's
SDKs reach the models here; the key may come in `x-api-key`. Errors come
back in Anthropic's shape.
### More of OpenAI's API
- **Responses:** `text.format` asks for structured output, `input_image`
sends images, and `previous_response_id` continues a conversation the
gateway keeps as a thread (for a project that keeps conversations).
- **Passed through:** `/v1/rerank` and `/v1/score` for vLLM's rerankers, and
`/v1/audio/transcriptions` and `/translations` as the multipart form came,
for an engine serving Whisper.
- **`/v1/moderations`** is answered by the guardrails' judge model.
### Requests kept, and replayed
- **A project can keep its requests and answers** (`llmgw keys requests
--project P --retain-hours N [--sample 0.05]`, or **Keep requests…** in the
console): what the model was sent, after redaction, and what the client
got. Off until asked; never under `no_store`; pruned hourly.
- **`llmgw ctl request ID --body`** shows one; **`--replay MODEL`** sends it
to another model and shows both answers.
### Batches
- **OpenAI's Batch and Files APIs.** `POST /v1/files` (purpose batch),
`POST /v1/batches`, and reading, listing, cancelling and deleting them, so
OpenAI's clients send batches here unchanged.
- **Serving:** each line goes as a request of the key that made the batch,
in the lowest admission band, a few at a time; rate limits are waited
out. Answers and failures are written to files kept 30 days.
- **Accounting:** each line is an ordinary usage record tagged
`batch=<id>`.
- **Restarts:** answered lines are kept as they are answered, and any
gateway on the store carries a batch on under a lease.
- **Limits:** `router.batch` bounds the file, the lines and the
concurrency. Refused to a project under a `no_store` policy.
### New weights tried on a share of the traffic first
- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer
weights of a model's repository and runs them beside it as `MODEL-canary`,
given that share of its requests.
- **Routing:** a session stays on one side; `x-llmgw-variant` pins a
request to `canary` or `stable` and says which served it. Clients cannot
name the canary.
- **Rolled back by itself** when, over 50 of its requests or more, its
error rate is five points above the model's or its time to the first
token half again as long; `canary_rolled_back` is sent to the
notification webhooks.
- **Then:** `--percent` changes its share, `llmgw ctl model promote`
moves the model to it, `--abort` takes it out, and `llmgw ctl model
canaries` compares the two sides. All of it is also on the console's
model menu.
### Requests can be tagged
- **Tags and end users on every usage record.** OpenAI's `metadata` and the
`x-llmgw-tags` header (`feature=search, env=prod`) tag a request; OpenAI's
`user` field names the client's own end user. Both are kept with the
request's usage and shown by `llmgw ctl request`. Tags are not sent to the
engine. A malformed one is refused with 400.
- **Counted by tag, for the tags named.** `router.tag_labels` lists the tags
counted in `llmgw_tagged_requests_total` and `llmgw_tagged_tokens_total`,
the first 100 values of each apart and the rest as `other`.
### A hardened install by default
- **`server init` and the setup screens write a hardened configuration.**
- **Guardrails:** a `baseline` policy on every request. It refuses
secrets in prompts and answers, masks card numbers and records injection
attempts.
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
(about half a second for a short text). It refuses violence, crime,
weapons, child abuse, privacy violations, hate, self-harm and sexual
content. The policy fails closed: a request it cannot read within 5
seconds is refused (503), and `judge_unavailable` goes to the
notification webhooks.
- **Keys:** they live at most 90 days.
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
leaves the judge out and keeps the pattern checks. A `server.yaml`
written earlier is never changed.
- **What is left to you:** the closing text now names the two things
init cannot do — a certificate for the client port, and the keys
backups are encrypted to.
### Notifications
- **Webhooks for what needs a person** (`notify.webhooks`): a model parked,
a node lost, a budget spent, new weights in a model's repository — sent by
the gateway itself, with no Prometheus in between.
- **Formats:** JSON signed with HMAC-SHA256 (`X-Llmgw-Signature`) when
`secret_file` is set, or a Slack-compatible `{"text": …}`.
- **Delivery:** queued, never holding up what raised it; tried three
times; the same event about the same thing at most once an hour, and new
weights once per revision.
- **Checking:** `llmgw ctl notify test`.
- **A judge not answering** (`judge_unavailable`) is one of the events.
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
chosen.
### Guardrails
- **Hooks for guardrails on the request path** (nothing judges yet; B2 adds
the policies). A guard, when one is set, judges a request once it is
authorised, within its limits and has its stored conversation spliced
in, and before it is dispatched; and filters the answer, buffered or
streamed, before the client sees it. A blocked request is refused with 400
and `code: "content_policy_violation"` — the first error with a code — and
costs nothing; a blocked answer ends with `finish_reason: "content_filter"`,
and a stopped stream with that chunk and `[DONE]`, the engine cancelled.
The Responses API reports either as `incomplete` with
`incomplete_details.reason: "content_filter"`. `x-llmgw-guard` and
`x-llmgw-guard-policy` say what was done; the usage row records the
action, policy, category and time taken — never the text.
- **Guardrail policies** (`guard:` in server.yaml; see docs/guardrails.md).
- **What is found:** personal data, each kind validated before it counts
(email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits,
IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt
injection seen through disguises (look-alike letters, invisible
characters, base64/hex/URL encoding); deny patterns.
- **What is done:** flag, redact as `[EMAIL_1]`, or block. A request can
be shaped too: a pinned system prompt, the caller's own system messages
dropped, an output limit, tools stripped or refused.
- **Answers:** judged buffered, or a sentence at a time when streamed,
never releasing half a value; tool calls are checked against allow
lists, a count and the request's own schemas.
- **Modes:** `shadow` records what it would do; `no_store` refuses
conversations, notes and models outside the cluster.
- **Where they apply:** `guard.default` everywhere; others attached with
`llmgw ctl guard attach` to an organization, team, project, user or
key, and narrowed like limits.
- **A judge model on the cluster** (`role: guard` in the catalog, `judge:`
in a guard policy). It reads what patterns cannot — Llama Guard 3,
ShieldGemma or a JSON-answering model — and its verdict flags or blocks
input, output or both.
- **Calls:** past admission and accounting, bounded by a latency budget
(250ms by default), after which the policy's `fail` decides.
- **Caching and streams:** answers are cached for five minutes, and a
streamed answer is read every 600 characters.
- **Isolation:** a judge model is hidden from clients and refused to them.
It is critical by default, and cannot be loaded on demand.
- **Seeing what the guardrails do.**
- **Metrics:** `llmgw_guard_checks_total` (stage, policy, action,
category, mode), latency for checks and for the judge, judge errors,
fail-open, redactions by kind, stopped streams, and blocks per tenant.
- **Alerts:** judged traffic served unjudged, a slow or unreachable
judge, a tenant's blocks spiking, a secret in an answer.
- **Pages and commands:** `/admin/guard` shows counts by place, policy
and category and the recent requests by id; `ctl request` shows the
verdict; the profile page lists the policies in force where the person
works; the console's organization rows have **Guardrails…** and a
column.
- **The chat page** says when a request was refused or an answer stopped
by policy, instead of showing an error.
### Security
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
tests and vet.
- **Releases are signed, and only signed releases are installed.** The build
now carries the release signing key (`internal/release/trusted.pub`); the
control plane upgrading itself, an agent told to upgrade and a node being
added all refuse a release whose checksums are not signed with it. The CI
release job stops rather than publish an unsigned release. Adding a node
no longer has the node fetch the release and compare it with a checksum
from the same server: the control plane fetches it for the node's
architecture, checks the signature and the checksum, and hands it over.
Releases are fetched over https only (loopback excepted), and
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
- **The agent port is not opened without TLS by accident.** A control plane
with no `agent_tls` refused nothing: its agent port was plaintext, on every
interface, and any machine reaching it could register as any node — be
given replicas and so be sent prompts. It now refuses to start unless
`agent_insecure: true` is set, and then listens on loopback unless
`agent_listen` names an address. `server init` writes `agent_tls`, as
before.
- **A model definition cannot run code on its nodes.** A profile's `env` and
`args` reach a process the agent runs as root. The environment was checked
against a short denylist that missed NCCL's and CUDA's plugin and
injection libraries; arguments were not checked at all, so
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
name for a path, plugin, library, address or credential, and llama.cpp's
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
writes or loads files, loads code, or moves what the agent sets (host,
port, key, model). Both are refused where a model is defined and again on
the node before the engine starts.
- **Engines no longer run as root.** The agent starts them as the system
user `llmgw-engine` (in the video and render groups for the GPUs), with a
home and caches of its own under the agent's work directory. `llmgw node
add` makes the user; a node without it keeps running engines as root and
the agent says so in its log. `supervisor.run_as` in agent.yaml names
another user, or `root`.
- **The admin API wants a token.** It listened on loopback with no
authentication, so any process or user on the control plane could add a
model, upgrade the fleet, read the engines' keys or dump the heap through
pprof. Its `/api` and `/debug` paths now demand the token the server
writes, 0600, to its state directory (`/var/lib/llmgw/admin-token`) on
first start; `llmgw ctl`, the console and `node add` read it, so they run
with sudo, and `LLMGW_ADMIN_TOKEN` or `ctl --token-file` give it from
elsewhere. pprof is off unless `debug_pprof: true`. `admin_listen` off
loopback is refused unless `admin_remote: true`. The read-only dashboard
pages, `/metrics` and `/health` are unchanged.
- **The replicated store asks for a password and speaks TLS.** Patroni's
REST API took changes from anyone who reached port 8008 — rewriting the
access rules to `trust`, or moving the leader — and PostgreSQL traffic
between the machines was plaintext (`sslmode=disable`). Patroni now asks
for a password before a change (kept with the cluster's others in
`/root/llmgw-pg`, and on the control plane in
`/etc/llmgw/secrets/patroni-rest` for `server add`); each member has a
certificate from the cluster's authority, PostgreSQL has `ssl` on, every
remote access rule is `hostssl`, and the control planes connect with
`sslmode=verify-full` against the authority's certificate. A new cluster is
set up this way; an existing one is changed in place, by reload, the
cluster serving throughout, by running **Replicate the store** in the
console again — or `sudo --preserve-env=SSH_AUTH_SOCK llmgw server init
--patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node
--no-model` — on the control plane that holds the authority, then on each
other control plane so it reconnects by TLS. etcd is not yet given TLS:
keep ports 2379 and 2380 to the cluster's machines.
- **A capacity fallback respects the key's allowed models, and is charged
at its own price.** When the model asked for had no capacity, the next in
its `fallbacks` served the request even if the key was not allowed it, and
the request was priced as the model asked for. Fallbacks a key may not use
are now skipped, and a request is charged what the model that answered
costs.
- **Suspending an account stops its keys.** Suspension closed the person's
sessions and left every API key they had made working. A key whose owner's
account is not active — suspended, pending, or gone — is now refused as a
revoked key is, within `auth.cache_ttl`, and works again when the account
is reinstated.
- **A join code is a credential only for its hour, and only on its
address.** The setup checked a code's expiry on the joining side alone, so
a code stayed good for as long as the setup ran; it now refuses an expired
one itself. Its listener binds the address the codes name rather than
every interface, and a joined machine is known by the address it connected
from, not the one it claimed.
- **The setup's sudo grant cannot outlive it.** Given a sudo password, the
setup writes a temporary `NOPASSWD` grant and removes it when it
finishes; a setup that crashed or was interrupted left it for good. The
grant now schedules its own removal within the hour as it is written, and
the password is dropped once the grant stands in for it. An address given
to the system ssh can no longer be read as one of its options.
- **Made-up keys cannot fill memory or the database.** Every well-formed key
presented, real or not, was cached for good and cost a database read the
first time. The cache is now bounded, and keys not in it are looked up at
most 20 a second per client address (50 at once); beyond that the request
is answered 429 without a read.
- **Slow or idle connections cannot pile up.** The servers bounded only how
long headers took. Every listener now also bounds how long a connection
may sit idle and how large its headers may be, and reading a request
(5 minutes on the client port, a minute or less elsewhere — never on the
agents' long-lived streams); the client port holds at most
`max_connections` (10,000) at once, and a message from an agent at most
16 MB.
- **The sign-in and signup forms resist scripts.** Ten wrong passwords from
anywhere locked an address's owner out for the window; attempts are now
counted per address and client, so somebody guessing elsewhere cannot lock
the owner out, with a looser count per address alone so guesses spread
over many clients are still bounded. The sign-in form carries a token its
cookie matches, so another site cannot sign a visitor into an account of
its choosing. Posting either form is rationed per client address (sign-in
30 a minute, signup 5), and a password hash that cannot get a turn within
ten seconds answers "busy" instead of holding the request. `server init`
now writes `signup_open: false`.
- **Forwarding headers are believed only from the proxies named.**
`X-Forwarded-Host` and `-Proto` were taken from anyone, and decide the
gateway address written into the `/setup` scripts clients pipe to a
shell; behind a caching proxy one request could poison them for others.
They — and `X-Forwarded-For`, now used for the per-address limits — are
believed only from `router.trusted_proxies` (addresses or CIDR prefixes).
**Behind a reverse proxy, set it**, or the pages and scripts name the
address the proxy reached and every client counts as the proxy. `/setup`
scripts are served `no-store`.
- **Removing a node revokes its certificate.** `ctl node remove` only
dropped the node from the inventory; its certificate, valid for a year,
still connected and was given work. Now the removal is recorded in the
store and the agent port refuses any certificate issued for that node
before it — onboarding the node again issues one that works. A mirror
serving its management API with `--tls-ca` now requires `--api-client`
(the control plane's name), since the same authority signs every node.
- **Engines listen on the management address only**, the one the control
plane reaches them on, instead of every address the node has — a public
one included. (When that address is a name, or not one of the node's,
they still listen on all.) **SGLang's key is off its command line**,
where every user on the node could read it: it reaches the engine in its
environment, and a short launcher hands it to SGLang inside the process.
Both take effect as engines restart under an upgraded agent.
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
recipients or SSH public keys) has `llmgw server backup` encrypt each
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
and every secret in the store, and is meant to be copied off the machine.
Open one with `age -d -i <key>`.
- **PyTorch's index gives torch and nothing else.** vLLM's CUDA 12.9 build
was resolved with PyTorch's index beside PyPI and `unsafe-best-match`, so
either index could supply any package — how torchtriton was hijacked in
2022. Now that index is explicit, named only for torch, torchvision and
torchaudio; every other package comes from PyPI alone, and the lock is
PEP 751's (`pylock.llmgw.toml`), pinning each file's address as well as
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
- **The control plane's own fetches stay off link-local addresses.**
Checking a model's source and reading a GGUF's header went wherever the
URL said, cloud instance metadata (169.254.169.254) included. They now
refuse link-local, multicast and unspecified addresses, while still
reaching the cluster's network, where the mirror is. The mirror's fetches,
which refuse private addresses too, now also see through IPv6 addresses
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
the target's addresses are checked before the request is handed to it.
- **A CPU-only node reporting a problem no longer panics the agent
handler**, and **thread ids are bounded** to 128 letters, digits and
`- _ . :`. A thread is readable by every key of its project that knows
the id; the docs now say to use unguessable ids.
- **`/v1/models` lists only what the key may use**, rather than every
model the gateway serves to anyone.
- **Portal sessions end after a day unused** (still at most 14 days in
all), and expired or idle sessions and used or expired key claims are
deleted hourly — they were refused, but never removed.
- **Every page carries a Content-Security-Policy**: scripts only from the
gateway itself and the page's own inline ones, each named by its hash —
no inline handlers, no eval, nothing a model's reply or a model card
could slip into a page. The few `onchange`/`onsubmit` attributes moved
to `/_static/pages.js`, htmx runs with `allowEval: false`, and the live
pages' "only while visible" refresh no longer needs an evaluated filter.
Pages now carry their headers even when answered with an error status.
- **KaTeX 0.16.47** (from 0.16.11), past the advisories fixed in 0.16.21.
- **Smaller hardening.** The mirror no longer serves its bookkeeping files
(a file's `.source` URL and `.sha256`) on `:8081`. A GGUF header nesting
arrays more than 4 deep is refused rather than recursed into. Serve and
update jobs keep the last 100 finished, and the mirror forgets finished
downloads after a week, instead of remembering every one until restart.
- **A firewall for llmgw's internal ports.** `server init` (and the store
members it sets up with `--patroni`), `node add`, `server add` and
`mirror init` now keep the machine's internal ports — the agents' stream,
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
cluster's own networks, in an nftables table of llmgw's own loaded by
`llmgw-firewall.service`. Only those ports are touched: the table accepts
everything else, so other services on the machine and any firewall it
already has are left as they were. Extra networks go in
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
node lacks it.
- **etcd and Patroni's API speak TLS.** etcd's peers and clients use
certificates from the cluster's authority, and a client must present one;
Patroni reaches it with its member's certificate and serves its API over
TLS. `server init --patroni`, run again, moves an existing cluster: with
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
control plane asks the API over HTTPS, verified, and over HTTP only where a
member has not moved yet.
- **A cluster keeping its access rules per member got no `hostssl`.** The
TLS step looked only at the shared configuration, and on a cluster set up
before the rules moved there said the rules were `hostssl` while
plaintext was still accepted. Each member's own rules are now made
`hostssl` too.
### Fixed
- **An engine that died after an agent restart went unnoticed.** The watcher
that reports a ready engine dying was tied to the stream it was started
on, so after a reconnect nothing reported it, and an engine adopted from
the previous agent had no watcher at all: its replica stayed "ready" and
was routed to. Every connection now watches every ready shard, started or
adopted. Found on the PoC: a killed engine on node06 stayed ready for
minutes.
- `server init` rewriting the store's address no longer leaves
`/etc/llmgw/secrets` unreadable to the service when it was made 0700 for
the service user, which kept the gateway from starting.
### Everything that can be changed on a running cluster, from the console
`sudo llmgw` only watched the cluster and edited the configuration files;
changing a model's replicas still meant `llmgw ctl`. Now Enter on a row opens
what can be done to it, as forms and questions:
- **Models**: replicas (min, max, idle timeout), class, owner, resources, LoRA
adapters, restart, retry when parked, why it is placed where it is, remove;
A adds one from a definition file.
- **Nodes**: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its
agent, delete model files nothing uses, remove.
- **Replicas**: restart, replaced or in place.
- **Quotas** (new): capacity quotas, set and removed.
- **Mirror** (new): what it holds and which models use it; download into it,
remove from it.
- **People** (new): accounts — approve, suspend, reinstate, the cluster
dashboard, memberships, limits, a new password, delete, create; API keys —
issue (shown once), limits, revoke; organizations, teams and projects — add,
rename, move, remove, limits for the level and for each member, capacity
quotas, how long conversations and agent notes are kept.
- **Serve a model** (new view): GGUF models on Hugging Face — trending, most
downloaded, most liked or newest, or searched with / — then one of the
chosen repository's files, a name and a replica count, and it is fetched,
checked, added and started. The registry is mirror.registry's when set.
- **Do**: upgrade the cluster, restart every replica, back up now.
The people side runs the same commands as `llmgw users`, `keys` and `org`,
against the same store, so their checks and messages are the ones shown. A
sidebar taller than the terminal now scrolls with the current entry.
### New weights in a model's repository, found and moved to
A repository on Hugging Face can publish new weights under the same file
names — a fixed chat template, a better quantisation — and nothing said so. A
model served from the registry now records its origin: the repository, the
commit, and each file's checksum there; a model served before is worked out
from the mirror's record of where it fetched each file. Every
`mirror.update_check` (six hours by default; 0 turns it off) the control
plane asks whether those files changed. A new commit that leaves them alone
is not an update.
- `llmgw ctl model updates [--check]` lists what each repository has.
- `llmgw ctl model update MODEL` fetches the new files into the mirror beside
the old ones, checks them against the registry's checksums and the nodes'
llama.cpp, points the catalog at them, replaces the replicas one at a
time and asks one to say hello.
- `llmgw ctl model rollback MODEL` goes back to the files the update
replaced, which stay on the mirror.
- The console shows UPDATE on Models and offers all three on a model's row.
A model from `server.yaml` is reported but not changed: its source is edited
there.
### The monitor: the cluster drawn like btop
The console has a **Monitor** under Watch: CPU, memory, GPU busy and memory,
requests, tokens, time to first token, in flight and queued, network and
disk, as Braille charts filled and coloured by height, over 5 minutes, an
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
each card's busy, memory, heat and power, and each model's requests, as
sparklines. A node's and a model's own pages carry their charts too.
The history is read from the Prometheus in `monitoring.prometheus`, through
the admin API (`POST /api/v1/metrics/range`); without one, the charts draw
what the console has seen since it opened.
The control plane now lists the cluster's machines for Prometheus' service
discovery, at `/prometheus/targets` on its metrics port — nodes, itself and
the store's members, each labelled with its llmgw id — and `llmgw node add`
installs `prometheus-node-exporter` on the machine (`--no-node-exporter`
leaves it off). `deploy/monitoring/prometheus.yml` scrapes them as the
`nodes` job.
### Serving and updating run in the background
Serving a model or moving one to new weights from the console starts a job
on the control plane and leaves the console free at once, instead of
closing it to follow a download that can take an hour. A new **Jobs** view
shows each job's progress and steps, and the sidebar how many are running.
`llmgw ctl model serve --detach` and `llmgw ctl model update --detach` do the
same; `llmgw ctl jobs` lists the jobs and `llmgw ctl jobs ID` follows one.
### The file that suits the hardware, and how to spread it
`llmgw ctl model plan REPO` now plans the files the repository actually has
against the fleet as it is, before anything is downloaded: the model's shape
comes from one GGUF file's header, read from the registry, or from its
config.json. Every file is listed with the best way it runs here — one card;
several of a node's cards (llama.cpp splits by layer, vLLM by tensor
parallelism, dividing the KV heads); several nodes' cards over one fabric,
pipeline-parallel between them (vLLM); or a CPU node's memory — what each
card or node would hold and whether that hardware is free now. Best first:
on a GPU, the fewest cards, then the better quantisation up to eight bits;
whether the cards are free now only breaks a tie. Busy is not unfit: a model
for cards that are in use is added and starts when there is room, the job
saying so rather than failing. `--precision` keeps the old plan in the abstract.
`llmgw ctl model serve` without `--quant` takes the plan's file, and without
`--cpu` or `--gpu-class`/`--gpus`/`--nodes` the plan's way of running it —
a GGUF model on GPUs is now given a GPU profile rather than always the CPU.
`--fallbacks` writes the next best ways too, lower in priority, so placement
uses one of them when the first is busy. The console's Serve a model shows
the plan for each file, marks the best, and asks for the way and the
fallback.
### Models of several files: vision projectors, original weights
- A vision model's projector (`mmproj`) is fetched beside its GGUF weights
and loaded with them: `llmgw ctl model serve REPO --projector FILE`, or the
choice offered by the console. The model is marked as taking images. Nodes
load it once their agent is this version or later.
- `--format safetensors` serves a repository's original weights, with their
config.json, tokenizer and shard index, on vLLM across `--tp` GPUs of
`--gpu-class` (the fleet's one class when there is one). In the console,
**F** on Serve a model switches to them.
- The mirror keeps where it fetched each file beside it, and reports that
and the checksum after it restarts. The console's Mirror view shows one
row per version of a model, with its files on Enter.
- An update watches every file of a model, the projector and the tokenizer
included; one whose files the mirror cannot trace is found on the
registry by name, commit and size, or given with `llmgw ctl model origin`.
- `llmgw_model_update_available` is published, and `ModelUpdateAvailable`
alerts on it.
### The manual, in the binary
`llmgw docs` reads every document — installation, operation, the
configuration reference, the README, the changes — and a page for every
command, built from the commands' own help, like `info`: topics on the left,
T for a page's contents, / to find, n for the next match. Piped, it prints
the page. In the console it is **Help › Manual**, and **?** opens the new
page about the console itself from any screen.
## v0.11.1 — 2026-10-03
### The console's overview no longer spills over the screen
On a cluster with a GPU, `sudo llmgw` printed the overview's GPU table onto the
terminal under the console every three seconds, scrolling across the full
screen. The summary's tables were written to standard output whatever they
were given; they are now written where the summary is, which for the console
is its panel. The test cluster the console was tried on had no GPU, which is
why it was not seen there.
## v0.11.0 — 2026-10-03
A minor version: the store can be replicated and the control plane doubled,
both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
usage is read from sums kept as it is written. Released without candidates.
**Upgrading.** Migrations 0015 (usage sums, filled from the request table
already recorded — once, at the first start) and 0016 (control planes) run by
themselves. An agent older than this ignores the list of control planes and
keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
The GPU and fabric work below began from the field report in issue #2, after
a deployment on GPU nodes.
### GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.
- **New series**: `llmgw_gpu_memory_utilization_percent`,
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
sample.
- **The node page** shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
- **The cluster page** charts the fleet's GPUs by class: average utilisation
and memory in use.
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
- **The cluster page's GPU map**: every card a square, a row per node,
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
with what holds it and its figures under the pointer; free cards dashed.
- **Cards held and doing nothing**: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
- **What each model gets from its cards**, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
- **What wears a card out**, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
`GPUNVLinkErrorsClimbing`.
- **GPU time by owner** on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### Usage is read from sums kept as it is written
Every chart and total — the usage tab, quotas, the dashboard, a person's own
page, and budgets restored at start — summed the request table again on each
view: a row per request, scanned whole. Usage is now also summed as it is
written, in the same transaction, into five-minute slices, hours and days per
key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
0015 fills them from what is already recorded).
- **Charts** read the coarsest of the three their slices are made of: a day in
20-minute slices from the five-minute table, a week or a month from hours.
- **Totals and budgets** read the rows up to the next five-minute slice, then
slices, hours and days — exact, not rounded to a slice. On three million
requests over thirty days, a month's total per person went from 705 ms to
18 ms on PostgreSQL.
- **Latency and time to first token** are kept as histograms in the sums, so a
percentile per team or model can be read after the rows are gone.
- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
that, hourly. Totals, charts and budgets are unchanged by it — they read the
sums — but `ctl request` finds nothing older. Unset keeps every record, as
before.
### Alerts per tenant and per budget
- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
tokens and time to first token per tenant. The tenant is the level
`router.tenant_label` names — `team` by default, or `org`, `project`, or
`none` — named by its path, "acme/platform". One level rather than every
project, so a cluster of thousands of projects does not become a Prometheus
of millions of series.
- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
over the budget for every organization, team, project and person given one,
refreshed every minute.
- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
`BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
`TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
`TenantSlowFirstToken`.
- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
(uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
refused by a limit, first-token p95, every budget's share spent, and who used
most in the last day — with a tenant chooser at the top.
### How each team's requests fared, not only what they cost
- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
per organization, team, project, person or model, read from the histograms
kept with the usage sums: on the usage tab's table, and in a branch's or
person's panel on the people page, beside the error rate and requests
refused by a limit.
- **Refused requests are counted against their owner.** A request a rate,
token or budget limit refused was counted only in Prometheus, by model; a
team running into its limit looked like a team that had gone quiet. It is
now recorded as a 429 against its key, person and project — no tokens, no
cost — and shown as "Limited".
- **Usage by model** is a grouping of the usage tab, by the model that
answered.
### The console watches the cluster and holds every setting
`sudo llmgw` on a control plane is a console in three groups down the left:
- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
memory, GPUs free, disk and agent version, and a node's detail — load,
engines, each card's busy, memory, temperature and power; every GPU in the
fleet and which model holds it; models, with a model's profiles, warnings
and replicas; replicas and where they run; the last day's use by team,
project or organization; and a node's agent log.
- **Settings**: every key of `server.yaml`, block by block, with what applies
when the file sets nothing and what each key is for, from its comment in
the code. Enter changes a key or opens a block; lists and maps of blocks
(endpoints, models, prices) take entries added and removed. A save is read
by the same loader the service uses first — nothing is written that would
not load — keeps the file's comments, leaves the old one beside it, and
offers to restart the service.
- **Do**: add a node, replicate the store, leave.
On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
The keys come from the configuration structs themselves, through the
generator that writes `docs/reference.md`, so a setting added to the code is on
the screen without anyone listing it, and a test fails if the two drift.
That generator told blocks apart by name alone, so `tlsconf.Config`,
`engines.Config` and `supervisor.Config` were each taken for the agent's
`Config` and dropped: the reference had no TLS, engine or supervisor settings
at all. It now has 37 blocks, six more than before.
### A second control plane with `llmgw server add`, and the screens as windows
- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
control plane, makes another machine a standby: this llmgw copied over, a
certificate from the cluster's authority, the same configuration and store
address, the machine let into the store (for a store under Patroni, one
change through Patroni's API), and the service started. Every node is told
of it within half a minute and fails over to it untouched. `--vip` keeps a
floating address on whichever control plane leads, with keepalived on both.
Tried on new machines: switching the leader off, the standby led after 14
seconds and the floating address answered from it after 23.
- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
ones in their configuration.
- **A node notices a dead control plane within a quarter of a minute.** A
leader whose machine stopped dead left its nodes' streams open until the
kernel gave up retransmitting, about fifteen minutes; the agent now pings
the stream after ten quiet seconds, and bounds dialling too.
- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
open like `/ready` — what keepalived follows.
- **The store's access rules live in Patroni's shared configuration** for a
cluster set up by `server init --patroni`, so another control plane is let
in without touching each member.
- **The screens are windows**: full screen, a title bar, the steps or actions
down the left, the content in a bordered panel, the keys along the bottom,
questions and errors as dialogs over it — the setup, the console, adding a
node and one-machine init alike. Keyboard only.
### Machines nothing may log into: `llmgw join`
For sites that allow no ssh into a machine, the setup screens offer a fourth
way to reach one: a join code. The setup listens on port 7444 while it waits
and shows a code for each such machine, with the commands that fetch llmgw
onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
checks by the certificate fingerprint in the code that it reached the setup
that made it, admits itself with the code's secret, and runs what the setup
sends, printing each step, until the setup says it is done. A code is good
for one machine and an hour, and is under 90 characters so it can be copied
whole from one screen into another.
To the setup a joined machine is one more way of running a script there:
the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
machine joining by code and another logged into with a password, in one
setup.
### Setting up from nothing in screens: `sudo llmgw`
Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
setup on a machine with nothing installed, and a console on a control plane.
In a pipe or a script it prints its help, as before.
- **Setup** asks one machine or three. For three — this one as the control
plane and etcd's third vote, two more for the store under Patroni — it asks
how to reach the other two: the ssh agent, a key file, or a password, and a
sudo password where sudo asks for one. Passwords stay in memory for the run;
a sudo password grants the user passwordless sudo until the install
finishes, and the grant is removed after, failed or not.
- **Every machine is checked before anything is installed**: login, system
(Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
role, clock against this one, NTP, the ports its role needs, internet, and
whether it reaches the other machines. Failures stop it, with what to do;
warnings let it go on.
- **Then the plan, and the command line that does the same** from a script,
then the install with its progress, and the administrator's password.
- **The console** shows `ctl status`, refreshed every five seconds, and offers
to add a node or, for a store on one machine, to replicate it.
- Machines are reached with Go's ssh client rather than the system's, so a
password can be used; a host key is recorded on first sight and refused if it
changes.
Tried end to end on three new machines whose user logs in and sudoes only with
a password.
### A store that survives losing its machine
`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
under Patroni on two machines, its leader lock in a three-member etcd, and
llmgw given every member with `target_session_attrs=read-write`. Measured: a
planned switchover cost one failed lease renewal; switching off the primary's
machine promoted the standby in 24 seconds with the gateway serving
throughout. Example configurations are beside it.
**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
third vote — the role and database, and the store already in use moved in.
Every step looks first; run against a cluster that is up, it changes nothing.
Tried from scratch on three new machines (two minutes, then a failover by
switching the primary off) and against the reference cluster (nothing
touched).
**Watching it**: for a store of several hosts the control plane publishes
`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
failed: the flag was sent as a bool to an integer column, which SQLite takes
and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
by being copied from SQLite, which is why nobody had met it.
`llmgw server backup` reads a store address of several hosts. It took the
list for one host name and reached the database only by falling through to
the second; it now hands libpq the list, and dumps from a standby when there
is one.
### People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.
Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.
The administrator's menu no longer has an Organizations entry beside it: `/org`
and `/org/<id>` send an administrator to that organization's panel here. The
page stays as it was for an organization's managers, who cannot see this one.
### The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs — `GPU`, `CPU` or
`GPU + CPU` — and the line under each reply names what answered it: `GPU
6000-24g` or `CPU`, beside its tokens a second.
Every answer through the gateway carries the same as `x-llmgw-hardware`:
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
none.
### A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.
The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
agent too old to send it keeps the reading from registration.
### A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:
- **Installing** the CUDA build asks it which devices it sees
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
own `ldd` could not say it: the backend is opened at run time.
- **After the upgrade**, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
- **Starting** a shard on GPUs is ready only when its process holds memory on
each of them (`nvidia-smi --query-compute-apps`, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.
### A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.
- **The explorer fetches a safetensors set with what the engine reads beside
it**: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a `.py` would be the engine
running what the repository ships, which is to be chosen on purpose.
- **A version is offered to nodes only whole.** The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its `model.safetensors.index.json` names, or a part of a
split (`-00002-of-00004`), is absent. A node reports "not complete on the
mirror yet" and is tried again.
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
- **`llmgw mirror add` takes several URLs** into one version.
- **A Hugging Face repository page given as a source** is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
such a model, as before.
### Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.
An error the page does not explain says the account was not created and
gives a reference (`err-…`), logged beside the error, for an operator to find
it; the database's own words stay in the log.
### An adapter no fabric uses does not keep a node out
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.
- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
- Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; `node add` still warns
once that adapters are up with no fabric given.
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
written afterwards.
### A restart with no room beside it
`ctl restart` places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:
- **The replica is the only thing in the way**, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
- **The replacement fits nowhere even then**: the restart is given up and the
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
- **Room being made** — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
### Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.
| | llama.cpp | vLLM |
|---|---|---|
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
| No NVIDIA GPU | the CPU build | none |
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
- **The install computes on the GPU before it counts**: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
- **The build can still be chosen per node** on the cluster page (auto, CUDA
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
at once.
- **Whether an engine computes on its GPUs is read from every process
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
`CUDA_VISIBLE_DEVICES`.
### Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly: `node add` refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and `--ignore-checks` passed over every other problem with it.
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
node joins without it and serves what it does run. `node add` checks the
vLLM the control plane installs, not the binary's own default.
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
- **The requirements follow the version**, as the table above sets out; the
threshold had been CUDA 13's for every version.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
### Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by `gpu_class`, which the form never
set. Now:
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
- **GPUs per replica** sets how many cards a replica takes on one node
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
how many a llama.cpp replica serves at once.
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
typed.
- A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
anyway** is ticked.
### llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
placement reserves N cards of its `gpu_class` on one node, the engine sees
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
(`--split-mode layer`, in proportion to each card's free memory, which for
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
`gpus_per_node`, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
`nodes` above one, is refused.
## v0.10.6 — 2026-10-01
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
format check and has no binaries.
Tried on the PoC cluster before release: LoRA adapters asked for by name
through the gateway, two on one model, each answering as trained and the model
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
requests on the preferred replica, 4 on the other); and a bronze request
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
live: a model served from an external provider, which needs a provider key.
### Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks as `llmgw ctl quota`.
- **A model's card** on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said.
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
### LoRA adapters
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (`--lora-scaled FILE:0`); a request that names an
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in `/v1/models`, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.
### Finding and attaching adapters from the mirror page
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
held model's card, *find adapters on Hugging Face* — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original.
- **Opening a repository** reads the first bytes of each GGUF file and marks it
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says.
- **Held adapter files** are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
lists and changes them from the command line, through
`POST /api/v1/models/{name}/adapters`, with the same checks.
### Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
model, api_key_file}` and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model's `fallbacks`, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md). `ctl models` and the cluster page mark it
external.
### Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.
### Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it. `admission.queue_capacity_total` now bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted as `displaced` in the admission rejection metric.
### Releases can be signed
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
public key in `internal/release/trusted.pub`, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.
### Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
## v0.10.5 — 2026-09-30
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
format check and has no binaries.
After upgrading:
- **Declare what CPU models need.** A CPU model added from now on must say its
memory; one stored before this still loads and is flagged in `ctl models`
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
are then reserved per replica and become the engine's thread count, for
replicas started after the change: `llmgw ctl restart --model MODEL`.
- **Upgrade the mirror on its own host**, now with one command:
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
mirror older than this). `llmgw ctl upgrade` says so at the end.
### CPU cores are reserved, and memory has to be declared
The first step of resource allocation for larger clusters: what a replica
holds of a node is reserved, not assumed.
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
node holds as many CPU replicas as it has cores for, and two on one node no
longer spin against each other. Agents report their CPU count; a node that
does not, and a profile that declares no cores, constrain nothing.
- **Memory is required of new CPU models.** A model with no GPU is all host
memory, and one declaring none looked to placement as if it fitted
everywhere, any number of times. Adding one without `min_memory_gb` is now
refused. Models stored before this still load, and say what is missing.
- **Declared against measured.** Each model shows what one replica reserves
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
cluster page — with a warning when nothing is declared or when it uses more
than a tenth over what it reserves.
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
`ctl model serve` and the mirror's serve form fill in cores — half the
smallest node the model fits on — beside the memory they already estimate.
Running replicas keep their old figures until restarted.
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
rc.7 the first was briefly `llmgw ctl resources`.
### Capacity classes, and preemption
A model has a class — `critical`, `standard` or `opportunistic` — saying what
its capacity is worth against another's when there is not enough for both.
Unset, an on-demand model is opportunistic and every other model standard,
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
sets it.
When a model cannot be placed, replicas are stopped to make room for it, by
these rules:
- A lower class gives way to a higher one.
- Within a class, only replicas above a model's minimum — room it grew into
under load — and only for a model still below its own minimum.
- A critical model's minimum is never taken, nor a replica that has only just
loaded, nor one spanning several nodes.
- Idle on-demand replicas go first, then what lower classes grew into, then
the same within the class, then lower classes' minimums; the longest unused
first within each, and of equally idle ones the smallest.
- The set is tried on a copy of the inventory first, at most three replicas,
and nothing is stopped unless it makes the room. Of a set that does, each
replica whose stopping turns out not to be needed is left running, so an
idle replica tried first on the wrong node is not stopped with the one that
made the room.
A replica stopped this way is drained and given 30 seconds to finish what it
is serving. Its model is held back for up to five minutes, or until the model
it gave way to has its replicas, so it does not take the room straight back.
`ctl models` shows each model's class and "gave room to X" while held; the
cluster page shows the class and how many replicas were lost; the metric is
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
### Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
one with no owner is shared and counts against nobody.
- **A guarantee is kept.** A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one.
- **Beyond it is borrowing.** Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum.
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
by scope and resource; alert: CapacityAtLimit.
### Sized by what a replica was measured to serve
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
replica served while meeting its SLO, and its throughput there — now drives
decisions rather than only being printed:
- **Scaling up** when requests per replica pass what was measured, not only
when the engines' queues fill: an engine takes requests into its slots long
after latency has gone.
- **Scaling down** only when the replicas left would stay under 70% of what
they were measured to hold, so a replica taken back does not put the rest
straight over.
- **Choosing a profile.** Among profiles of equal priority, the one measured
to serve the most per GPU — or per core — is tried first. Priority set by an
operator still wins.
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
replicas that takes, how many run, and whether the rest fit, with the reason
when they do not; `--apply` sets it if they do. Headroom reports what one
more replica would serve.
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.
### Upgrading a machine the control plane does not reach
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
the mirror's host above all, whose service runs unprivileged and cannot replace
its own binary as an agent does. The release's checksums are checked, the old
binary kept, the unit restarted and watched until it stays up.
### Nodes report their own heat and power
The first step of facility-aware placement (docs/facility-aware-placement.md),
observe-only: nothing is decided on these yet. Every agent reads, at most every
ten seconds, and sends with its heartbeat:
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
it, power draw and the enforced limit, and what is holding the clocks down —
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
at registration; it is now current.
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
the package temperature, and the board's inlet air sensor where it has one.
- **CPU power** from RAPL's energy counters.
A sensor a machine does not have is not reported, never shown as zero. New
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
table of temperature, power against its limit, and what is throttling it.
### The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
explorer and the fetch form, one under another, with every visit waiting on
Hugging Face for a list it might not show. It is now three tabs — Held (with
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
URL — each with its own address. Only the tab shown does its work, and only
Held reloads while something downloads. A download started from the explorer
links to Held, where its progress is; a fetch the mirror refused comes back on
the form it was sent from.
### A held model opens to where it runs, and how many
Clicking a model under Held opens a card beneath it: each replica with its
node, state, age and the CPU and memory its engine uses, and how many more
replicas the cluster has room for now, on which nodes, and why not more. The
replicas it always runs and the most it grows to under load are changed there.
Room is found by trying the scheduler itself on a copy of the inventory — the
same decisions the next passes would make, not an estimate beside them — and a
number of always-running replicas it cannot place is refused with the reason,
rather than accepted and left short. A ceiling above the room is taken, and
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
API.
Each row also says how its model runs without being opened: replicas ready of
those it should have, how many are still starting, and the nodes they are on —
in amber when short, in red when parked or when it does not fit. It is one line
per model beside its mode, and nodes drop the part of the name every node
shares (node05 for llmgw-node05, the full name on hover).
### One line per record, in every table
The same rule across the admin pages. Headers do not wrap; a record is one
line; free text is cut to its column with the whole of it on hover rather than
given a row of its own; a row across the whole table is only a note or an open
form. On the cluster page that means a node is one line (its address and logs
beside its name, not under it), a model's description is a column rather than a
second row, node names drop the prefix they all share, and the GPUs, classes,
fabrics and fallbacks columns appear only when some row has something in them —
on a CPU-only cluster they were three columns of dashes pushing the table past
the page. On People, a person in many teams shows two and a count, the rest on
hover.
## v0.10.4 — 2026-09-30
Published first as release candidates v0.10.4-rc.1 to rc.9.
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
thread count below only when they start.
### CPU replicas no longer fight over the CPUs
llama.cpp computes on every CPU of the machine unless told otherwise, and the
agent did not tell it. Two busy CPU replicas on one node then spin against each
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
0.8 each with both busy, in two rounds of the same test. The agent now starts a
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
node — all of them suits a node that only ever runs one CPU replica — and a
profile's own `--threads` still wins. Measured on VMs whose host is
oversubscribed; the proportions, not the numbers, are the point.
### Tracing
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
request is traced over OTLP/HTTP: a span for the request, carrying its
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
it. A caller's own `traceparent` is continued, and the engine is sent one for
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
without a decision. Off, it costs nothing.
### Restarting replicas
`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
ones. Each replacement is started beside the replica it replaces, which keeps
serving until the new one is ready; replicas of one model go one at a time.
Where no second copy fits anywhere, the old replica is stopped after the drain
grace period and the new one starts in its place. This is how running engines
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
version.
### A model that keeps failing is parked, and everybody is told
A model whose replicas failed at every start was retried forever: kev-0.5b,
whose architecture the nodes' llama.cpp did not know, was started 155 times in
eight hours. After five failures in a row a model is now parked — not placed
again until its definition changes or `llmgw ctl retry MODEL` is run — and
`ctl status`, `ctl models` and the admin API say so with the last error.
A request for a model that will not come up — parked, or fitting on no node —
is answered at once with the reason, instead of "the model is loading; retry
shortly", which a client would retry forever. Each request still makes the
control plane try placing it afresh, so a model that fits once room is made is
not refused on an old answer. "no placement for model" is logged when its
reason changes or every ten minutes, not on every pass: it was 250 identical
lines in two days for one model.
### Metrics and alerts for what goes wrong
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
`llmgw_model_replicas_ready` is now published at 0 for every model that should
be running; it was published only while a replica was ready, so
ModelHasNothingServing could never fire. New alerts: ModelParked,
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
`ctl status` counts a model against the replicas it should have now rather
than its minimum, so an on-demand model that was asked for and fitted nowhere
no longer reads as satisfied.
### Backups
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
secrets, and the certificate authority's key — to one archive readable by root
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
nightly. The control plane publishes the newest archive's time as
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### Replicas follow load between min and max
A model's `max` was a ceiling nothing reached: every model ran exactly its
minimum however long its queue. Now, when requests are waiting at the engines —
every slot of every replica busy — at one or more per replica for a minute, one
replica is added, up to `max`; ten minutes with nothing waiting takes one of
those away, down to `min`. One step at a time, and not while the previous one
is still starting, so a spike that is over before a replica loads does not
leave several behind. Each decision is logged with the load that made it.
`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
### A model from the registry in one command
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
mirror fetch it and checks what arrived against the registry's published
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
header, adds it to the catalog, waits for a replica — stopping early with the
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
is shown as it happens. It runs in the control plane as a job, so a download of
hours does not depend on a terminal staying open. `--on-demand` adds it without
starting a replica.
### Upgrading from one command
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
machine and then every node in turn. Each agent is told the version over its
existing stream, fetches the release for its own platform, checks it against
the release's checksums, keeps the old binary beside the new and restarts,
while its engines keep serving. A node that does not come back on the new
version, ready, with its replicas, stops the upgrade before the next one.
Running it again resumes. Agents from before this have to be moved by hand
once.
### Following a request
Every response carries `X-Request-Id` — the client's own when it sends a plain
one, otherwise one the gateway makes — and the engine receives it too. One
access log line per request records it with the model asked for and served,
the replica, status, time queued, time to the first token, total time,
tokens, and the key, project and user. Usage records keep the id, replica,
queue time and time to first token, and `llmgw ctl request ID` reads one back:
the answer to "why was this slow?".
### Rate-limit headers
Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
tenancy tree will refuse first, so SDKs and editors can slow down before they
are refused.
### Memory sized from the model
The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
weights, plus the KV cache for the context pool from its layers, KV heads and
head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
to 5.25 GB across the models on the live mirror, which the old "weights plus
half again plus two" could not see.
### Security
- **Alertmanager answered anyone on the network**, including requests to
silence alerts. The monitoring README now binds it to loopback, where
Prometheus beside it is its only client.
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
64 MiB of argon2, with no bound on how many ran at once, and the per-address
throttle grew by one entry per address tried. Hashes now run a few at a time
and the rest wait; the throttle expires old entries, and when it is still
full it refuses new addresses rather than dropping live ones — which would
have let a flood of made-up addresses reset the throttle on a real one.
- **Any key holder could grow Prometheus without bound** by asking for made-up
model names, each of which became a metric label. Requests are counted under
the catalog's name for their model, or "unknown".
- **A SQLite backup left a full copy of the store readable** by local users
while it ran. The snapshot is now made in a private directory, and
llmgw-backup.service sets `UMask=0077`.
- **An engine archive could write outside its directory through chained
links.** Archives are unpacked through `os.Root`, and a link may only point
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
follows that rule.
- **Onboarding made every node trust the cluster's root for everything.** It
was added to the system store, so a certificate the control plane signed
would pass for any site with every TLS client on the node. The agent trusts
it through its own configuration only; onboarding no longer adds it, and
takes it out of a node that an older version onboarded.
- **Any node could manage a mirror running mutual TLS**, since one root signs
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
the certificate with that common name, and the mirror warns when mutual TLS
runs without one.
- **The administrator's password stayed in `init.log`.** It is shown once, on
the screen, and the log keeps a placeholder.
- **Auth off on a public address was silent.** The server now warns at start
when the API listens beyond loopback without keys.
- **The actor recorded with a change was whatever the caller sent**, line
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
records the person who ran sudo rather than root.
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
reaches it. It is off. The head itself still has no authentication, and
SGLang still takes its key as a flag; docs/install.md says what to do about
both. The monitoring README says how to keep Prometheus's queries to the
control plane.
### Fixes
- Tests that slept a fixed time and then asserted now wait for their
condition. They are the likeliest cause of CI's intermittent failures on a
loaded runner.
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
`ctl restart` did on its first real use.
- The status each failure is answered with — 499, 502, 503 with and without
Retry-After, 504, 429 — is now tested case by case.
### Releases
Small fixes are published as release candidates of the next version —
v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
notes come from this section, and they are marked as pre-releases. Onboarding
treats a release candidate as a release and downloads it.
## v0.10.3 — 2026-09-30
Security fixes from a review of v0.10.2. Three of them were confirmed on the
live cluster before being fixed.
### Engines demand a key only the router holds
Engines listened on every address of their node with no key. A request from a
neighbouring node reached the model directly, past the gateway's
authentication, quotas and accounting. Each replica's engine is now started
with a key derived from the replica id and a secret in the store. The router,
its metrics scrape, the agent's readiness check and `ctl bench` send it.
llama.cpp and vLLM receive it through their environment, not their command
line. Engines started before this keep running without a key until their
replica is next started.
### The mirror's management API answers only who it was told to
Without `--api-allow`, the management API answered anyone who could reach it,
including every node. It now answers this machine only; the control plane is
let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
add `--api-allow <control plane address>` to its unit.** Downloads started
through the API go to http and https only, and never to loopback, private or
link-local addresses (`--allow-private-sources` to permit that). Model,
version and file names can no longer be `..` or start with a dot.
### The admin listener refuses what a browser page could send
The admin API has no login and trusts whoever reaches it. A page open in an
operator's browser, reaching it over an ssh tunnel, could post a model
definition. The listener now requires Host to be an address or localhost,
refuses cross-site requests, and takes changes only as `application/json`.
A profile's `env` may no longer set variables that make an engine run other
code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
They are refused where a model is defined and dropped again before an engine
starts.
### Fixes
- The agent installs only engine versions that look like one, whatever the
control plane sent; a version of `../x` could have installed outside
`/opt/llmgw/engines`.
## v0.10.2 — 2026-09-29
- After `server init`'s screen closes, only the administrator's password and
the next steps are printed, not the whole run again under the screen that
just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
only, and printed in full when init fails.
- The screens' spinner and text field are llmgw's own rather than from
bubbles, one dependency fewer. Every llmgw command still asks the terminal
for its colours as it starts, which bubbletea v1 does on purpose; a terminal
that never answers waits for its timeout.
## v0.10.1 — 2026-09-29
### `llmgw ctl fabric-test`: two nodes, rail by rail
`llmgw doctor` checks a node on its own, and every check can pass while two
nodes still cannot reach each other: a cable into the wrong switch, a
partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
one rail at a time, through the agents — no SSH between nodes, and perftest is
installed the first time — and grades each against its link's rate: ok at 80%,
slow at 50%, fail below or when nothing got through.
Agents now say which requests they answer beyond the usual (`features`), so
one too old for the test is refused at once instead of waited for.
Written against perftest's real output and tested with it, but not yet run
over real InfiniBand or RoCE: this cluster has neither.
### The mirror on its own machine, in two commands
`llmgw mirror init --control-plane <address>` sets a machine up as the model
mirror: the user, `/srv/models`, and a service serving files to every node on
:8081 and the management API on :8082. `server init --mirror
http://<mirror>:8082` uses it instead of running one beside the control plane.
It took a unit written by hand and an edit to `server.yaml` before.
The management API can fetch and delete models; `mirror serve --api-allow`
makes it answer only the addresses given and the machine itself, and `mirror
init` gives it the control plane's. With `--no-node`, the starter model is
fetched and catalogued anyway, unpinned, and starts on the first node added.
Tried on three empty VMs: mirror, control plane and a node, and a request
answered by the starter model on the node.
### `server init` asks, and `node add` takes several nodes
- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
administrator's email, the other names nodes will reach the machine by
(showing the addresses it already found), SQLite or PostgreSQL, and whether
to run the mirror, a node and the starter model here. Then its steps fill
in as a list, downloads updating in place, and the administrator's password
stays in a box until the end — and is printed again after, with the whole
run, so it is in the terminal's history. The last screen says how to add a
node and to back up the authority's key. `--plain`, or no terminal, prints
lines as before.
- **`llmgw node add a b c`** adds each in turn and ends with which were added.
A fabric named for one node's adapters is recognised on the next by what the
adapters report, and not asked again.
Both tried on empty VMs: init from the form to a starter model answering, and
two nodes added to it in one command in 40 seconds.
## v0.10.0 — 2026-09-29
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
installed by the nodes themselves; people in many places at once; and a chat
page, editor setup and a model explorer for the people using it.
### Independent InfiniBand fabrics, told apart and drawn
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
with several independent fabrics, two given one name would have had a replica
spread across both — started, then hung on its first collective. Now:
- **Identity.** Each active InfiniBand port reports the master subnet manager
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
asked once a minute through one port per fabric. Ports whose managers are
listed together are one physical fabric, so a failover does not split one in
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
installs `infiniband-diags` where there are adapters and no `sminfo`.
- **Placement** divides a declared fabric by that identity and forms a replica
on one part only. A node whose adapters for one name are cabled to two
fabrics is refused; a node that cannot say is kept apart from those that can.
- **Subnet managers.** The cluster page lists each fabric's master, its
priority, its standbys and its master changes over the last day, and warns
when a fabric has no standby, when its ports disagree on the master, when a
port is up with no subnet manager (INIT), and when one name covers two
fabrics or two names one fabric.
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
each heartbeat: state, physical state, rate, error and data counters. A port
going down takes its fabric off the node within seconds, and back when it
returns — it used to wait for the agent to restart.
- **Maps.** Every five minutes one node on each fabric
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
the fabric: switches in their tiers, adapters under their leaves, cables
coloured by rate, degraded ones — trained below the rest of their kind —
marked in red, rail alignment, and the cluster's own adapters live, with
their traffic running along the cables. It zooms, pans, highlights what a
switch or adapter touches, and opens detail on a click. Placement prefers
the nodes of one leaf and puts nodes with a degraded cable last.
### NVLink, on every build and across nodes
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
which was the only source of NVLink peers: on every NVLink machine the
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
peers, each card's NUMA node, and how near each network adapter is.
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
within it, then — for a multi-node shard — nearness to the fabric's adapters,
then NUMA nodes spanned. It used to grow one card's peers, which took cards
joined to that one but not to each other.
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
found from each GPU's fabric state, ClusterUUID and CliqueId, with
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
domain first, on nodes whose every card is joined and whose IMEX is up, and
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
### Multi-node launches fixed before they meet hardware
- **vLLM launches multi-node its own way by default.** An empty
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
it now means `mp`, every node running `vllm serve` with `--nnodes`,
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
documents. The `mp` path used to give workers no rank and the leader's
address as their own. Set `launch_mode: ray` to keep Ray.
- **NCCL bootstraps over the node's own interface.** The agent's default of
`eth0` meant it never looked for the interface holding its management
address, and the control plane sent one name for every node. A node on
`ens3` or `bond0` had NCCL bind nothing.
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
address; NCCL's default is a v1 link-local one that does not route.
- A Ray worker now waits for its head before joining.
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
### `llmgw doctor`
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
wrong and how to fix it, most serious first: NVSwitch without the fabric
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
shard on each fabric would be given. `--json` writes every fact it read, to
send back; it exits non-zero when something fails.
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
model, where llama.cpp runs several times slower and nothing else looks wrong —
and whether vLLM can be installed on the node, and why not.
### vLLM installs itself on GPU nodes
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
that can run it, with nothing done by hand. A Python 3.12 environment is built
by uv — itself a GitHub release checked against its checksum — from PyPI,
resolved as PyPI stood a day after that vLLM was published and installed with
`--require-hashes`, so every node gets the same files whenever it is built. The
resolved list stays in the environment as `llmgw-requirements.txt`.
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
25 GB free while it installs. A node that lacks one says which on the
cluster page; a node with no GPU skips vLLM without a word.
- gcc is installed with it, since Triton compiles its kernel launcher on the
first model load.
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
version.
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
itself was not started there.
A node is asked only for engines its agent says it can install. An agent older
than this release says nothing, and is asked for llama.cpp alone, so a cluster
half way through its upgrade does not show every node as failing vLLM.
### `llmgw node add`: a node in one command
Adding a node was a bash script from the repository, not installed anywhere,
that needed the node's name, the control plane's address and a dozen flags.
It is now part of the binary:
```
sudo llmgw node add ubuntu@10.0.0.21
```
run on the control plane. The node id is the machine's hostname and the
address its agent dials is this machine's on the route to it, checked against
the server certificate. It installs llmgw on the node — the release for its
architecture, fetched and checked there, which works on RHEL too — runs
`llmgw doctor` there and stops on a problem, issues and installs the
certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
the node to join and install llama.cpp.
In a terminal it is a checklist that fills in, and asks what a script cannot:
which fabric each group of active adapters is on — grouped by the subnet
manager or RoCE network they report — and whether to go on past a problem.
Elsewhere it prints one line per step. Running it again on a node is safe.
`llmgw node remove` forgets a node that is gone for good, which otherwise
stays listed as lost until the control plane restarts; only a lost node, or an
unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
longer lists vLLM as skipped on every CPU node.
Tried on an empty VM, from the terminal screen and plain: joined with
llama.cpp ready in under a minute, and 17 seconds to add it again.
### People belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the
gateway before that. Now:
- **Memberships.** A person can be a member of any number of organizations,
teams and projects, or of none. A member of a team or organization may use
every project in it. `llmgw users join/leave`, and the people page adds and
removes them from either side. Approving no longer requires a project.
- **Personal keys.** A key can belong to a person with no project. On
`/profile` a person chooses what a new key spends against: one of their
projects, or Personal. `llmgw keys create --for <person>` does the same.
`auth.personal` sets default limits for personal use, or turns it off.
- **Limits for people.** `--scope user` limits a person across every key,
project and the chat page. `--per-member` on an organization, team or
project gives each member their own share there. Both are on the quotas
page, and are enforced alongside the tree, each in its own pool.
- **The chat page** asks what to spend against when a person has more than one
choice, and checks it on every message.
- **Spend is recorded with who spent it**, so a person's budget survives a
restart, personal use and the chat page included.
Leaving a project revokes the keys the person made there, unless another
membership still reaches it. A personal key keeps no conversations or notes:
those are kept per project.
### An organization's own page
A member of an organization can be made its **manager** (`users join --org …
--role manager`, or the star on the people page). A manager gets `/org`: the
organization's usage over time by team, project or person, its teams and
projects with limits and spend, and its members — read-only, and only what was
spent in its projects. Administrators can open every organization's page.
### The chat page
- A model that is not running shows a progress bar while it loads — waiting
for a node, downloading, starting — and the message is sent once it is up,
rather than failing after 45 seconds with "the model is loading".
- A finished reply is rendered as markdown, by the server under the model
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
- Stop, while a reply streams; its tokens, speed and time to first token under
it; a system prompt and a temperature; and the conversation kept in the
browser across a reload — the gateway still keeps none.
### Connect an editor, and the Responses API
- The account page makes a key for **Continue**, **Codex** or **Cline** and
shows what the tool needs, filled in: Continue's assistant file, Codex's
provider and profile, Cline's four settings. Continue and Codex get a one-line
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
no secret — the key comes from the environment — keeps a copy of whatever it
replaces, and leaves the rest of the file alone.
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
chat completions and back — instructions, messages, function and custom
tools with their calls and outputs, usage, streamed or not — down the same
path as any request.
### Your own use
The account page (was "Your keys") shows the last thirty days of a person's own
use and, for each place they spend, a bar for every budget that applies — their
own, or a project's shared with others, said as such.
### The mirror page
- **Explore models** opens on what is trending on Hugging Face, with views for
new and most downloaded, filters for task and size, and each repository's
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
base model and license. "Only what these nodes can run" is on by default.
- A repository opens beneath its row with its model card, and a download
started there starts in place, its progress followed in the table above.
- What the mirror holds is grouped by model family.
- An importance matrix published beside a model's quantisations is no longer
offered as a model.
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
demand, or stopped — and it can be changed there, without the CLI. The serve
form asks the same: on demand loads the model when somebody asks for it and
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
mode and says to change it there, because the file is read on every start.
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
was downloaded, catalogued and placed, and failed on every node it was tried on.
The architecture was in the first kilobytes of the file all along. Now:
- The mirror page's registry listing shows each file's architecture and, for
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
it. A vision projector (`mmproj`) is marked as not a model on its own.
- A fetch reads the file's header first — a range request, not a download —
and refuses one that cannot be served, unless "fetch anyway" is ticked.
- Serving from the mirror and `llmgw ctl model add` refuse the same.
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
not refused, only not checked.
### Each node's CPU and memory, and a page per node
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
used and available, and each engine's cores and resident memory. The cluster
page's node table shows CPU and RAM as bars; each node has a page with its
current load, its engines, its fabric ports, its GPUs, and an hour of charts
from Prometheus.
### The admin pages
- The cluster page leads with capacity, nodes, models and replicas, then usage;
it refreshes in place instead of reloading, and its usage table names people
and projects instead of ids.
- The usage page's legend gives each line's total, not its last hour.
- The people page keeps an account on one line and its rarer actions in a menu.
- Every page fits a phone: a wide table scrolls in its own box.
- Static files are named by their content, so a deploy's new styles and
scripts reach browsers at once instead of a day later.
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
order, the page you were on sometimes listed and sometimes not. Every signed-in
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
left for an administrator, Chat, Account and Sign out on the right for
everyone — with the current page marked. The chat and key pages name the
person's project by its path rather than its id.
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
its organization held it to sixty requests a minute. Each limit now shows the
value it is held to from above, muted and marked ↑, with where it comes from on
hover; a larger value set below a stricter one says what caps it, for rate and
token limits as well as budgets. Spend and budget share a column with a meter
that turns red past 90%. The tree is indented like the people page, opening a
form or saving one no longer reloads the page, and the explanations moved under
"How limits combine". Projects are placed under their team by id rather than by
the start of their path, which misplaced them when two organizations shared a
name.
### The project's website, in every gateway
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
the reverse proxy configurations, and the release's files to download. It is
served without a key like `/`, whose footer links to it, and says nothing about
the installation. Its fonts are in the binary, so it makes no request elsewhere.
The published website is rendered from the same template.
### Metrics
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
### Fixes
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
rewriting read it as the start of a string. Comments are now skipped.
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
- Usage by team, project or organization, and the spend on the quotas page,
left out every request made on the chat page: they were joined through the
key, and the chat page has none. They now count where a request was spent.
- `/org` was answered with "no api key was provided" for a signed-in browser;
a test now drives every session page through the authentication middleware.
### Upgrading
Upgrade the control plane first, then the agents: an older agent is served as
before and simply reports none of the new measurements. Existing agent units
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
that set `mgmt_interface` keeps it; one that did not now uses the interface
holding its management address rather than `eth0`. Multi-node vLLM replicas
start with `mp` unless `launch_mode: ray` is set.
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
attributes past usage to people where a key had an owner. On SQLite it rebuilds
the keys table; it was run against a copy of a production PostgreSQL store
before release. `llmgw users move` is gone: use `join` and `leave`.
vLLM is installed on every GPU node that can run it, once its agent is
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
replaced by `llmgw node add`; it still works, for a machine holding the
authority but not this llmgw.
## v0.9.2 — 2026-09-28
### A new control plane serves a model by itself
`llmgw server init` now also runs a model mirror on the same machine, makes the
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
and every step is skipped on a rerun when it is already done.
Not yet run end to end on a real machine.
## v0.9.1 — 2026-09-28
### The mirror page is one line per model
What the mirror holds used to take two table rows per file — seven columns and
the address under them — so a sharded model filled the screen. Now each version
is one line: model and version, the file (or "4 files" for a set), total size,
and which catalog models use it. The address moved into a "copy address" button;
the checksum is in the file's tooltip. A version already in use is not offered
for serving again, and removing one warns that the catalog uses it. The header
counts versions rather than files.
## v0.9.0 — 2026-09-28
### Managing people and the tenancy tree
Organizations, teams and projects can now be renamed, moved and removed, and an
approved person can be moved to another project or removed — none of which
could be done at all before, from the web or the CLI.
- **`/admin/users`** shows the whole tree, each project opening onto its people
and keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level. A search
over every account sits below it. Changes update the page in place (htmx,
embedded in the binary) and it still works with scripts off.
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
- **`llmgw users move`** and **`llmgw users delete`**.
- **`llmgw keys list --all`** lists every key with its project and owner.
- Every command that takes a project, team or organization accepts an id, a
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
be omitted when there is only one. Users are named by id or email.
Moving a person takes the keys they made with them; keys an operator issued stay
with the project. Only empty things are removed, and a project that ever had a
key is never removed, because usage history refers to it.
### Changes
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
built from the web has no project of that name. With one project it is
chosen; with several the command lists them.
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
rather than micro-units. `--budget 50000000` now means fifty million units;
divide existing scripts' figures by 1 000 000.
### Fixes
- `keys init` could silently move an existing team or project to another
organization or team by redeclaring it. It now refuses.
- An account rejected while pending and then reinstated became active with no
project, and failed at its first key. Reinstating one is refused; approving
gives it a project. Rejecting a signup now removes it.
## v0.8.8 — 2026-09-28
### Fixes
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
executable file not found". The PostgreSQL project's own packages (PGDG)
provide `postgresql-server` too, so dnf can answer that name with
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
reads the data directory from it, and runs `initdb` itself when the directory
is empty. A PostgreSQL that was already installed is used as it is.
### Known gaps
Not yet run on a real machine. Three arrangements are not handled: a
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
and says so), one listening on a port other than 5432, and several PGDG
versions where the highest is not the one on 5432.
## v0.8.7 — 2026-09-28
### PostgreSQL in one command
```
sudo llmgw server init --postgres
```
Installs PostgreSQL with the machine's own package manager when there is none,
creates the `llmgw` role and database with a generated password kept in
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
already running on SQLite it stops the gateway, copies the store across so
every account and key survives, points `auth.store` at PostgreSQL and starts
the gateway again; if the move fails, the gateway is started again on SQLite.
Given on the first run, there is never a SQLite store.
`server.yaml` is edited in place with its comments intact, and the previous
file is kept beside it. A PostgreSQL that init did not install is left sized as
it was, since something else may depend on it.
The PostgreSQL installation itself has not yet run on a real machine; the move,
the configuration edit and the rerun behaviour are tested.
### Fixes
- **A new control plane with no models did not start.** A configuration with
no static endpoints and no models was refused as having nothing to serve, so
under systemd the unit restarted every two seconds and nothing listened on
8080. It now runs as a control plane with an empty catalog.
## v0.8.6 — 2026-09-28
### Reverse proxy configurations
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
port, and both are attached to every release beside the binaries. They do not
buffer streamed replies, their read timeout outlasts a model's wait before its
first token, their body limit matches the router's, and they set
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
cookies `Secure` from the first and shows the second as the base URL on its
landing page.
## v0.8.5 — 2026-09-28
### A control plane in one command
```
sudo llmgw server init --admin you@example.com
```
Creates the llmgw user and its directories, the certificate authority, the
control plane's certificate for the machine's hostname and every address it
has, `server.yaml`, the default project, and an administrator whose password is
generated and printed once — then installs the systemd unit, starts it and
waits until it answers. What used to be two steps of install.md is now this.
Each step is skipped when its result exists, so running it again finishes an
install that stopped and changes nothing on one that did not. `--host` adds an
address the machine cannot see of itself (a DNS alias, a NAT or floating
address) and reissues the certificate keeping the old ones. It never overwrites
`server.yaml` or an authority, and certificates issued by an authority kept on
another machine are left as they are.
### Fixes
- **The configuration install.md showed could not start.** An empty catalog
without `control.enabled: true` is refused as having nothing to serve. The
documentation and the file `init` writes both set it.
## v0.8.4 — 2026-09-28
### Node logs on the admin page
Each node on the cluster page links to its logs — the agent's own, and the
output of every engine running there — with an option to follow them. `llmgw
ctl logs NODE` reads the same. They come over the agent's existing stream, so
no ssh, jump host or root is needed to see what a node did. The agent keeps its
last 2000 lines in memory; each engine its last few hundred.
## v0.8.3 — 2026-09-28
Found by building a node from nothing with v0.8.2.
### Fixes
- **A new node's first engine install failed on apt's lock.** cloud-init and
unattended-upgrades run apt in a machine's first minutes, and
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
package command held up by a lock now waits for up to ten minutes, and says
so; a failed install is retried after a minute rather than ten.
- **A new node reported no disk** until its first download created the weights
directory. It is created when the agent starts.
- **Onboarding took one control plane address.** It takes several, comma
separated, so a new node fails over like the rest.
## v0.8.2 — 2026-09-28
A node no longer runs whatever engine somebody once put on it. It installs its
own.
### Engines are installed by the node
`engines:` in server.yaml names the engines every node has and their versions,
defaulting to the llama.cpp build this release was checked against (b10991).
Each agent converges on it: it picks the build for its hardware — CUDA with its
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
packages that build needs with the machine's own package manager, downloads the
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
anything that would land outside its directory, and runs it. `ldd` names a
missing library or a system too old for the build ("libstdc++.so.6 is too old
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
again every ten minutes.
An engine is reported only once installed and seen to run, or when the agent's
configuration names it on purpose. A node that has installed nothing yet is not
a candidate for any model, where before it was a candidate for all of them.
`ctl nodes`, the cluster page and placement's refusals say where each
installation stands, and warn about a CPU without AVX2.
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
installed this way and must still be named in the agent's configuration.
### Onboarding
Onboarding installs llmgw from the CI release for the node's architecture,
checked against the release's checksums, and writes an agent configuration of
only the control plane's addresses and the TLS files.
### Fixes
- A race on the weights fingerprint, which two heartbeat loops touched during a
failover.
## v0.8.1 — 2026-09-28
Nothing here changes a configuration format: every file that worked with
v0.8.0 still works. What changes is how much of it needs writing, and how much
of running the cluster can be done from a page rather than a shell.
### A node describes itself
An agent file used to name its node, its address, the path to llama-server and
its weight directories — different on every node and kept true by hand. The
agent now finds these out: its id is the hostname, its address is the one on
the route to the control plane, llama-server is looked for on PATH and in
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
Memory is re-read on every heartbeat and installed engines once a minute, so a
VM given more RAM or a node given an engine becomes a candidate without an
agent restart; the control plane logs both. The agent file shrinks to the
control plane's addresses and the TLS files, the same on every node.
### Models from the mirror, without writing YAML
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
and serve a mirrored version from its row — name, source, engine and an
estimated memory requirement filled in. A model can say what it is for, and
`/v1/models` shows it. Placement no longer hands the same memory to two models
or puts an engine on a node that lacks it, and says when a replica runs an
edited profile.
### Tenancy, quotas and usage on pages
The tenancy tree, quotas and budgets are managed from the admin pages; an
organization's budget bounds its teams, a group's quota counts against the
group, and a budget that cannot be spent against says so. `/admin/usage`
charts requests, tokens and spend over time by organization, team, project or
person, and a request is priced by the model that served it.
### Watching the cluster
The cluster page has its own charts, backed by a year of Prometheus history,
with alerts that were made to fire before being trusted — including for the
disk the metrics share with the models.
### Chat
The picker offers each model once rather than once per alias, never asks for
more tokens than the model allows, and offers a model that loads on demand —
sending it a message is what loads it.
## v0.8.0 — 2026-09-27
The store stops being a file on one machine's disk. PostgreSQL, a way to move
an existing installation onto it without reissuing every key, and the two
things a second control plane needs so that running one is safe rather than
clever.
### A second control plane, and a way to fail over to it
Leadership is a lease in the store, so two control planes against one database
cannot both place work — which they otherwise would, both seeing a model
wanting replicas and none running, leaving twice the replicas each holding GPUs
the other believes are free.
A standby also refuses agent streams. Commands travel down the stream the agent
opened, so a standby holding one is a node the leader cannot command, and the
leader's placements would fail with "no agent connected" for a node that is
plainly connected. Agents therefore take a list of addresses rather than one,
trying each in turn: a standby refusing, a stopped control plane refusing the
connection and a crashed one never answering are the same thing from a node.
Measured on the cluster. Leadership moved three seconds after the leader was
stopped — it released the lease on the way out rather than letting it lapse —
and all five agents had registered with the standby six seconds later. The
standby adopted every replica rather than restarting any: six replicas, all
ready, engines that never stopped serving.
Everything is unset-means-leader, so a single control plane and every
deployment that existed before this behave exactly as they did.
What this does not do is move client traffic. The standby served inference
correctly on its own port while the public address returned 502, because
nothing in front of them moved — that needs a load balancer or a shared
address, and it is the remaining piece.
### PostgreSQL
The store runs on PostgreSQL, chosen by what the address looks like: a path is
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
address out of a file so a DSN carrying a password stays out of a configuration
that gets copied and pasted into issues. Every existing installation keeps
working untouched.
One set of queries rather than one per database, because the SQL here is almost
entirely portable and two copies would drift one query at a time. The rewrite
of placeholders is a wrapper around the handle rather than a change at sixty
call sites — which found six places reaching past it, including all three
manual transactions and two prepared statements, every one of which worked on
SQLite and would have failed on PostgreSQL.
Exactly one query is not the same sentence in both: the note search, MATCH
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget. It uses the 'simple' configuration
rather than English — these are an agent's own notes, in whatever language it
chose, and stemming English rules over Turkish would quietly stop finding
things.
`llmgw keys move-store` copies an existing store rather than starting a fresh
one, because keys are stored hashed: a migration that drops them cannot
recreate them, and every person and every script would need a new one.
Testing it against a real database found three things nothing else would have.
`schema_migrations` is created in Go rather than in a migration file, so it
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
and a millisecond timestamp does not fit. The copy order followed the order the
migrations create tables in, and `users` is created after `api_keys` even
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
difference pgx cannot see, so it arrived as NULL.
The driver is pgx, which is pure Go: the whole binary is built with
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
and a cgo driver would take that away from every deployment, including the ones
staying on SQLite.
## v0.7.0 — 2026-09-25
What a cluster needs to be lived with rather than demonstrated: somewhere to
keep a conversation, somewhere to keep a note, and an honest answer about the
disk underneath both.
### Memory notes for agents, and threads without API keys
Notes an agent writes for itself and reads back in later conversations, offered
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
`forget`). A note is a key, text and tags, searched by words or tag, private to
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
tools into chat requests: an agent that is not given the server never sees
memory.
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
and the retention is required, as it is for threads.
On a gateway with no authentication, a `threads:` block (`store`,
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
nothing refuses it with an explanation instead. The shared pool is for one
trusted machine: anyone who can reach the port and knows an id can read it.
### Conversations that can move between models
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
llmgw replays the stored history to whichever model is serving, which is what
lets a conversation begun on a large model continue on a small one — including
when a spent budget moved it there and the caller never chose.
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
stores no message content anywhere, as it never has, and that default is worth
more than the feature. The retention is required rather than optional, because
a conversation store with no expiry is a transcript of everything a team ever
typed kept forever by omission, and it is measured from the last turn so a
conversation in progress does not expire underneath it.
History that does not fit the serving model is trimmed oldest-first and the
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
are never dropped — losing one silently changes what the model is for while it
carries on answering — and neither is the turn the caller just sent.
Threads are scoped to a project, and reading, writing or deleting one that
belongs to another answers exactly as one that never existed: two answers would
let a caller map which ids are real in projects they cannot read.
Capturing the reply is opt-in and bounded, so a request without a thread
accumulates nothing on the hottest path in the system, and one with a thread
stops at a cap rather than growing to it.
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
operator: the person who said the thing should not have to ask someone to read
it before it can go.
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space in the same struct was discarded, so the only way to find
out why a node was filling up was to ssh to it.
Nodes now report it, per filesystem rather than per configured directory: free
space belongs to the disk, and two model directories on one disk are one pool
of bytes. Shared filesystems are kept out of a node's own figure, because space
on a filer belongs to everyone mounting it and twenty nodes would otherwise
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
Prometheus, and a node that reported nothing reads as unknown everywhere — an
older agent must not look like a full disk.
A profile may declare `min_disk_gb`, and a node without the room is no longer a
candidate, with a reason `explain-placement` prints. Skipped on a node that
already holds the weights: it needs no room for them, and without that the node
best placed to run a model is the one most likely to be rejected.
`ctl weights` lists model files llmgw downloaded for models no longer in the
catalog, and `--collect` deletes them. The design is mostly about what it will
not do. Only files llmgw fetched are candidates, and that is a calculation
rather than a heuristic — the cache name is a hash of the source, so the wanted
set is computable exactly. Anything an operator put on a node is left alone
however unreferenced it looks. Each node checks every path again before
unlinking it, refusing anything outside its cache directory or without the
cache name, so a control plane that is wrong cannot talk a node into losing a
model that took a week to copy.
### Fixes
- **A capability was lost at an indirection.** The controller is handed a
forwarding dispatcher, and reaches past its interface by type assertion to
find what lies beyond scheduling. Deleting weights was implemented on the
agent service and not on the forwarder, so it was simply absent at runtime —
with the whole suite green, because the dispatchers the tests use do not
implement it either and its absence there is the expected case.
- **`ctl tool-check` blamed the engine for what the model did.** It had one
verdict for a failed attempt and told operators not to try another model.
Measured: the engine renders the tool prompt correctly, the coder models do
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
the same engine returns a populated `tool_calls` array. A failed attempt is
now two verdicts, told apart by whether the reply used the wrapper.
## v0.6.0 — 2026-09-18
Phase 4: a replica can be bigger than one machine, and the fabric it spans
decides where it is allowed to go. Around that, everything a cluster needs
before anyone other than its operator is let near it — keys, budgets, a way in
through a browser, and enough visibility to answer "why is nothing scheduling".
### Multi-node replicas
A replica is now a set of shards with ranks, a leader and a bootstrap address,
rather than one process. Two invariants are enforced rather than documented: a
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
several times slower, and nothing says which.
A replica missing a shard is not ready. It was previously derived from the
shards that had reported, so a two-node replica with one node up looked
servable and took traffic into a model that could not answer.
Adoption after a restart happens one agent at a time, because that is how
agents reconnect. Merging them as they arrive is what keeps a multi-node
replica whole instead of being torn into fragments that are then killed as
strays — and an adopted engine is health-checked before it is called ready,
rather than being assumed alive because its process exists.
### Fabrics are distribution domains too
An isolated fabric is not only a compute boundary: weights present in one are
not present in the other. Node weight inventories are reported and summarised
per fabric, so "the model is on the cluster" is never mistaken for "the model
is reachable from the nodes that would run it". A fabric name that matches
nothing a node actually has is refused at registration rather than becoming an
unschedulable replica hours later.
### Who may use it
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
still no cgo). Rate limits and budgets are enforced on the request path; usage
is written off it, in batches, so accounting never adds latency to inference.
Budgets survive a restart — counters are rebuilt from the store, because a
limit anyone can clear by bouncing the process is not a limit.
A spent budget is now served a cheaper model where the catalog offers one,
rather than refused. For a team trying models out, being dropped to something
smaller is a far better answer than a wall, and the response says which model
answered.
People can sign up, be approved, and issue and revoke their own keys. A key is
shown once; only its hash is kept.
### The browser surface
A landing page that says how to connect, a read-only cluster dashboard, and an
in-browser chat that never puts an API key in the browser. The dashboard shows
who is using the cluster and what they are running — per person and per model,
with requests, tokens, errors, spend and latency.
All of it is served from the client-facing port behind a session, deliberately
not from the admin port, which also serves pprof — and pprof can dump the heap
that holds every API key in plaintext.
### Models without a restart
The catalog is live. Models can be added and removed from the command line or
the API, take effect immediately, and outlive a restart; configuration still
wins over the store, and the whole catalog is validated before anything is
written.
A model is only listed once it can actually serve. It was listed as soon as it
was known, so the first thing a new user did was ask for a model that returned
503.
### Loading on demand, and letting go
A model whose scaling allows it is started when someone asks for it, and
stopped when nobody has for a while. An idle replica is evicted to make room
for one that was asked for, with guards that exist because each is a way to
lose work quietly: only on-demand models, only single-node replicas, never one
that has never been used, and never one younger than its minimum lifetime.
Unknown is not idle. A replica with no recorded use is left alone rather than
treated as idle since the epoch.
### Knowing whether it will run before it runs
- `ctl plan` works out whether a model fits the cluster and how it would be
split, in the precision the chosen engine can actually serve — recommending
a quantisation the engine cannot load would cause the failure the command
exists to prevent.
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
installed. Flags drift between releases, and the alternative is finding out
during a model load.
- Placement now knows a model's context length and memory footprint, and node
RAM, so a CPU profile is not placed on a machine that cannot hold it.
- Placement waits for the fleet to settle after a restart rather than deciding
against a half-reported cluster.
### Weights
Nodes report the model files they can see, without llmgw owning the storage:
a shared filer, an image exported over the fabric, rsync or a copy made by hand
all look the same to placement, which only cares whether the files are there.
Reporting nothing means "not asked to look", not "has nothing".
Where the simple arrangement is not enough, a node can fetch a model by URL —
single-flight, resumable, written to a temporary name and renamed atomically,
so a killed download never leaves something that looks like weights.
### Seeing the fleet
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
scrape time rather than from a background goroutine, and served on their own
endpoint rather than through the one that also exposes pprof. The same view is
in the dashboard, so nothing else needs installing to see it.
### Onboarding a node
Adding a node is one command. There are systemd units, a private CA and a
from-nothing install guide that starts with a machine that has no CUDA and no
binary.
### llama.cpp, and CPU profiles
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
API without a GPU, so placement, supervision, scraping, scheduling and admission
can all be exercised on ordinary machines.
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
engines that can actually serve without a card may omit it — for vLLM an omitted
class stays an error, because otherwise a typo becomes a replica placed on a
machine with no hardware and diagnosed minutes later during model loading. CPU
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
allocation to stop two of them sharing a machine's cores.
### Metric mappings are per engine
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
than one for the cluster. A mixed fleet has two names for the same three numbers,
and a single global mapping left half of it unmeasurable.
A mapping that leaves a metric unnamed now means the engine does not publish it,
and the field stays zero. Naming one the engine does not serve is still a contract
violation. `Running` and `Waiting` can never be omitted.
### Mutual TLS on the agent stream
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
certificates on both ends, and the control plane refusing any agent whose claimed
node id does not match its certificate's common name. Public authorities cannot
supply these: they stopped issuing the `clientAuth` usage.
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
agent turn it on.
### Fixes
- **An agent told to stop did not stop.** Cancelling the context is not enough —
HTTP/2 watches it while the request body is being written, and a bidirectional
stream spends its life blocked on the read side. Measured on a deployed node:
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
- **An agent could not explain why it was refused.** The session cancelled its own
context and then asked that context whether it was shutting down, so every
stream error read as a clean close: no message, and the backoff reset.
- **A broken metric contract was invisible.** It was logged per scrape at debug
level, so a cluster could look healthy while every routing decision was made
blind. It is now a warning, once, with a matching line when it clears.
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
over about 2 KB was undercounted, so tokens-per-minute limits let through far
more than they were set to.
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
bfloat16 does not fit two 80 GB cards. It does.
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
already written the response header, which is the one point at which headers
stop being changeable. A client had no way to tell a fallback from the model
it asked for.
- **Fleet metrics published nothing.** They were registered after the registry
had been built, so the collector existed and was never scraped.
- **GPU discovery broke on older drivers.** A field added for utilisation was
required rather than optional, so a driver that does not report it produced no
inventory at all instead of an inventory without that number.
- **Replica ids and ports were reused** while the processes holding them were
still alive.
## v0.5.0 — 2026-09-16
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
control plane places replicas, and `ctl` makes all of it visible.
This release covers both phases because their exit criteria depended on the same
piece — the agent stream — and neither was demonstrable without it.
### ctl
```
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
```
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
the answer when a node looks idle but nothing schedules onto it.
- `explain-placement` prints every candidate considered and the reason it was
rejected. "Unschedulable" on its own says nothing about which of six
constraints bit.
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
is placed elsewhere, and the original is stopped only once the replacement is
serving. Bounded by a grace period, because draining the only node that can
host a model would otherwise never finish and leave it undrainable.
- `cordon` is the weaker operation: no new placement, nothing disturbed.
The management API is on the admin listener, not the client-facing one. The two
have different audiences: one serves inference to whoever holds an API key, the
other lets a caller cordon nodes.
`ctl logs` is not here. It needs a request/response pair the agent protocol does
not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
### Phases 2 and 3, in detail
Written as the work landed, which is why it reads as a log rather than as
notes. Everything here shipped in v0.5.0.
### Added
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
without IB is not an error — it is a single-node island.
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
would force the whole binary, server included, to be built with a C
toolchain. Health flags anything that should keep a card out of placement:
uncorrected ECC errors, and memory already held by another process.
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
always match the marketing name: an L40S is sold as 48GB and reports 44.99
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
box, because the number decides whether a model fits.
- **Engine adapters** for vLLM and SGLang: command and environment
construction, kept free of process handling so it is testable without GPUs.
vLLM supports both Ray and multiprocessing launch modes, because which one
works depends on the version.
- **NCCL environment generation**, the part plan §5.3 marks critical.
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
NCCL pick an adapter on another fabric where the peer is unreachable, and the
job hangs rather than failing. Bootstrap is pinned to the management
interface for the same reason.
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
compute capability 8.9 rather than silently falling back to a slower path.
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
process group and are stopped as a group, because engines fork workers that
hold GPU memory and outlive a signal sent only to the leader. Bounded log
tail, a state machine that separates Starting from Loading (a 70B model takes
minutes, and an operator needs to know that is normal), and orphan adoption
through pid files fingerprinted with the command line — pids get reused, and
adopting an unrelated process is worse than leaking the original.
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
says the weights loaded. Routing on `/health` alone sends traffic into a model
that is still loading.
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
as a table or JSON. Run it before adding a node: it is the fastest way to find
a card another process is using, or a port with no subnet manager.
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
a tensor-parallel group off the slow path inside a node. `Discover` falls back
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
topology. CI compiles the tagged build so it cannot rot unnoticed.
### Not verified
The NVML path compiles but has never run against a driver — there is no GPU
machine to try it on. Treat the NVLink domains it reports as unverified until
someone runs `llmgw agent inventory` on a real node.
- **Control plane wiring** (`internal/control`): Ready replicas become router
endpoints, with no static configuration involved. The router keeps live state
for endpoints that survive an update — inflight counts, breaker state, drain
rate — so a replica that was already serving does not start from cold because
another one moved. The sink is an interface, so when the router and control
plane split into separate processes this becomes a watch stream and nothing
above it changes.
### Phase 3 exit criterion, measured in `test/e2e`
A node hosting a live replica is killed while four clients stream continuously.
The control plane marks it unreachable, then lost, rebuilds the replica on a
survivor, and republishes. Across three runs: **~370 requests completed, zero
failures**, and no replica remained on the dead node.
This proves the control logic, not the transport: the agent fleet is simulated,
and the real path still needs the stream.
### A real bug the race detector caught
`startReplica` read the replica's fields after publishing it to the map, and
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
status reports, which arrive from another goroutine and mutate exactly those
fields. Under the simulated fleet the reports came back fast enough to hit the
window every time. Everything is snapshotted under the lock now and the
functions work from values.
### The agent stream
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
carrying registration, heartbeats and shard status upward, and start, stop and
drain commands downward. The agent dials out, because GPU nodes sit behind
firewalls far more often than a control plane does and an outbound connection
needs no inbound rule on every node in the fleet.
- **Registration carries the whole inventory**, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it — they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
the reconciler's dispatcher: it holds the open streams, so it is the only
thing that can deliver a command to a node.
- **Agent** (`internal/agent`) validates a profile before launching anything —
an engine that refuses a profile does so after minutes of loading — then
supervises the process and reports every transition, including the engine's
output tail on failure.
- **Reconnects forever with jittered backoff.** A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock the control plane over again. Engines keep serving throughout: the
router talks to them directly.
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
any model declares a profile. The static endpoint list remains for
router-only runs against sim-engines.
### Verified with real binaries
Server and agent as separate processes over a real stream: the agent registers,
a replica is placed, an engine is launched through the vLLM adapter's actual
argument construction, streaming traffic is served through the router, and
killing the agent moves the node through unreachable to lost with placement
explaining exactly why it can no longer run anything.
### Surviving a control plane restart
Not persisting replica state left one hole that had to be closed before it
mattered: restarting the control plane orphaned live engines. They kept holding
GPUs the inventory believed were free, so the next placement failed to allocate
with an error naming the new process rather than the old one.
The fix is on the agent, not in a database, and it cannot disagree with reality
the way a stale row can:
- The supervisor now writes the full shard spec, not just a pid and command
line. A pid identifies a process; the spec is what lets it be described back
to the control plane and matched to a replica.
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
waited on — it is not this process's child — so it is watched by polling, and
stopped by signalling its group directly. Its earlier output is gone, and the
log says so rather than presenting emptiness as silence.
- Every `Register`, not only the first, reports what the agent is running. A
reconnect may be a control plane restart and has to recover the same way.
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped: nothing will
route to it and it is holding cards.
- An engine the agent cannot identify — a corrupt state file — is killed. A
process holding GPUs that nothing can describe is worse than no process,
because placement will hand those GPUs out again.
One mechanism covers agent restarts and control plane restarts.
### A test that was passing for no reason
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
tracking hijacked connections. The agent never reconnected and the assertion —
on the connection map, which still held a stale entry — passed anyway. Tests now
sever connections through their own listener and count opened streams.
### Multi-node launch
Placement and environment generation had landed early, which made multi-node
look further along than it was: a multi-node replica could not come up at all.
Three things were missing, and all three came down to two assumptions that do
not hold once a replica spans nodes.
**Not every rank serves HTTP.** The agent health-checked every shard at its own
port, but only the leader answers. Workers timed out, reported Failed, and took
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
worker is judged by staying alive instead. That is weaker evidence on purpose —
the replica's real readiness is the leader answering, which only happens once
every worker has joined.
**A shard is not always one process.** `Engine.Commands(spec)` returns a
sequence: every command but the last is a sidecar kept alive for the shard's
lifetime and torn down with it, and the last one's exit is the shard's exit.
vLLM under Ray is why — the leader runs a Ray head and then the server against
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
had been written and tested but never called, so `launch_mode: ray` would have
run `vllm serve` against a cluster that did not exist.
A command can also require the rendezvous point to be listening before the next
one starts, bounded by a timeout so a head that never appears fails the shard
rather than hanging it. Sidecars are killed with the shard: a Ray head left
behind holds the port the next replica needs.
**Start order was backwards.** Workers were started first, on the theory that
the head should wait for them. Both engines put the rendezvous point on rank 0,
so a worker started first has nothing to connect to and spends its startup
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
needs, so the leader can start immediately and waits on its own.
None of this is verified against real engines. It is tested with stub processes;
the Ray sequencing and which launch mode a given vLLM version wants can only be
confirmed on hardware.
### Still missing for phase 2
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
network.
### Phase 3: placement
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
their fabric attachments, and what is allocated where. A re-registering agent
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
without doing so either. A GPU with an ECC fault or another process on it is
not counted as free. Replicas are released whole, because they fail whole.
- **Placement** (`internal/control/placement`): picks the highest-priority
profile that fits and explains every candidate it rejected. Two constraints
are enforced structurally rather than checked afterwards — there is no code
path that can combine nodes from two fabrics, and only GPUs of the profile's
class are ever considered.
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
large models that cannot be split. Multi-node replicas get the opposite
treatment: they are the least movable thing in the cluster, so they take the
emptiest machines.
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
domain where the topology is known. Without the NVML build it falls back to
the lowest free indices, which is usually right but only by accident of how
vendors number slots.
- **Property-based invariant tests** (plan §13): 400 randomly generated
clusters, checking on every placement that a multi-node replica lives on one
fabric, that each shard receives only that fabric's adapters, and that no
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
The table-driven tests cover the cases someone thought of; this covers the
ones nobody did.
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
rather than at request time — a typo in a fallback should stop a deployment,
not surface during the outage when the fallback is finally needed. The
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
load instead of minutes into model loading.
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
loop. It places one replica per pass, because placement depends on what is
already allocated and starting several from one snapshot would hand them the
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
vLLM under Ray needs its workers attached before the head serves.
- **Replica failure is whole-replica failure.** One dead shard makes the others
useless — they wait on collectives that will never complete while holding
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
exponential backoff carried on the model rather than the replica it just
destroyed.
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
blip must not tear down replicas that are still serving, and rebuilding a
multi-node replica costs tens of minutes of model loading.
- **Scale-down drains before stopping**, so in-flight requests finish instead of
being cut off mid-stream.
## v0.3.0 — 2026-09-16
Flow control. A batch workload no longer starves interactive traffic.
### Added
- **Admission control** (`internal/router/admission`), inserted between quota
checks and scheduling. It answers *when* a request may proceed; the scheduler
still answers *where* it goes. Keeping them apart is the point: an engine
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
no amount of endpoint selection rescues the request behind it.
- **Saturation detector.** One number per pool, the worse of queue depth and KV
occupancy. Either signal alone misleads — a pool can have an empty queue and a
nearly full cache, or the reverse.
- **Priority bands.** Each band has a saturation ceiling; above it, that band
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
"flow up to the healthy limit"; above 1 means never wait.
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
one caller sending a thousand requests occupies one slot in the rotation
rather than a thousand.
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
(504), because sending the GPU work the client has already abandoned helps
nobody. Cancelled requests leave the queue instead of holding capacity.
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
`x-llmgw-queue-ms` when a request waited.
### Phase 1.5 exit criterion, measured in `test/e2e`
While 500 concurrent bronze requests run continuously, interactive p99 in the
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
With admission switched off the same load pushes it to **55–60x**. Both
directions are asserted, so the test fails if admission regresses *and* if the
load generator ever stops being a real load.
### Two lags, two different fixes
Getting this right took longer than the queueing itself:
- Engine metrics are scraped, so they lag by an interval. The router's inflight
count does not — but it includes *running* requests, and treating those as
queue depth makes a healthy engine look overloaded every interval. What is
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
the metric contract, so it is learned as the high-water mark of observed
`running`.
- Between admission granting a request and the router dispatching it, the
request is invisible to every signal. A simultaneous burst therefore passed
the check en masse — exactly the batch arrival pattern this layer exists to
survive. The controller now counts its own outstanding leases against
dispatched inflight.
### Known limitations
- The band comes from a trusted header until authentication arrives in v1.0.0.
Anyone who can reach the port can promote themselves to platinum.
- The tenant key is the session id, for the same reason.
- Displacement (dropping queued low-band work to make room) is not implemented;
the plan defers it, and bounded queues plus TTL cover most of the need.
## v0.2.0 — 2026-09-16
The router serves traffic. `llmgw server` is real: it fronts a static set of
engine replicas with an OpenAI-compatible API.
### Added
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog.
- **Retry and fallback.** Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it via `x-llmgw-served-model`.
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched.
- **Usage accounting that does not depend on the client.** The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way.
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time.
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
### Phase 1 exit criterion, measured in `test/e2e`
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
### Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing.
- The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
View Git Blame Copy Permalink