ilker/llmgw
2
0 Fork 1
Code Issues Pull Requests Actions Packages Projects Releases 95 Wiki Activity
Files
main
llmgw/CHANGELOG.md
T
Ilker ManapandClaude Opus 5.5 57f0fd2ed0
CI / test (push) Failing after 6m53s
Details
Slurm's jobs in containers: pyxis and enroot, installed by the agents 
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:31:20 +01:00

5428 lines
314 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### Added
- **An organization's admin.** Besides member and manager, a membership of
an organization may now be `admin` (`llmgw users join … --org Acme --role
admin`, the role chosen on a membership on the people page, the console,
the admin API). An admin sees the organization's page as a manager does and
also changes the organization from it: adds, renames and removes its teams
and projects, adds people with an approved account by their address and
takes them out, makes its members managers or admins, and revokes keys spent
in its projects. Limits, approving and suspending accounts, the
organization's name and the rest of the cluster stay with the cluster's
administrators; an admin cannot take away their own role. Every change is in
the audit log.
- **Slurm beside llmgw, for training and batch work.** `llmgw ctl slurm setup
--controller NODE --nodes NODE,...` (or `--all-nodes`, or `server init
--slurm`) has the agents install munge and Slurm from the machines' package
sources: slurmctld on one node, slurmd configless on the others, slurm.conf
and gres.conf written from what each node reports. Every node stays llmgw's,
drained in Slurm, until `ctl slurm lend NODE`: its replicas move off, then
Slurm runs jobs there. `ctl slurm reclaim NODE` takes it back once its jobs
end. `ctl slurm status` and the console's **Slurm** show each node. The
controller also keeps the jobs' records: slurmdbd, in a MariaDB on that
machine alone (Slurm takes MySQL or MariaDB only). Slurm's ports
(6817–6819, srun's 60001–63000) answer the cluster's networks only, as
llmgw's own do.
- **Slurm's shared files and users.** `slurm.shared` (`ctl slurm shared`) is
one path on every Slurm machine: the site's NFS export mounted, an llmgw
node exporting it over NFS (what `server init --slurm` chooses, `/shared`
from the first machine), or a file system the site mounts. `slurm.users:
llmgw` (`ctl slurm users on`) makes each active account a Linux user on
every Slurm machine, the same login and UID (from 200000, never reused)
everywhere, its home on the shared path; people add SSH keys in the
portal's Profile (`ctl slurm users keys` for an operator). They log in to
the controller; to another Slurm machine only while a job of theirs runs
there, their session put in the job (pam_slurm_adopt), so nobody uses a
node's cards outside Slurm, nor on a node llmgw serves from. Root and the
machines' own users are not held to it.
- **Slurm's jobs in containers.** `llmgw ctl slurm containers on`
(`slurm.containers`) installs enroot and pyxis on every Slurm machine, at
pinned versions checked against their digests, and nvidia-container-cli on
GPU machines; Slurm loads pyxis once every machine has it, and then `srun
--container-image=...` runs a job in that image, as its user, with the
shared path inside.
### Changed
- **Forms only for those who may send them.** A viewer sees the people,
limits, mirror and cluster pages without their forms, an operator the
mirror's and a node's engines but not the rest; before, they were drawn and
answered "not found".
- **Smaller forms.** A label sits close to its field, a field is as wide as
what goes in it rather than the page, buttons are smaller, and the short
forms — a new key, a password, connecting an editor, a capacity or a limit —
sit on one line or two instead of a screen.
### Fixed
- **The serve forms start at the longest context that fits.** The mirror
page's form began every GGUF model at 4096 tokens, and the console's at
4096 or the plan's figure, whatever the model was trained for. Both now
start at what it was trained for, as long as its KV cache fits where the
form places it beside the weights, else at the most that does, in whole
thousands; the mirror page's llama.cpp slots are as many of that, up to
four, as the room left holds, and its memory figure is for that pool. A
plan says what the model was trained for (`trained`). Strata stays at
its 32768; vLLM's form is unchanged.
- **A model that failed to start was started in the same place again.**
Placement given the same inputs chooses the same node and card, so an
on-demand model whose replica failed to load was retried there, past each
backoff, until it was parked, with another card or node free all along.
The control plane now remembers where a model's replicas failed (node and
cards, up to eight places) and places the next one elsewhere when anywhere
else fits; with nowhere else it is tried there as before. Coming up, a
change to its definition or `ctl retry` forgets the places. The log says
`placing a model away from where it failed to start`, with both places.
- **Deleted model files came back after a control plane restart.** An agent
registers with the files its startup scan found, and sends the list again
only when it changes; a file deleted after the agent started was listed
again by a control plane that restarted, until the next change. A new
stream now gets the list on its first heartbeat.
- **`ctl weights` offered models in use for deletion.** A model of several
files (a set, its address ending in `/`) is fetched into a directory named
from the address without the slash; the sweep named it from the address
with it, so every set in the catalog looked like nobody's. On the PoC that
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
`--collect`, and `ctl model remove --purge` of a set, would have deleted
them. Both now compute the name the fetch does.
- **A sign-in took an organization's manager role away.** A group from single
sign-on, SCIM or Active Directory that puts someone in an organization wrote
the membership again at every sign-in or sync, as a plain member: a manager
made one on the people page was one only until they next signed in. A
membership the directory names is now added if missing and otherwise left
as it is.
- **The limits form on the people page's panel** was laid out on one line, its
title squeezed into a column.
- **Deleting the model files nothing uses said what it was asked to free,
not what it freed.** The console's "Delete model files nothing uses" and
`ctl weights --collect` added up every file the nodes were asked to
delete as soon as the request was sent, and a model of several files —
fetched into a directory of its own — was refused by the node as "not
empty", so a sweep said tens of gigabytes freed and nothing went. The
node now deletes such a directory whole (a link only as a link), and the
sweep waits for the nodes to list their files again: freed is what they
no longer list, and what they kept is said beside, with where to read
why. Removing a model with its files reports the same way.
## v0.16.3 — 2026-11-03
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
the threads it is given rather than the whole machine's memory.
**Upgrading.**
- No migrations. `llmgw ctl upgrade v0.16.3`: the control planes carry the replicas per profile, the model card and
the speed in the serve form and in `ctl model plan`; the agents carry the measurement of one core's memory rate,
without which the speed is reckoned the old way, from the whole machine's.
### Added
- **Replicas counted per profile: seven on cards and four on CPUs, under one
name.** `llmgw ctl model replicas <model> gpu=7 cpu=4`, the console's
Replicas per profile… and the mirror page's model card set how many of a
model's replicas run on each of its profiles, rather than every replica
on the best profile that fits. For a GGUF model with no profile on CPUs
one is added (`--cpu-memory`), preferred below the cards, so the router
fills the cards first and sends to the CPUs once they are full. A change
starts the replicas short before stopping those over: the model never
runs fewer. `ctl models` shows the counts; a scaling of another total
ends them.
- **Every setting of a served model from the mirror page.** A served
model's row has a Settings link, and its card now changes the context
and longest answer, the cards a replica takes, the replicas on each
profile (with the CPU's speed beside), stops and starts it, restarts its
replicas, and takes it out of the catalog with its files or without —
beside the replicas, class, owner, memory and adapters it held before.
Stopped on the row is now the model stopped and kept, started again as
it was, rather than its scaling set to nothing.
- **How fast a model answers on the CPU, before it is served.** The serve
form says, as the cores are changed, about how many tokens a second one
reply gets on the CPU and on one card of the fleet's fastest class, and
warns when it is slow (under 15) or very slow (under 5, a reply of 500
tokens taking minutes). Very slow on the CPU is refused until "run on the
CPU anyway" is ticked, GPUs free or not. `llmgw ctl model plan` and the
console's Hugging Face screen show the same figure for every way a model
can run, and `ctl model serve --cpu` says it in its log.
- **The speed follows the threads a replica is given.** Each agent now
measures what one core reads beside what all of them read: a replica of
four threads on a large server reads at about four cores' rate, not the
server's, which the estimate used to assume and overstated several times.
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
and a model of experts reads its experts' share, from the GGUF header.
Agents need upgrading to report one core's rate; until then the old
estimate is used.
## v0.16.2 — 2026-11-02
Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
engine for models larger than the card; a served model's longest answer changed in place; and fixes found putting
the router under load and fetching large models: a multi-file model fetched once per node however many replicas
start there, downloads that carry on where they were cut off, and a router that does twice the work on two cores.
**Upgrading.**
- No migrations. `llmgw ctl upgrade v0.16.2`: the control planes carry active-active routing (off until
`llmgw ctl config set router.active_active true`) and the router fixes; the agents carry the set fetched once
per node and the set's progress and checking; the mirror carries the resumed downloads.
### Added
- **Standbys that answer the client API themselves.** With
`router.active_active: true` in the cluster's settings, a standby control
plane routes `/v1` to the leader's replicas instead of passing every
request to the leader, so more machines carry the traffic. It hears the
replicas from the leader every two seconds over the agent port (no engine
keys travel) and reports back what it used, so the leader does not evict
a replica busy behind a standby, and the on-demand models it was asked
for. Rate limits are the cluster's: each router's request and token
counts reach the others through the leader every two seconds, so a key
spread over several control planes gets its limit, not that many times
it. Budgets are brought up to the store's every 30 seconds. Off by
default; the portal, `/admin` and `/mcp` still go to the leader.
- **A served model's longest answer, changed in place.** `llmgw ctl model
context <model> --max-output N` (and `max_output` on `POST
/api/v1/models/{name}/context`, and the console's Context length… form)
sets the longest answer on its own: the catalog's limit, in force at once,
with no replica started again. With a context it is set with it. Strata's
context can now be changed too (its setup's `--context`), so neither
needs the model removed and added again.
- **FreeToken, an engine for models larger than the card.** Mixture-of-experts
models several times a card's size, from one card, with the experts in RAM
and streamed over PCIe — read from Hugging Face safetensors checkpoints as
published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
MiniMax), with batching. Installed like vLLM on every node it runs on: a
card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with; older cards are passed over (tried on a
Turing card, where 0.2.1 has no kernels). Profiles are one card
(`engine: freetoken`); the key goes in its environment, and flags that read
files or fetch are refused. Readiness now also waits for an engine whose
`/health` answers 200 while saying `loading` in its body. FreeToken is
followed release by release: a weekly workflow fails when PyPI has a version
newer than `DefaultEngines` installs.
- **How much of each GPU works, beside whether anything does.** The busy
figure is nvidia-smi's `utilization.gpu` — the share of time any kernel
ran, which a kernel on one SM fills — and it is now said to be that: on
the dashboard, the node page, the metric's help and Grafana. Beside it,
each card's draw against its power limit (the efficiency table, the fleet
and node charts, the console's GPU work chart), and where the card's
profiling counters can be read, its SMs, SM occupancy, tensor cores and
memory interface: `llmgw_gpu_sm_active_percent`,
`llmgw_gpu_sm_occupancy_percent`, `llmgw_gpu_tensor_active_percent`,
`llmgw_gpu_dram_active_percent`. The agent reads them from NVML's GPM on
Hopper and later, and from a dcgm-exporter on the node elsewhere
(`discovery.dcgm_exporter`, default `http://127.0.0.1:9400/metrics`, `off`
for none). The GPU map colours by this "work" by default — SMs active
where read, power against the limit otherwise — and keeps busy as a mode.
On cuda01 decode read 92% busy with the SMs at work 69% of the time.
### Fixed
- **Several replicas of a multi-file model on one node fetched it once
each, over one another.** Four replicas placed on a node's four cards
wanted the same safetensors set at the same moment, each wrote the same
`.part` files, and all but one failed to start ("rename … no such file")
until a retry found the set finished. A set is now fetched once per node
and every replica waiting on it gets it, as single files always were.
- **A model of several files showed one file's download as the whole.** A node fetching a set reported the part it
was on — "59% of 46.1 GB" for a 188 GB model — instead of the set: it now reports the set's bytes against its
whole size.
- **A set's files checked with nothing said, one at a time.** After the last byte arrived a node checked each file
against the mirror's checksum, 195 GB on one core for eight minutes, while the replica still read "fetching 100%".
It now checks four at once and says so: `checking 40% of 195 GB` in `ctl replicas`, "Checking the weights" in the
chat page; and a check that takes long is not taken for a stuck download.
- **A mirror download the registry cut off started again from nothing, or not at all.** Hugging Face's CDN ended a
60 GB transfer at 23 GB (`stream error … CANCEL`), the mirror deleted what had arrived, and `ctl model serve`
failed. The mirror now tries again up to six times, carrying on from where it got to (`Range`), keeps what arrived
when it gives up, and carries on from it after a restart too.
- **A model from an outside provider took three or four requests at once.** Admission read every request in flight to
an endpoint it does not measure as a queue, so an external model (OpenAI, another llmgw cluster) was held to what a
saturation of 0.85 allows of "in flight / 4"; the rest waited and timed out with 504. An outside provider queues for
itself and says 429 when full, which its breaker already hears: it no longer counts towards saturation.
- **A valid key told to slow down under load, every thirty seconds.** When a busy key's cached entry expired, every
request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
at once now share one read and one allowance.
- **A streamed answer was one write to the client a token.** Each event was flushed alone, a system call each: a
third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
and the first token at 128 at once from 91 to 45 ms.
## v0.16.1 — 2026-10-30
The mirror page's Strata tab, Strata's ready-made packs served with Strata fetching them, and two fixes found
serving Strata's models: repositories that keep each size in a directory, and DeepSeek on cards its engine does not
run on.
**Upgrading.**
- No migrations. The control planes carry the tab, the plan and the serving; the agents carry the refusal of a
Strata profile its card cannot run (`llmgw ctl upgrade v0.16.1`). Mirrors need nothing new.
### Added
- **The mirror page's Strata tab.** Every family and size Strata runs, with its download, the RAM it keeps and where
it would run here; a GGUF size fetched into the mirror in one press (checked against Hugging Face's checksums),
and the packs Strata fetches itself — NVFP4, DeepSeek V4.1 — served at once, their repository the source.
`ctl model plan` and `ctl model serve` take those repositories too, and the explorer names a `strata-ple`
repository for what it is: part of a pack Strata knows, or one Strata 0.2.7 does not.
### Fixed
- **A repository that keeps each quantisation in a directory of its own** (ISTA-DASLab's `IQ2_XS/`, `IQ3_S/`) listed
no weights: its file tree is now read recursively, by the explorer, the plan and `ctl model serve`.
- **DeepSeek V4.1 offered on a card its engine was not built for.** Strata's DeepSeek engine runs on RTX 30 (compute
8.6) and newer; on a Turing card the tab and the plan offered it, and Strata's setup refused it once placed. Now
neither offers it there, the tab says why, and the agent refuses such a profile before starting it.
## v0.16.0 — 2026-10-30
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
are, and do not fail a node that came back on the version.
**Upgrading.**
- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
(`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
fetches as before.
- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
they are.
- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
### Added
- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
RAM a size keeps its experts in.
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
of a vector down and up again, some thirty times more. `llmgw-kb` does so with a gateway that says it can
(`server_embeds` in a collection's answer); `--embed-here` keeps the old way.
- **Weights pinned to a commit, and checked against what Hugging Face published.** A model added from Hugging Face
(`hf:`, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at
then — the file's URL at that commit, or `--revision` — and each file's published sha256 goes in its origin. A node
refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for what `ctl model
serve` fetches, and `llmgw mirror add` for a Hugging Face URL (`--sha256` for others). Nodes need this version to
check; older ones fetch as before.
### Fixed
- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
- The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
- The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
control plane waits longer than the download may take; a second request while one runs is refused.
- A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
- The console's node upgrade waits as long as the control plane, and offers no development version.
- systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
of one. The path now reaches the leader as it was sent.
## v0.15.3 — 2026-10-29
A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
**Upgrading.**
- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
- The mirrors carry the fixes: on each mirror's machine, `sudo llmgw upgrade v0.15.3 --service llmgw-mirror`.
A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
### Added
- **A model stopped and kept, and a model removed with its files.** `llmgw ctl model stop|start MODEL`: none of its
replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
administrator stopped it. `llmgw ctl model remove MODEL --purge` (`DELETE /api/v1/models/{name}?purge=true`) also
deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
stopped; what another model uses stays. Both are in the console's model menu.
### Fixed
- **`llmgw mirror add-peer` on a mirror with a machine of its own.** It set the second mirror up and then
failed for want of the control plane's server.yaml; now it prints the `llmgw ctl config set mirror.peers`
to run on a control plane.
- **Models fetched before checksums were kept reach the second mirror.** A mirror kept in step copies only what it
can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
- **A file removed with `llmgw mirror remove` is gone from the mirror's answers too.** The running mirror kept
reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
## v0.15.2 — 2026-10-29
A served model moved to more cards, or fewer, while it is served.
**Upgrading.**
- No migrations, and nothing changes until the command is used. The control
planes carry it; the nodes need nothing new.
### Added
- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
GPUs… in the console's model menu. How many cards of one node each replica
uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
the model's KV heads, and the layers split for llama.cpp. Refused for a
count no node has, or whose cards would not hold the model and its
context, with the figures; its replicas are then started again on it, one
by one, beside the old — or, with `--in-place` when there is no room
beside them, each stopped first.
## v0.15.1 — 2026-10-29
A 125B model on one card, a second mirror, and weights that run code
refused. Strata runs as an engine: a mixture-of-experts model with its
experts in RAM, served from one 24 GB card. A second model mirror keeps in
step with the first, so an HA install has no single machine left. Pickled
weights are refused unless a model says otherwise. A served model's context
can be changed in place. The explorer says how fast a model would answer
here before it is fetched, and a card in the wrong PCIe slot is said so.
**Upgrading.**
- No migrations. Upgrade the control planes before the nodes, for the
pickle refusal below.
- A model whose weights are in a pickle format (`.bin`, `.pt`, `.pth`,
`.ckpt`, `.pkl`) stops being placed: its node refuses the source. Give it
its safetensors or GGUF, or add it again with `allow_pickle: true`
(`llmgw ctl model add -f`) if its files were checked. Look at the
models' sources before upgrading the nodes: one ending in
`.ckpt` or `.bin`, or a mirror version holding `pytorch_model.bin`. Upgrade the control planes first: a node on
this version refuses those weights until its control plane can say
`allow_pickle`.
- Strata is opt-in: nothing installs it until `engines` or an
`engine_overrides` entry names a version (`0.2.7`).
### Added
- **Strata as an engine: a 125B model of experts on one card.** Strata
(github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones
used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves
Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client;
more clients share about that, its slots buying fairness rather than
throughput on one card. It runs a few families only — Qwen3.8-Flash-Next
and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a
profile's args with `--family` and `--size`
(`--family qwen --size IQ2_XS --context 32768`). The source is the
directory of the model's GGUF files, or the repository, which Strata then
downloads itself, pinned. The agent installs it on an NVIDIA node of
x86-64: its source pinned by digest, in a Python environment, and its
release's engine checked against the digest GitHub publishes — the CUDA 13
build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal
to Ada, a V100 included, on driver 575. A replica prepares its model the
first time (minutes; its draft layer is a 5 GB download) and loads in
about a minute and a half after. One card of one node per replica, for
now; not in an offline bundle, its packages coming from PyPI.
- **A served model's context, changed.** `llmgw ctl model context MODEL
TOKENS [--slots N]`, `POST /api/v1/models/{name}/context`, and Context
length… in the console's model menu. The catalog, the engine's arguments and
the memory change together, checked against what the model was trained for
and what its cards hold; its replicas are then started again one by one. The
models API shows `context_length` and `max_output`.
- **A second model mirror, kept in step.** In a three-machine install the setup screens put one on a database machine
(`--second-mirror 1|2`). An install with one mirror gets one with `sudo llmgw mirror add-peer user@host`.
- **In step:** each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
removes what the other removed, even if it was down at the time (`llmgw mirror serve --peer`).
- **Control planes:** they name both (`mirror.peers`) and use whichever answers.
- **Nodes:** while one mirror is down, a node is given a model's file on the other.
This takes away the last single point of an HA install: with the first machine off, models are still added and
fetched.
### Fixed
- **An engine that computes in a process of its own group was not its
replica's.** A GPU replica read as running on the CPU, and the memory it
held on its card as somebody else's, which placement then kept free of
other work. A process the replica started, at any remove, is now the
replica's. And what a replica computes on, once sent, was never sent
again: a wrong reading kept from before an agent's restart stayed wrong.
It is sent again when it changes, and read again when a GPU replica was
kept as the CPU.
- **VS Code's sign-in lost its place through the sign-in page**, so the
prompt to confirm came every time; a link to the same gateway by plain
http could take an https setting down to http; and a new build of the
extension was not installed over an old one (the command now says
`--force`).
- **The install guide's recordings were not checked for secrets**, only
the page; every frame is now.
- **MCP server fields could not be emptied**, an address could carry a
password, the cluster's own server could not be changed from the
console, and a grant to a key that does not exist was said done.
- **A guardrail service two policies named was asked as the first said**,
not the stricter; a `null` answer read as safe; and with `restore` the
request log kept the answer with the redacted values back in it.
- **What an engine downloads by a Hugging Face id was not held to the
pickle refusal** — an adapter by its id, a TensorRT-LLM model's; a model
with either now needs `allow_pickle`.
- **`server init` run again on a machine already set up was refused its
memory**, as on an 8 GB install fixing its certificate with `--host`, and
judges that would not be installed (no mirror) were counted.
- **Removing an MCP server left its sealed credential in the store**, and
replacing a credential under another header kept the old one sent. A
server's secrets now go when it no longer reads them.
- **A cloned repository could point the VS Code extension at another
gateway.** `llmgw.gatewayUrl` is a machine setting now, never taken from a
workspace.
- **A node's memory bandwidth could be measured in its cache** on a busy
node, and was 0 after a reconnect until the next heartbeat. The buffer is
at least eight times the processor's last cache.
- **A project kept in the cluster could have its questions embedded outside
it.** The similar-question cache now asks the embedding model as the
question's project would, external fallbacks refused, and skips a model
that is itself outside. Its comparing no longer holds the cache's lock,
which every project's lookups waited on.
- **Restored values were left as placeholders in streamed tool calls**, in
completions and in reasoning, and a value with a quote in a whole answer's
tool-call arguments broke their JSON.
- **An outside guardrail service read what `redact` had taken out**, and a
streamed answer was sent to it twice. The docs now say a service stops a
streamed answer only under `stream: buffer_all`.
- **A replica found slow once could stay unused.** Its time to first token
is forgotten after five minutes without an answer, so it is tried again.
- **Weights that run code got past the refusal in four ways.** A mirror
version already on a node was used without being looked at again; a
`?query` or `#fragment` on an `hf:` source hid the file's name; `.pt2` was
not on the list; and a profile could write `--load_format`,
`--trust_remote_code` or an abbreviation such as `--trust-remote`, which
vLLM and argparse read as the flag, or give SGLang `--load-format`. All
refused now. TripoSR's preset says `allow_pickle`, without which it could
not be added, and a shared ComfyUI environment takes its first member's.
- **A temperature written as a string was refused.** Since the GenAI traces
read it, `"temperature": "0.7"` got a 400 before reaching the engine; it
is read loosely again, for the trace only.
- **A reused answer could carry another caller's personal data.** With
`restore: true` and the answer cache on, two callers' requests that
redacted to the same text shared one answer — with the first caller's
address put back in it. Such requests are now never answered from the
cache nor kept in it.
- **Granting a server's tools could take others away; revoking could close
a level to every tool.** A level with no tool list sees what the levels
above grant; giving it its first list made it see only that list. And a
level left with an empty list was closed to every tool, which the screens
did not show. Now a first list starts from the nearest list above, taking
a server away leaves the rest or goes back to no list, the screens say
when a level above keeps granted tools out of reach and which levels are
closed, and the console reads and writes the level at once rather than
from what it loaded.
- **Similar questions about different pictures got each other's answer.**
A question with a picture, a sound or a file in it is reused only when it
is the same request.
### Added
- **A card in the wrong PCIe slot is said so.** Each agent reads how every
card is linked — the fastest generation card and slot allow together, and
the lanes in use — against the card's own most, from the kernel (the
driver's figure is what card and slot allow together); a card on fewer lanes or
an older generation is a node warning, with what it carries and what it
could, on the dashboard, the console and in `llmgw doctor`.
- **And on a web page, `/admin/mcp`.** For administrators: the same list,
adding a server with its credential sealed, testing it and listing its
tools, granting them in the tree beside that level's other grants and
limits, revoking, changing, replacing the credential, turning it off,
removing; and the line that points an agent at this gateway's `/mcp`.
In English and Turkish; every change in the audit log, the secret not.
- **The company's MCP servers in the console.** A screen of its own: each
server with its state, who has its tools, its rate, the names of the
credentials it is sent and its address; add one with a bearer token or
a header kept sealed, test it and list its tools, grant them to a team,
project, organization, person or key beside that level's other grants,
revoke, change, replace the credential, turn off, remove.
- **And on the cards, as this cluster would lay it out.** The explorer's
**GPU tok/s** and a repository's files give one reply's speed for each
layout the planner offers here — one card, tensor parallel across a
node's cards, llama.cpp's layer split, a pipeline across nodes on a
fabric — the fastest of them, with the layout in the tooltip. From each
card's memory bandwidth by its model name, NVLink or PCIe between a
node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed
classes is judged class by class. A model of experts reads 1.6 times its
active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144
tok/s; qwen3-vl-8b Q8: 62).
- **How fast a model would answer on a CPU, before it is fetched.** Each
agent measures how fast its node's memory reads when it starts (`Host`
gets `mem_bandwidth_gbs`, the node's view `mem_bandwidth_gbs`); the
explorer's **CPU tok/s** column and a repository's GGUF files say about
how many tokens a second one reply gets on the fastest node that holds
the model — the bandwidth over the bytes a token reads, its active part
for a model of experts. Calibrated on the PoC against llama.cpp's own
timings: within 15% from 1.5B to 14B.
- **The explorer lists what this cluster can run.** Judged by the nodes
as they are now — every card's memory and compute capability, every
node's RAM — through the same planner as serving a model, rather than a
size picked from a list. A model too big for any node, in FP8 without a
card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and
counted; the page says what it judged by. With cards, safetensors repositories are
listed beside GGUF. On a repository's page a file too big for the
cluster is shown with what it needs and no Fetch button.
- **Weights that run code are refused.** Python pickles — `.bin`, `.pt`,
`.pth`, `.ckpt` — run whatever they hold when loaded, which is how the
malicious models on Hugging Face work. The node refuses such a source,
or a mirror version with one in it, before fetching it; adding such a
model is refused; vLLM and SGLang downloading a model by its id read its
safetensors only; the mirror page marks the file. `allow_pickle: true`
on a model lets it through on purpose. The mirror's explorer no longer
lists a repository with nothing a node loads (no GGUF, no safetensors)
while "runs here" is ticked, and a repository's pickle files are not
offered, only counted. BACKLOG M lists what is next:
checksums against Hugging Face everywhere, its own scan, trusted
publishers, engines without the internet.
## v0.15.0 — 2026-10-25
The company's tools and editors through the gateway, and guardrails that
can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
extension the gateway serves. A guardrail policy can ask an outside
service, and can put redacted values back in the answer. Chat goes to the
replica answering soonest. A similar question can get the answer already
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
GenAI conventions, and Kong is written down from trying it. Also an install
guide with every screen, at `/install`.
**Upgrading.**
- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
(`projects.cache_similar`), applied when the control plane starts.
- `llmgw server init` now refuses a machine that would serve models without
the memory for them: the starter model and the judges need 12 GB on one
machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
`--no-node` make it fit. A running cluster is not checked again.
- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
`llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
- Nothing else changes until it is turned on: `services` and `restore` in a
guardrail policy, `keys cache --similar` with
`router.cache.embedding_model`, `cached_per_1k` in a price.
### Added
- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
gives a chat question the answer already given to one like it in the same
conversation, compared by `router.cache.embedding_model`, as
`x-llmgw-cache: similar` with the similarity. Held in memory beside the
exact answers; nothing written to the store but the setting.
- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
charges the prompt tokens an engine or provider read from its cache (the
answer's `cached_tokens`) at their own rate; without it they cost what any
prompt token does, as before.
- **Chat goes to the replica answering soonest.** The router keeps each
replica's time to first token, smoothed over its last answers, and the
`chat` profile scores by it beside the queue and the KV cache: of two
replicas of one model, the one on the slower card or the busier host gets
less, and a replica not yet measured is tried.
- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
no completion tokens of a streamed answer, so bill from llmgw's usage.
- **A guardrail that is someone else's service.** A policy's `services`
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
HTTP: a body template, keys from files, dotted paths to where the answer
says unsafe and why, on input, output or both, block or flag, and the
policy's `fail` when it does not answer.
- **Redacted values put back in the answer.** `restore: true` on a guardrail
policy's input: the model reads `[EMAIL_1]`, the client reads its own
address in the reply, streamed or not. Only the request's own values, held
in memory for its life; the output rules read the answer as the client
will.
- **Traces in the OpenTelemetry GenAI conventions.** An inference request
and each attempt at an engine carry `gen_ai.operation.name`, the requested
and the answering model, `max_tokens`, `temperature` and `top_p` when set,
the session as `gen_ai.conversation.id`, and the input, output and cached
tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
Tools that read the conventions show llmgw's calls as model calls with no
dashboard of their own. No prompt or answer text is ever in a span.
- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
the binary itself: plain JavaScript, no dependencies, no build, and so no
marketplace needed at a site with no internet. It offers the gateway's chat
models to VS Code's language-model API — the chat's model picker, agent
mode, and every extension that asks VS Code for a model — with the sizes
and the tool and vision capabilities the catalog says, streamed, with tool
calls and their results both ways. **VS Code** is a tool on the account
page's **Connect your editor**: a key, the one-line install, and an **Open
in VS Code** link that hands the key to the extension, which keeps it in
VS Code's secret storage. **llmgw: Sign In** opens that page with a state
of its own; a link without it is confirmed first, the gateway named.
- **An install guide with every screen, at `/install`.** From three bare
Ubuntu machines to a first answer: what each machine needs and how to
check it, the release fetched and checked, the setup screens one by one,
the system's state, and a first request with a key from the account page.
The screens are a real three-machine install, recorded and kept as
terminal frames — sharp at any size and a few kilobytes — with each
session played back on request; the administrator's password and every
key were masked before it was kept. `/about` says more of what llmgw does
(documents, guardrails, sign-in, high availability, pictures and 3D), and
its install steps are the ones of today, leading to the guide.
- **The company's MCP tools under the guardrails, a rate and a breaker.**
- **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
ticket is withheld as a pasted one would be, and personal data is redacted.
- **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
- **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
- **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
admin API has `/api/v1/mcp/servers`.
- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
as `llmgw.*`, its tools say what `llmgw ctl` does:
- `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
- nothing that changes the cluster.
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
`Authorization: Bearer`.
- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
`--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
`llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
so no other model is placed on the judges' machine.
### Fixed
- **A machine too small for the models it would serve is said so first.**
The control plane that is also a node, serving the starter model and two
judges, needs about 10.5 GB. On 8 GB the install finished every step and
then had no room for the starter. Now the setup's checks fail its memory
with what it would serve, and `server init` refuses before it installs
anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
- **Turkish pages are Turkish to the end.**
- **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
catalog", the settings changed meanwhile, and others) are now in the page's language.
- **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
- **Footer:** the time at the foot of a page is in the language's date form.
- **Stale forms:** they are refused with a sentence in that language rather than "bad request".
- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
`--replace` writes every limit, as the console's limits form does.
- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
offered, as is one that divides them.
- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
The timeout now runs from when a node's download last moved.
- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
## v0.14.9 — 2026-10-19
Sign-in set up from the console, with Active Directory tried before it is
kept. A node's console now shows what the machine is doing, and a node's web
page shows what it runs. Also fixes found trying Active Directory against a
Samba domain controller on the test cluster. Sign-in through it, nested
groups and a disabled account suspended by the sync all worked there.
**Upgrading.**
- No migration.
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
block written into `server.yaml` on a cluster with a store was never
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
again from the console's Sign-in screen.
- A Samba DC's own autogenerated certificate is refused (a negative serial
and no SAN). Give it one from a CA.
### Added
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
GPUs.
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
### Fixed
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
domain controller needs.
## v0.14.8 — 2026-10-18
Sign-in with an on-premises Active Directory, and Entra ID made safe for
people in many groups. Also several keys for one outside provider, and a
measure of what the gateway adds to a request.
**Upgrading.**
- Migration 0048 adds the `user_directory` table, applied when the control
plane starts.
- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
Directory", has an example. It has been tested against a fake directory,
not yet against a real domain controller.
- A judge installed by an earlier `server init` is refitted by the leader
by itself (below); the steps given for v0.14.7 are no longer needed.
### Added
- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
more than half the accounts is refused. See docs/install.md, "Active Directory".
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
### Fixed
- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
registration. docs/install.md has an Entra ID section.
- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
and the log says so once.
- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
another no longer wait out the five-minute grace each.
## v0.14.7 — 2026-10-16
The judge installed by `server init` sized for what it does. On a machine
of fewer than eight cores v0.14.6 left every judgment past its budget, so
`fail: closed` refused every request. A single-machine install of v0.14.7
on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
**Upgrading.**
- No migration.
- **Upgrading does not change a judge already in the catalog.** v0.14.6
and earlier installed two judges, reserving no cores or one, and 2 or
4 GB. To give them what they need:
`llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
`llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
on a single machine; below that, a longer `latency_budget` or
`fail: open`.
- On a node whose cores are all reserved, `ctl restart --model` cannot
start the new replica beside the old one and waits. Stop the old engine
process, and the replica comes back with the new figures.
### Fixed
- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
stays there, and reserves 2.
- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
`latency_budget`, where requests must not wait.
## v0.14.6 — 2026-10-16
Setup that says what each part of an install is and lets every one be
chosen and set; three machines that each run the gateway, so any one can
be lost; a cluster that loses its store's primary without losing a
request; one ComfyUI for an environment's models; pictures, 3D and
workflows as MCP tools; alerts asked for at setup; extended thinking.
**Upgrading.**
- No migration.
- **Control planes**: with `control.lease_ttl` unset the lease is now 10 s
(was 15). Upgrade the standbys first and the leader last; a mix of
versions meanwhile is fine.
- **The store** (PostgreSQL): each member is dialled for 3 s before the
next is tried. A store so far away that connecting takes longer needs
`connect_timeout=10` in its address.
- **Nodes** notice a control plane that died without closing its
connection in 8 s (was 15) once they run this release.
- **Judges installed by an earlier `server init`** keep reserving no cores
and 2 GB. To give them their own:
`llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2`, then
`llmgw ctl restart --model qwen3guard-0.6b`.
- **Patroni** replaces a lost primary within 20 s on a new cluster. On an
existing one, run `server init --patroni ...` again from the machine with
the authority; it changes only that.
- **A floating address** keeps VRRP router id 51 until `server add --vip`
runs again, which moves it to one of its own; run it for every standby.
- A request whose judge is not there yet waits up to the policy's
`latency_budget` for one before `fail: closed` refuses it.
### Added
- **Setup lets every part be chosen, each with its settings, and says what each is.** `sudo llmgw` on a new machine lists
the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model,
pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline
bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens
its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or
any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's
categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the
default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is
left out, with the same install as a command. Every setting is a `server init` flag too (`--starter`,
`--judge-fail`, `--media`, `--max-key-ttl`, ...).
- **Three machines are high availability for the gateway, not only the store.** With three machines, setup makes the two
database machines control planes too, standing by while the first leads, with an optional floating address
(keepalived) that follows whichever leads. Any one machine can now be lost. By command: `server init --patroni ...
--standbys --vip 10.0.0.160/24`. The mirror's management API answers the three control planes, so a standby that
comes to lead can still fetch models.
- **Pictures uploaded to work from**: `/v1/files` takes `purpose=vision` or
`user_data` besides `batch`, kept as long as job results, so a picture is
sent once and named by its `file-…` id in the MCP tools and a workflow's
inputs.
- **One ComfyUI for an environment's models**: ComfyUI models in one
`comfyui.environment` (two or more) share one replica — the catalog
makes it a hidden entry, `comfyui-env-<name>`, on the first member's
profiles with the most memory any member needs, carrying every member's
weights and custom nodes. Asking for any member starts it; it answers for
each with that member's workflow and checkpoint, and the members get no
replicas of their own. A picture model and a 3D model on one card stop
handing it to each other with a cold start each way.
- **Pictures, 3D models, vision and workflows as MCP tools** at `/mcp`,
beside the memory notes, each offered only where the gateway has a model
for it: `generate_image` (with `concept`, `side` and `texture` styles),
`generate_mesh` (from a picture or a prompt), `review_images` (a vision
model, to a JSON schema), `run_workflow`, `job_status` and
`gateway_status`. Each makes its request under the caller's key, so
allowlists, limits, licences and budgets apply and it is charged like
any request; what it makes is kept as the caller's file and answered by
id and download URL. A client that takes a stream gets
`notifications/progress` while a mesh is made and waits up to ten
minutes; one that does not, 50 seconds, then the job id.
- **Setup asks where alerts go**: `llmgw server init --webhook URL`
(repeatable) and a **Notification webhooks** field on both setup screens.
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
any other the event as JSON (`slack:` or `json:` in front says which).
They are added to the cluster's `notify.webhooks` beside those already
there, and each is sent a test once the control plane runs on them.
- **Extended thinking on `/v1/messages`**: `thinking: {type: "enabled"}`
tells a model that thinks by its chat template to (`enable_thinking`),
and its reasoning comes back as `thinking` blocks before the text,
streamed as `thinking_delta` events; `disabled` tells it not to. Not
asked for, the reasoning is left out as before.
- **How long on-demand models hold a card** is a setting, in force on the
leader within seconds: `on_demand.evict_after_idle` (2m unless set) and
`on_demand.min_lifetime` (5m). On one card shared by a picture model and
a 3D model, 1m each brings the wait between a picture and its mesh to
about a minute.
### Fixed
- **The judges and the starter model divide a machine's CPUs instead of each taking all of them.** `server init` gave the
judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter
on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a
node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge
replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A
judge also reserves 4 GB, not 2: it uses about 3.2.
- **A lost store machine stops costing requests most of a minute.** The store's connections try each member for 3 seconds,
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
VMs with the primary's power cut: about 45 seconds of hanging requests before.
- **Leadership stays put while the store's primary is replaced.** Replacing a lost primary takes longer than the
leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
8 seconds (was 15), so it is on its way to the new leader as that one takes over.
- **A judge replica that hangs no longer costs the request.** All the tries of a judge request shared the policy's
budget, so a replica that hung used it all and the try on another replica had none left: under `fail: closed` the
request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same
question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut
short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken
are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before.
On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge
on a GPU.
- **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
judge that is slow is not asked again, and an alarm is raised once per request, not once per try.
- Node discovery for Prometheus listed only the control plane asked and the
store, so a machine that is both was a control plane to one and a store
member to the other — scraped twice — and a standby asked alone lost every
node. Every control plane lists all of them alike; deploy/monitoring's
prometheus.yml says to scrape and ask each.
- A model whose weights and context pool did not fit its card was added
and then failed to make its context on every start, taking the card from
others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on
24 GB). `ctl model serve` shrinks the pool to what the card holds, saying
so, and refuses weights that do not fit at all; the portal's serve form
refuses such a pool and says the largest that fits.
- Removing weights from the mirror that a catalog model still loads is
refused, naming the model: removed from the mirror alone, the model stayed
in the catalog and its replicas went on loading the node's copy.
- Jobs ran one at a time, so a 3D job waiting for the card held up another
model's job that could have run. They run side by side, one per model at
a time and eight in all, on the leader as before.
- `llmgw ctl model serve` served an embedding model as a chat model: no
`--embedding`, four slots, a 512-token batch, `embeddings` unset and a
"say hello" smoke test it could not pass. A model whose GGUF says it
pools (or `--embeddings`) is served for embedding: 32 slots of 1024
tokens unless sized otherwise, a batch that holds a whole passage, and
one vector asked for as its smoke test.
- `llmgw ctl upgrade` checks every node's version once the waves are done
and fails naming any not on the release, and says when the control plane
has no nodes registered; `llmgw upgrade` on a control plane says how many
nodes it left behind and the command that upgrades them. A site's nodes
stayed a release behind with nothing to say so.
- The cluster and node pages swapped their live part every few seconds,
forms and all, so filling one in raced the clock: what was typed, an
open choice and the focus were lost. The refresh now waits while a field
in it has the focus or a form in it has been typed into and not sent;
the mirror page's reload during a download waits the same way.
## v0.14.5 — 2026-10-14
Picture and 3D models that add themselves where a card fits them, with a
card on the portal and a screen in the console to see and add them; a
mesh's colour brought to the picture it was made from; and the whole
licence written into every mesh.
**Upgrading.**
- The store gains a table (migration 0047: preset_state); nothing to do.
- **`presets.auto` is on**: once upgraded, the leading control plane adds
flux-schnell and triposr, on demand, where a node's card fits one, the
cluster installs ComfyUI (`engines.comfyui`) and the catalog makes
nothing of that kind. Nothing is fetched until somebody asks for one.
Turn it off with `llmgw ctl model presets auto off`.
- **Nodes** need this release for `albedo_match` (it is in the save node
the gate puts in each replica); a ComfyUI replica takes it when it
starts again.
### Added
- **A mesh's base colour brought to its picture's**: `LLMGWSaveMesh` takes
`albedo_match` with the picture and its mask, and scales the texture in
linear light to the object's brightness in the picture. Stable Fast 3D
takes the light out of what it sees and read a dark basalt (0.17) as a
light stone dimly lit (0.67); matched, it is 0.17. On sf3d it is the
`albedo_match` input (off unless asked).
- **Picture and 3D models by themselves**: once a node has a card that fits
flux-schnell or triposr and the catalog makes nothing of that kind, the
leading control plane adds the preset on demand — so a new cluster's
portal has its Images and 3D pages without a definition written — where
the cluster installs ComfyUI. `presets.auto` turns it off; a preset
somebody removes is not added back. `GET /api/v1/presets` says how each
stands and why one is not added; `POST /api/v1/presets/{name}` adds one
(and sets `engines.comfyui` if the cluster has none); `PUT
/api/v1/presets/auto`. Each automatic addition is in the audit log.
- **Presets on the portal and in the console**: the cluster page's
**Pictures and 3D models** card lists each preset — what it makes, the
classes it fits here, whether and by whom it was added, why the leader
has not — with an **Add** button that says first when it would install
ComfyUI and restart the control planes, and the switch for `presets.auto`.
The console's models screen shows the same; **A** adds one, **P** flips
the switch. `ctl model add PRESET` and `ctl model presets` go through the
API (and `ctl model presets auto on|off`), falling back to the entry as
v0.14.4 sent it on an older control plane.
### Fixed
- A mesh carried only its licence's name, URL and notice; it carries the
whole licence as `/v1/models` gives it — id, commercial use, excluded
regions, revenue cap, attribution — for a pipeline that sorts assets by
licence.
- A workflow's `{{name|false}}` default filled a BOOLEAN input with the
word rather than a boolean.
## v0.14.4 — 2026-10-13
Any ComfyUI workflow as a job, with typed inputs; picture and 3D models
added by name; profiles that fit whatever card a site has, and ComfyUI on
a V100; licences a machine can match and a person can leave out; a job
waiting for the one card says what holds it. Many fixes to on-demand
models taking turns on one card.
**Upgrading.**
- The store gains a table (migration 0046: user_excluded_licences);
nothing to do.
- **Nodes** need this release for ComfyUI on Volta or Pascal cards (the
cu126 build), for a driver from NVIDIA's open kernel module to be read,
and for a node that loads its weights by repository name (sf3d); a
ComfyUI already installed is kept.
- **Control planes before nodes**: an entry using the built-in `triposr`
workflow, or a `gpu:` need instead of a class, is refused by an older
control plane.
- Models added with `llmgw ctl model add <preset>` take the preset's name;
remove an entry of that name first.
### Added
- **The client API described**: OpenAPI 3.1 at `/v1/openapi.json`, no key
needed, generated from a route table a test holds to the router's own;
and `/docs/api`, Swagger UI (served from the gateway) over it and, for
staff, over the admin API's document.
- **A job waiting for the card says so**: `progress` is
`{"state":"waiting_gpu","held_by":…,"eta_seconds":…}` while another
on-demand model holds it; a direct request's 503 carries
`X-Llmgw-Waiting-For` and `X-Llmgw-Eta-Seconds`.
- **Any ComfyUI workflow, as a job**: `POST /v1/workflows/runs` with
`{model, inputs}`. A model's catalog entry declares its inputs
(`comfyui.inputs`: text, int, float, bool, choice, image, mask, audio,
video, file, with bounds and defaults) and its output (`comfyui.output`:
pictures, meshes, video, audio or files); a request is checked against
them before a job is written, file inputs are base64, a data URL or a
`/v1/files` id, and `/v1/models` lists each model's `workflow` so a
client can build its form. A PNG result carries the licence in an
`iTXt` chunk, a GLB in `asset.extras` as before.
- **Environments**: `comfyui.environment` names ComfyUI models that agree
on their custom nodes and may share one replica; the catalog refuses
one that cannot.
- **Profiles that fit the hardware**: a profile may say `gpu:
{min_compute_capability, min_memory_gb}` instead of `gpu_class`, and
stands for every class in the cluster that meets it, as
`<profile>@<class>`; a ComfyUI profile may carry its own `comfyui`
workflow, checkpoint and nodes, and any profile its own `source`. One
catalog entry then runs fp8 where the card has it and fp16 on Turing.
- **Custom nodes that compile**: `nodes[].build` names directories of a
node that are C++/CUDA packages; the gate builds each as a wheel for the
card the replica runs on, keeps it on the node, and installs it beside
the replica. `comfyui.huggingface_repo` shows a model's weights to a node
that loads them by repository name.
- **A licence a machine can match**: `license.id`, `commercial`,
`excluded_regions`, `revenue_cap_usd`, `attribution`.
- **On-demand models in `/v1/models`**, with `loaded` saying whether one is
up now; a picture is cut out and put on grey before TripoSR sees it
(`LLMGWPrepareForMesh`, rembg when it has no alpha).
- **Licences a person leaves out**: on the profile page, by licence, or by
rule — every licence forbidding commercial use, every licence granting
nothing in a region — so a model under one is neither listed to nor
served for that person's keys and sessions (403 `licence_excluded`).
- **Presets: a picture or 3D model added by name.** `llmgw ctl model add
flux-schnell` or `triposr` adds an entry that comes with llmgw — its
workflow (TripoSR's now built in), custom nodes, packages and licence,
and a profile by card need rather than class — and has the nodes install
ComfyUI when the cluster's settings name none. `llmgw ctl model presets`
lists them; the console's **A** on the models screen offers them beside
a definition file.
### Fixed
- A model waiting for another on-demand model to give up the card failed at
once as "cannot be placed"; it now waits, and a job with it.
- A model evicted for being idle was placed again on the next pass, ahead of
the one it made room for.
- A job or batch leased by a standby control plane, whose router has no
endpoints, waited on "loading" for good; they are served only by the
leader. A job told its model fits nowhere asks again for five minutes
before failing, which an agent re-registering makes true for seconds.
- An on-demand replica adopted after a restart was never idle enough to
evict, so a model waiting for its card waited out its whole idle timeout;
it counts as used at adoption.
- A picture in a prompt was counted as its base64 by the context check, so
one 1280x720 PNG was "138 thousand tokens" and a vision model refused it;
a picture or a sound now counts as about a thousand.
- An on-demand model of a higher class was evicted, idle, for a lower-class
one two minutes into a lull; it keeps the card through its idle timeout,
and a model waiting on it is told so.
- A model removed from the catalog stayed wanted, so added again it loaded
unasked.
- A node that loads its weights by repository name (sf3d) went to Hugging
Face for a gated repository and got a 401: the agent did not hand the
gate `comfyui.huggingface_repo`, so the weights on the node were never
linked into its cache.
- **ComfyUI on a V100.** torch's CUDA 12.8 and newer builds have no Volta
kernels, but its cu129 build was taken to run from compute capability
7.0, so on a V100 both ComfyUI builds were installed, about 10 GB each,
and failed with "no kernel image". Cards older than Turing now get
ComfyUI on PyTorch's CUDA 12.6 torch (build `cu126`); a Turing or newer
node keeps the build it has.
- **The open NVIDIA driver read by its version.** With the open kernel
module, the default since 560, the agent read the driver as "for" and so
as unknown, and tried every engine build whatever driver it needs.
## v0.14.3 — 2026-10-10
3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
with their progress, their licence written into the mesh, a price per mesh
and a page to make them. Also: a profile's own weights, a band for bulk
work, and fixes for control planes behind a balancer and agents that
restart.
**Upgrading.**
- The store gains tables and columns (migrations 0043–0045: notify_sent,
jobs, usage meshes); nothing to do.
- **Nodes** need this release for their engines' output to survive an
agent restart and for ComfyUI's custom nodes and progress; engines
started by an older agent keep its pipe until they are restarted.
- A 3D model needs a model that makes them (see install.md, "A model that
makes them"); none is added.
### Added
- **Jobs: 3D models asked after, not waited for.** `POST
/v1/3d/generations` answers 202 with a job id; `GET
/v1/3d/generations/{id}` says its status, progress and, when done, the
file its result is in (`/v1/files/{id}/content`); `POST .../cancel` stops
it, in the engine's queue too. The request is replayed by whichever
gateway holds the job's lease through the same path a client's request
takes — key, allowlist, limits, admission, guardrails, accounting — so
it is charged like any request, tagged `job=<id>`, and a model still
loading is waited out for as long as the job has (24 h) rather than
refused. A job survives the gateway that took it; results are kept for
`router.batch.retain_days`. Migration 0044.
- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The
picture is uploaded to the replica's ComfyUI and the workflow's
`{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
stronger ones can be added the same way.
- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in
each replica's own `custom_nodes` by the gate, with its packages
installed beside the replica, never into ComfyUI's environment.
- **A job's progress**: the gate listens to ComfyUI's websocket and the job
says queued and how many ahead, then the node running, its step and
percent.
- **A model's `license`** (name, url, notice), in `/v1/models`, in the job,
on the portal and in the GLB's `asset.extras`.
- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count
`meshes`.
- **`/3d` on the portal**, beside `/images`: a picture chosen, the job
followed, the mesh in a viewer with a download and its licence; and
**`llmgw ctl 3d list|get|cancel`** for every owner's jobs.
- **A profile's own `source`**: the weights it loads instead of the
model's, so an embedding model can run on vLLM from its repository on a
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
nodes holding each profile's weights, `ctl model add` checks every
source, and `ctl weights` collects none a profile loads.
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
long its requests may wait for room, instead of the admission's 30 s. A
band for bulk work — a knowledge-base import — waits behind people rather
than being turned away.
### Fixed
- **An engine's output no longer breaks when its agent restarts.** Engines
wrote to a pipe of the agent's, which an agent that restarted and adopted
them left with no reader; an engine's next large write failed, and in a
Python engine took the request it was reporting on with it. Each process
now writes to its own file in the agent's work directory, and an adopted
engine's log keeps what it wrote before the restart.
- **Single sign-on behind a balancer.** The cookie that carries a sign-in
across the identity provider was signed with a key of each process's own,
so with two control planes a callback that reached the other one was
refused. The key is now a secret in the store, the same on every control
plane.
- **Notifications sent once by two control planes.** Each kept its own
hour of quiet per event, so a budget spent on a team both served was
announced twice. The control plane that claims an event in the store
(migration 0043) sends it; with the store unreachable it is sent anyway.
- **A request turned away for a busy cluster says when to come back.** A
request that waited as long as it may (504) or found the queue full (503)
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
of sending it again; both now say 10 seconds.
- The usage sums and exports count pictures (`images`), as the request
records did; pictures made before are added to the sums.
## v0.14.2 — 2026-10-10
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
cluster's own cards. The agents install ComfyUI themselves, the gateway
prices pictures by the picture, and the portal has a page to make them.
Also: on-demand models that survive a failover, and knowledge-base
embeddings fixes.
**Upgrading.**
- The store gains a column (`usage.images`); nothing to do.
- **ComfyUI is installed only where asked**: `llmgw ctl config set
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
or run engine `comfyui`.
### Added
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
key, allowlist, limit, admission and guardrail checks as any request. A
model with profiles on engine `comfyui` is placed on a GPU node like any
other. A ComfyUI server outside the cluster can be an external model with
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
that leaves takes its workflow off the queue. `/v1/models` says `images`
for such a model.
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
gate listens on the replica's port, demands the replica's key, and lets
through only what the router asks. The model's `comfyui:` block names
the workflow and, for a source that is a set of files, the checkpoint.
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
checked against a digest llmgw holds, since GitHub publishes none, and a
version without one is refused. Its requirements are resolved as PyPI
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
CUDA 12.9 one for an older driver, and the result is checked on the card.
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
agent.yaml.
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
prompt's tokens. A model priced only per picture is not free, so a spent
budget still stops it. Each usage row keeps how many pictures were made,
shown by `llmgw ctl request`.
- **`/images` on the portal**, beside the chat, for the models that make
pictures. The chat no longer offers them. The page offers size, how many,
seed, steps and a negative prompt, and shows each picture with its seed
and a download link. Nothing is kept but the usage.
- **An embedding model's `query_prefix`**, put before the question a
knowledge-base search embeds and never before documents, for Qwen3-Embedding
and models trained like it.
### Fixed
- **An awake on-demand model survived no restart or change of leader of the
control plane.** A control plane that had heard no requests read every
on-demand replica its nodes reported as wanted by nobody, and drained them
all. An adopted replica now gets a whole idle timeout.
- `ctl models` said `not placed` for an on-demand model that had been woken
and was serving; it now says `ready (on demand)`.
- A batch of embeddings was refused as one over-long prompt when the
passages together exceeded the context, though each one fit. The longest
input is what has to fit.
- The chat's kept conversation sat in the browser under one key for
everybody, so the next person to sign in on the same browser opened the
last one's conversation. It is kept per account, and signing out clears
what the pages stored.
## v0.14.1 — 2026-10-09
Adapters trained on the nodes from a knowledge base and kept private to
their owners; the cluster's and the agents' settings edited in one place;
the tenancy tree and keys over the admin API and in Terraform; knowledge
bases that hold up with many documents; the firewall setting on every
machine.
**Upgrading.**
- **Nodes** need this release to follow `agents.*` in the cluster's
settings; until a node is upgraded its agent.yaml is what it runs on.
- **A separate mirror machine and store members that run nothing else**
follow the firewall setting once what reads it is installed: run `sudo
llmgw mirror init` on the mirror again with the flags it was set up with,
and `llmgw server init --patroni` again for the store's members. A control
plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
- **The store's PostgreSQL memory** applies after each member is restarted,
one at a time, the leader last; until then it runs as before.
- The store gains tables and columns; nothing to do. Existing knowledge-base
collections get their full-text index when a control plane starts, and
keep `vector` storage; only new collections use `halfvec`, and only on
pgvector 0.7 or later.
- Cached knowledge-base passages in vLLM are made again once, under the new
`cache_salt`.
### Added
- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
for administrators. It edits the document `llmgw ctl config edit` opens:
checked as a control plane would load it, refused when made from a
version since replaced, with the history to view and roll back. Changes
are audited by size, never by contents.
- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
`agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
say but the agent's id, control planes, address and certificate. They are
sent to the agents as they change; an agent starts again on new settings,
or on an edited agent.yaml, with its engines serving throughout. Settings
that would not load are refused on save, and set aside on a node rather
than keep its agent from starting. Nodes need this release to follow
them.
- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
writes question/answer pairs from a collection with a model of the
cluster. `llmgw ctl train start` trains an adapter on a node with its
installed vLLM's torch and a small overlay, on a card with enough memory
free (beside an engine, if it leaves that much), converting to GGUF for
llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
accepts uploads) and makes it the model's adapter.
- **Adapters with owners.** An adapter's `owner` (org, team, project, user
or key) is the only part of the tree that sees it; to anyone else it is a
model that does not exist. An adapter trained on a knowledge base belongs
to a part of the tree that may read it, and is never made public.
- The knowledge bases' vLLM `cache_salt` follows the collections searched,
not the organization, so two teams of one organization no longer share
cached passages.
- **Organizations, teams, projects, API keys and memberships in the admin
API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
`PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
issue (the key is in the answer, once), list and revoke; `/api/v1/users`
and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
and `users` on a control plane before. The Terraform provider manages them
as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
sensitive `secret`) and `llmgw_membership`.
- **The firewall setting reaches every machine.** A separate mirror and the
store's members follow `firewall.enabled` and `firewall_networks` as nodes
and control planes did: the leading control plane tells the mirror over
its API, and keeps the state in the store for members that run no llmgw,
which a timer on each reads. The mirror also lets in the network of every
node the leader knows. `mirror init` and `server init --patroni` install
what follows it; run them again on a cluster set up before.
### Changed
- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
machine. Now each member gets a share of its own RAM, a smaller one where
a control plane or agent shares the machine, and the share is read again
every time `server init --patroni` runs. The new cache is applied by
restarting one member at a time, the leader last.
- **Knowledge-base searches hold up with many documents.** New collections
store their index as `halfvec` where pgvector is 0.7 or later (half the
memory, and embeddings up to 4000 dimensions instead of 2000); older
collections, and clusters on pgvector 0.6, keep `vector`. Each collection
has its own full-text index, as it has its own vector index, made for
existing collections when the control plane starts. The text arm ranks by
the cheap measure first and scores only those candidates.
### Fixed
- A knowledge-base search asking for more than 10 passages
(`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
raised to the number wanted, up to 1000.
- `llmgw server backup` and `restore` against a replicated store over TLS
failed looking for `~/.postgresql/root.crt`: pg_dump was given
`sslmode=verify-full` without the cluster's authority.
## v0.14.0 — 2026-10-08
Knowledge bases: a company's documents, put in by people from their own
machines, kept apart per tenant, and answered from.
**Upgrading.**
- **pgvector on a cluster set up before this release.** Knowledge bases
need pgvector, which `server init` and the high-availability database
setup now install, but a rerun on a running cluster does not:
1. Install `postgresql-<version>-pgvector` on every database machine.
2. As postgres, run `CREATE EXTENSION vector` in llmgw's database.
3. Restart the control planes.
Until then everything else works, and `llmgw ctl kb list` says why
knowledge bases are off. On a SQLite store they stay off.
- **An embedding model.** A collection needs a catalog model with
`capabilities.embeddings`, for example bge-m3, whose weights are in the
mirror.
- **The client.** People download `llmgw-kb` for their system from this
release.
### Added
- **Collections and who may read them.** A collection is owned by an
organization, team, project or person. Grants add up down the tenancy
tree and never cross an organization. A collection a key may not read is
answered as one that does not exist. Chunks are kept in PostgreSQL with
pgvector, partitioned by organization, with an HNSW index per collection,
and row-level security on the table as a second line behind every
query's own filter. A project, team or organization that owns collections
is not deleted; a person's collections go with their account.
- **`llmgw-kb`**, its own download for Windows, macOS and Linux, run by
people with their own key. It reads documents where they are and sends
text and vectors, never the files, for what changed only; a SQLite ledger
on the machine remembers what it did.
- It reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML,
MHTML, mail with attachments, notebooks, subtitles, ZIP archives, and
text and source code in any common encoding.
- Legacy Office files are read with LibreOffice, and scanned pages with
Tesseract, when those are installed.
- `--dry-run` says what would leave the machine; `--delete` removes only
what that machine sent.
- **Answers from documents.** A chat request names collections in
`X-Llmgw-Knowledge`, or its part of the tree has a default (`llmgw ctl kb
default`). The best passages, chosen by meaning plus a little for the
question's words, are added to the system message, numbered, and
`x-llmgw-sources` says which were used. This happens before the
guardrails judge the request, and the answer is never served from the
answer cache. On vLLM replicas the organization's `cache_salt` keeps
tenants' prefix caches apart. It works through the Responses and Messages
APIs too.
- **Administration**: `llmgw ctl kb` (create, list, grant, revoke, default,
docs, rm, search, delete), `/api/v1/kb`, the console's Knowledge screen,
and a Documents picker on the portal's chat page.
- Metrics `llmgw_kb_retrievals_total` and `llmgw_kb_retrieval_seconds`;
[docs/knowledge.md](docs/knowledge.md).
## v0.13.0 — 2026-10-08
The whole system stopped, started and restarted from one command, a
guardrails judge that no longer refuses ordinary coding requests, and the
rest of the field report from v0.10.6.
**Upgrading.** A cluster keeps the judge its policies name: one installed
before this release goes on asking Llama Guard 3 1B until `llmgw ctl guard
judge use qwen3guard-0.6b`, which the leader then fetches and serves by
itself, and which replaces the policies' Llama Guard category codes.
### Added
- **A judge on a GPU and one on CPUs together**: a policy's `judge.models`
is a list asked in turn, the first that answers giving the verdict; the
next is asked when one has no replica serving, fails, or misses its
`latency_budget`. `llmgw ctl guard judge use qwen3guard-4b
qwen3guard-0.6b` sets the order for every policy with a judge, and
`guard judge status` says which judge stands in for which. A judge on a
GPU is served as a `standard` model, so it takes a card only from what
may give one up.
- **A judge a policy names is fetched if it is missing**: the leader
fetches the weights of a judge llmgw knows (`qwen3guard-0.6b`, `-4b`,
`-8b`, `llama-guard-3-1b`, `-8b`) into the mirror and serves it, when a
policy names it and the catalog does not have it — and again if it is
removed. `guard judge enable --gpu` serves Qwen3Guard-Gen 4B on the class
with the most free cards.
- **The whole system stops, starts and restarts from one command**: `llmgw
stop` takes every model down and answers every request for one with 503
(`system_stopped`), keeping the control planes, agents and mirror running
and no model's definition changed; `llmgw start` brings every model back.
It is the cluster's setting `serving.stopped`, live on every control plane.
`llmgw restart` restarts every node's agent and every control plane, the
leader last, with the engines serving throughout. `llmgw status` says
whether the system serves. One machine's services are `llmgw service
status|start|stop|restart`.
- **A replica says what it computes on**: `ctl replicas` (ON), the console
and the admin API show `CUDA` with the GPUs its processes hold memory on
and how much, as nvidia-smi sees them, or `CPU` — for every engine, kept by
the agent across its own and the control plane's restarts. llama.cpp's
layer count is added when it says it, which it does only at `-lv 4`.
- **`ctl engine-check llamacpp` computes a token on the GPU**: a small GGUF
(`--model`, or the smallest in the weights cache) is loaded with every layer
on the GPU at `-lv 4` and asked for one token; a build whose CUDA backend
does not load fails, by llama.cpp's own account of its layers. With no
`--bin` it checks the build the agent installed.
- **Engine versions and compute capability on nodes**: `ctl nodes` shows
`llamacpp b10991` and `6000-24g (7.5)`; the admin API gives
`engine_versions` and `compute_capabilities`. A profile's `engine_version`
restricts it to nodes whose engine version starts with it, and
explain-placement says which version a node has, or which engines.
- **`ctl restart --max-surge N` and `--max-unavailable N`**: that many of a
model's replicas restart at once, beside themselves or stopped first.
- **Multi-node readiness apart from the node's**: doctor gives both, with
fabric findings in a section of their own; `node add` says the multi-node
one; `ctl nodes` shows a declared fabric's adapters that are down.
- **A forgotten password is set through a link**: `/forgot` mails one when
the cluster's `mail` settings say how (SMTP, sender, the public
`base_url` it points at), saying the same whether or not the address has an
account; `/reset` sets the password once and closes every session. Without
mail, `llmgw users reset-link` makes the link for an operator to hand over.
### Fixed
- **A replica restarted in place keeps its room**: the replica was stopped
and the pass went on to other models before placing its replacement, so a
model waiting for room took the freed cards. On the reference cluster a
GPU judge waiting for the only card took it from the model being
restarted there. The replacement is now placed in the same step.
- **The judge no longer refuses ordinary coding requests**: Llama Guard 3
1B, the judge `server init` installed, could not tell "continue", "devam
et" or "how do I kill all child processes in bash" (0.51 to 0.92 unsafe)
from harmful requests (0.97 to 1.00), at any quantisation. The judge is
now Qwen3Guard-Gen 0.6B (format `qwen3guard`), which gave the same
ordinary requests 0.15 at the most and the harmful ones 1.00, on CPUs,
in 0.8 GB. Its third verdict, controversial, counts as safe unless the
policy says `controversial: unsafe`. `guard judge use qwen3guard-0.6b`
moves an installed cluster to it, and replaces Llama Guard's category
codes, which Qwen3Guard never names, with the baseline's.
- **A long message is judged rather than refused**: a judge read the whole
last user message, and one with a file pasted into it overflowed the
judge's context, so it gave no verdict and `fail: closed` refused the
request. It now reads the start and the end, about 6000 characters.
- **A replica no longer stays "starting" while it serves after a restart**:
an agent that registered before its adopted engines had answered their
health check never said they had, so the control plane kept them starting
and routed nothing to them — on the reference cluster, after `llmgw
restart`, the only replica of a model. The agent now reports each shard
the registration called not ready once it is.
- **A failover no longer replaces healthy replicas**: a standby that took
over placed at once, before the agents had reconnected and said what runs,
so it started a second copy of replicas it did not know of and then drained
the healthy ones as surplus — on the reference cluster the guardrails'
judge and a model's replica, within twenty seconds of a leader restarting.
Placement now waits after a takeover as it does at start.
### Changed
- **A GPU placement llama.cpp says it cannot compute on is refused** when it
warns that it found no usable GPU, even where nvidia-smi cannot be asked;
where it can, holding no memory on its GPUs refuses it as before.
## v0.12.2 — 2026-10-06
The guardrails turned off and on as a whole, and a judge that refuses
ordinary requests explained: the hazard it saw is told to the client and
kept with the request, and `llmgw ctl guard judge ask` shows what it says of
any text. Nothing to do to upgrade; the store gains a column.
### Added
- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
`on`, which is `guard.enabled` in the cluster's settings — in force within
seconds, the policies and their attachments kept. `ctl status`, `ctl guard
list` and the guardrails page say when they are off.
- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
its answer as given and as read — for finding out why ordinary requests
are refused. Nothing of the text is kept, not even in the audit.
### Changed
- **A judge's refusal names the hazard**: the client is told `(judge:
S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
the request (`guard_rule`): `ctl request`, the server's log and the
guardrails page show it. Llama Guard 3's codes come with their names,
S9 Indiscriminate Weapons, without a categories list.
## v0.12.1 — 2026-10-06
The guardrails' judge, served and watched without server init: a judge
missing, served as the wrong kind of model, or not answering is said on
`ctl status` and the console, and `llmgw ctl guard judge enable` puts it
right. Nothing to do to upgrade.
### Added
- **`llmgw ctl guard judge status` and `enable`**: each judge model the
policies ask, whether it is in the catalog, as a judge, with replicas
ready and answering — and serving the missing one, Llama Guard 3 1B on
CPUs under the name the policies give it, with `--replace` for a model of
that name that is not a judge. No more `server init` to put a judge back.
Also **Do → Serve the guardrails' judge** in the console.
- **A missing or broken judge is said where it is looked for**: `llmgw ctl
status` and the console's Overview list each judge with a problem — not in
the catalog, not a judge, no replica ready, or failing to answer — and
what that does to requests (refused or unjudged).
### Fixed
- **A Llama Guard file served from the mirror page is the judge**: it was
added as a model clients use, under whatever name the form had, so the
policies never found it and refused everything they judged
(`guard_unavailable`). It is now added as a judge, under the name the
policies ask.
- **`ctl` says so when the API answers with a page**: a command a control
plane does not know — the leader on an older release, through a standby —
failed with "invalid character '<'"; it now says which and why.
## v0.12.0 — 2026-10-06
A minor version, because where the configuration lives changes: everything in
server.yaml that is not one machine's own is now the cluster's settings, kept
in the store, the same on every control plane and in force within seconds of
a change. Nothing needs doing by hand to upgrade. The first control plane to
start on v0.12.0 brings its server.yaml's cluster keys into the store, with
the baseline guard policy when the file had none. From then on, change them
with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
ctl upgrade` does, the leader last): until the leader runs v0.12.0,
`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
the firewall switch.
### Changed
- **The cluster's settings live in the store, and every change is in force
within seconds.** Everything in server.yaml that is not one machine's own
(guardrails, limits and prices, routing, engines, notifications, the
mirror's policy, declared models, offline, federation, facility budgets) is
a versioned document in the store, the same on every control plane.
`llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
console's Settings screen edits it. A change is checked, kept with who made
it and why, and applied: guardrails live, anything else by every control
plane starting again on it, standbys first and the leader last. A change to
a machine's own keys in its server.yaml is noticed and applied the same way.
No "restart now?" any more, and no second copy to keep in step. The first
start against a store brings in the file's cluster keys, with the baseline
guard policy when the file had none, so an upgraded cluster gets its
guardrails; after that, those keys in a server.yaml are ignored, with a
warning.
### Added
- **llmgw's firewall can be turned off and on for the whole cluster**:
`llmgw ctl config set firewall.enabled false`. Every node's agent takes its
table out within seconds, and each control plane's new
`llmgw-firewall-sync.path` does the same there; `true` puts the rules back
as they were. `firewall_networks` changes reach the nodes the same way.
- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
configuration, the authority, secrets, state, users — after listing it and
asking for the machine's name; `--models`, `--store` and `--all` take
weights, the store's database here, backups and the binary too. Also
**Do → Wipe this machine** in the console.
### Fixed
- **The mirror is given an address the nodes reach**: `server init` and
`mirror init` took the machine's first address for the mirror's `--base`,
which on a machine with Docker could be 172.17.0.1 — a node then asked its
own bridge for every model and was refused. They now take the source of
the default route, else the first address not on a container's or a
bridge's interface. And `node add` has the node reach the mirror's files
before it is added: one that cannot fails the checks, naming the address
and what to change.
- **A standby that becomes the leader reads the store again**: models added
or removed, engine builds chosen and domains held through the leader before
it, rather than placing on what it read when it started.
- **A node on another network joins**: `node add` set up the node's firewall
but left the control plane's to the networks it was installed with, so a node
on another subnet was installed and then never heard from (lost). It now lets
the node in on the control plane it runs on — its subnet, or its address
when a router is between them — and says so; a network already let in
changes nothing.
## v0.11.4 — 2026-10-05
Two control planes, run in anger on the reference cluster.
### Added
- **The console sets up a second control plane**: Do → Add a control plane
asks for the machine and a floating address and runs `server add`.
- **Who leads, everywhere.** `ctl status` and the console's Overview have a
`CONTROL` line: which control plane leads, which stand by, one that does not
answer, and whether the floating address is on the leader.
- **A standby's console is the leader's.** Its admin API passes `/api/` to the
leader over the agent port, with the cluster's certificates both ways, so
`sudo llmgw` and `llmgw ctl` show the same cluster on either machine. Control
plane certificates are now issued for client use too; `server add` reissues
older ones.
- **`ctl upgrade` in waves.** Nodes are upgraded one, then two, four… up to
`--parallel` (20) at once, so hundreds take minutes; a wave that does not
come back stops it. The other control plane is upgraded first, over ssh.
- `server add` says when the mirror's `--api-allow` would refuse the new
control plane.
- **Guardrails in the console**: a view of the policies and what each checks,
whether their judge models are serving, where they are attached (attach
and detach from it), and what they did in a window. `GET
/api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
what each policy does.
- **A standby's portal is the leader's.** Its client port passes everything
but `/health`, `/ready` and `/leader` to the leader, and its admin port the
dashboard as well as the API; a proxy left pointing at a standby showed an
empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
- `ctl model serve --tools` says in the catalog that a model makes tool
calls, for agents and editors.
### Fixed
- **Continue's agent did nothing.** The configuration the portal writes told
Continue every model uses tools, and nothing of its context: a model whose
calls came back as text stalled the agent, and prompts over a CPU
replica's 4096 tokens were refused. It now says `tool_use` only for a
model whose catalog says tools, and the context and longest answer the
gateway allows.
- `ctl model serve` gave llama.cpp four slots whatever was asked: a
`--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
refused what the gateway admitted. Slots are now what the pool holds of
the context. The longest answer grows with the context (a quarter, from
1024 to 8192) rather than being 1024 for every model.
- `ctl upgrade` upgrades the leader last wherever it is run; run on the
standby, it upgraded the leader first and the leadership moved twice.
- **Failover took 40 seconds; it takes a few.** A stopping control plane held
every agent's stream for the thirty seconds of a graceful shutdown; it now
gives up its lease first and lets the agents go at once.
- A control plane that lost its lease while running kept its agents, leaving
the new leader with no nodes. It now lets them go.
- An agent's reconnect backoff carried over from one outage to the next, so
each failover made the next one slower; it starts again after a session that
worked.
- The `CONTROL` line named no floating address.
## v0.11.2 — 2026-10-05
### Routing by what each vLLM replica's cache holds
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
publishing its KV cache events (`--kv-events-config`) to a port on
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
computed outside Python. The events never leave the node.
- The agent mirrors each engine's set of cached blocks and sends the net
changes up its existing stream to the control plane four times a second
(new `KVEvents` agent message), starting every new stream with a full
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
remote ones and a hybrid model's second cache group are not indexed.
- The control plane passes them to the router's index, only for replicas
the sending node runs, and empties what a stream described when it closes.
- For a model with `kv_events`, the router asks one of its replicas to
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
chat template is applied exactly), hashes the tokens into blocks as vLLM
does, and scores replicas by the share of the prompt's leading blocks they
hold. The imbalance guard still applies. With no answer — no events yet,
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
back to the approximate tree (`router.prefix`) or to load.
- The agent checks first blocks against llmgw's hashing; an engine that
hashes another way (a profile overriding the algorithm, a changed vLLM) is
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
though it published nothing.
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
- Experimental: tested against vLLM's event format and hashing as its source
defines them, and against libzmq and msgspec publishing — not yet against a
running vLLM.
### LoRA adapters on vLLM, loaded on demand
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository
id) is registered at start with `--enable-lora --lora-modules`, and a
request names it as its model, as on llama.cpp. `scale` applies on
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
its adapters. Replicas start holding none (vLLM with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
profile's args set their own). The router loads an adapter on a replica
through `/v1/load_lora_adapter` when a request first wants it there, and
unloads the least recently used one no request is running on to make room.
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
llama-server can rescale an adapter it started with but not load another.
- **Adapter affinity:** a request for an adapter goes to a replica that has
it, or is loading it, while that replica is not saturated; otherwise to one
with room for another adapter; otherwise where load sends it — which is how
a busy adapter spreads to a second replica. Requests arriving during a load
wait for it instead of loading the adapter again.
- **Self-correcting:** each replica's `/v1/models` is read when the router
first meets it and every 30 seconds, so adapters loaded before a restart or
by another router are known; a 404 for an adapter the router thought loaded
loads it again and retries. An adapter vLLM refuses (not found, rank above
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
on every replica.
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
`llmgw_adapter_loads_total{model,outcome}`,
`llmgw_adapter_evictions_total{model,outcome}`.
- The admin API and `ctl model adapters` read the file as GGUF only for a
model with llama.cpp profiles.
### Federating clusters: another llmgw cluster as a provider
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
makes another llmgw gateway a peer. It is checked first (its `/ready`,
timed, and its `/v1/models` with the key) and kept in the store (migration
0029, `federation_peers`) with the models it offers.
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
follow the peer's own list every `federation.interval` (10s). A model the
peer drops goes, unless a model here still falls back to it: then it is
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
The peer's aliases and its own peers' models are not taken.
- A peer whose `/ready` fails, which refuses the key, or which does not answer
is taken out of routing until it answers again. `/ready` now counts only a
gateway's own endpoints, not its peers'.
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
overflow can go to a peer's model.
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
have passed through (each cluster's id is kept in the store). A request that
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
request came from is not sent it.
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
org=,team=,project=` (ids); off by default.
- In-cluster residency and `no_store` keep requests off peers, by name and as
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
as the replica.
- `ctl federation list|remove|sync`, a Federation view in the console,
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
API `/api/v1/federation`, and a "Federating clusters" section in
docs/install.md. New `federation:` block in server.yaml (`name`,
`interval`).
- A response header the gateway already set (X-Request-Id,
x-llmgw-served-model) is no longer repeated from the upstream's.
### Facility-aware placement: racks, PDUs, holds and power caps
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
is kept in the store (migration 0031, `node_labels`), wins over the agent's
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
A node with no location places as before.
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
node sharing no rack or PDU with its others when there is one: an ordering
above weights already on disk and bin packing, never a filter.
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
multi-node replica's nodes in one such domain.
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
restarts (migration 0031, `facility_holds`).
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
profile's `power_watts`, or its cards at the power limit they report
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
per node. Use is the larger of reserved and measured, node by node; a shard
that would take a domain past its cap less the margin is refused, and a
multi-node replica's shards are counted together. Unknown draw never blocks.
- **Seen everywhere it matters.** `ctl explain-placement` names the held
domain or the kilowatts, and keeps the rejections of a replica that was
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
with nodes, use, limit, reserved, measured and holds. The console's Nodes
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
- Docs: `docs/facility-aware-placement.md` says what is built and where it
departs from the design; `docs/install.md` and `docs/console.md`.
### SGLang measured, checked and benchmarked as itself
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
SGLang was started without `--enable-metrics`, so it served no `/metrics`
at all: every scrape failed as though the replica were unreachable, quietly,
and requests were routed to it blind. SGLang is now started with
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
prompt and generation token counters and `sglang:num_requests_total`, as
0.5.21 publishes them.
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
`--help` and checked SGLang's flags against Python's. It now asks
`python -m sglang.launch_server --help`, in the environment SGLang was
installed into (or `--bin`).
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
abbreviation of `--tp-size`; it is given `--tp-size`.
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
key given for `--target` — a client's key when the target is the gateway,
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
engine's key, refused every read, and the run reported a queue depth of
zero and suggested no thresholds without saying why. The engine key is now
asked of the admin API for the replica at `--metrics`' address (or given
with the new `--metrics-key`); metrics that cannot be read — refused, or in
another engine's names than `--engine` — fail the run before any load,
saying why; and a step whose metrics went unread shows `-` and the reason
rather than zeros.
### Reusing answers
- **A response cache, per project, off until asked for.** `llmgw keys cache
--project P --ttl 1h` has the same request — same project, endpoint,
model by its catalog name, and body with `stream`, `stream_options`,
`user`, `metadata`, `store` and `service_tier` left out and its keys in
any order — answered from the gateway's memory instead of the model, for
the TTL. `--off` stops it; without either it says how it stands. The
console has it as **Cache answers…** on a project.
- **Only answers that come out the same are reused:** chat and completions
asked at temperature 0 or with a seed, and embeddings, rerank and score.
`--any` reuses sampled answers too. Only a whole 200 from the model asked
for is kept — not an error, a fallback's or a canary's answer, one a
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
- **Streams both ways:** a streamed answer is kept put back together into
one completion (tool calls and reasoning included), and a kept answer is
streamed as chunks — with the usage chunk if the client asked — to a
client that streams. `/v1/messages` and `/v1/responses` get it too.
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
- **Policies still apply:** limits and the input guardrails run before the
cache is looked in, the output policy judges a reused answer as it would
a new one, and a request under `no_store` or continuing a stored
conversation is never served from it nor kept in it. A project's answers
are never looked up by another project's keys.
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
marked `cache_hit` on the usage record (migration 0027), and shown by
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
together in each gateway process, least recently used dropped first;
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
`llmgw_cache_bytes` and `llmgw_cache_entries`.
### Offline sites and Kubernetes
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
collects what a cluster would otherwise download — llmgw's release for
each `--arch` with its signed checksums, every llama.cpp build for those
architectures (or the `--variant` ones) with the checksums GitHub
publishes, the starter model and guardrails' judge, and any `--model
name/version=hf:owner/repo/file` pinned to the registry's commit and its
checksum — each checked as it arrives, into a directory or one `.tar`
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
bundle. Run again into the same directory, it downloads only what is
missing. `llmgw bundle show --verify` reads one back.
- **`llmgw bundle import`** checks every file against the manifest and
llmgw's release against the project's signing key, and only then puts them
into the mirror — releases and engines under `_llmgw/`, models where the
mirror keeps models, with their checksums — so a damaged bundle leaves the
mirror as it was; a different file under a version already there is
refused. On a control plane it also records the engines' checksums in
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
does that alone, for a mirror on another machine.
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
outside: nodes install llama.cpp from the mirror, each file held to the
checksum the control plane sends them over the agent's authenticated
connection — never one from the mirror — and `releases` defaults to the
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
from it; no registry is asked about newer weights. A release may now come
over plain HTTP when the build trusts a signing key, since the signature
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
be unsupported on an offline node rather than attempted.
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
fields on the setup form: installs a control plane offline — the bundle
into the mirror, the checksums trusted, `offline:` written to server.yaml,
the starter model and judge served from the bundle or skipped when it does
not carry them.
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
it lists everything that does, and what an offline cluster changes.
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
port's certificate and the admin token from Secrets, the API and metrics
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
and ServiceMonitor, `config` laid into server.yaml, and the offline
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
### The admin API described, and a Terraform provider
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
and query parameters, and the schema of each request and answer with the
meaning of each field, in
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
every control plane at `GET /api/v1/openapi.json` (with the token, like the
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
in it. Generated by `go generate ./internal/docgen` from a route table in
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
the fields' comments as descriptions; a test fails when an endpoint is
served and not in the table, another when the committed document is stale.
Checked with openapi-spec-validator and openapi-generator-cli's validate.
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
what a request must carry — an address in `Host`, `application/json` on
every change, even a `DELETE` — and some calls with `curl`.
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
force, with the defaults the catalog filled in. The listing has only its
state and its profiles' names.
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
own so the gateway does not carry its dependencies. `llmgw_model` (a model
added at runtime: scaling, class, owner and adapters change in place, the
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
against the OpenAPI document, endpoint by endpoint and field by field, so a
change to the admin API that breaks it fails its tests; CI runs them.
Built and installed by hand for now: see its
[README](terraform-provider-llmgw/README.md).
### Tracing beyond the router
- **The control plane traces what it does to the fleet.** A reconcile pass
that does something is a `reconcile` trace — passes with nothing to do are
not traced at all — with what it did beneath it: `scale up` with `place`
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
candidate turned down and why, as `ctl explain-placement` would print it),
`scale down`, `drain replica` and `stop replica` each with its reason,
`evict`, `preempt` with its victims, `reap replica` with the error and
when the model is tried again, `restart replica`, `restart refused`,
`node unreachable` and `node lost`.
- **A replica's start is one trace, down to the node.** `replica start`
stays open from placement until the replica serves, fails or is stopped,
with each shard's progress as an event and how long it took. The command
to each node is a `send start` span whose context travels in the command
(a new `trace` field on the agent protocol's `ControlMessage`), and the
agent continues the same trace: `start shard`, with `fetch weights`
(local, cached, downloaded, or downloaded by another shard; a `download`
with its bytes and where a resume began, the source's credentials and
query left out), `launch engine` (and `wait for rendezvous` on a worker
of a multi-node replica), and `wait for engine` with how often the engine
was asked and its last answer. "Why did qwen take four minutes to start"
is that one trace.
- **Stops, drains and adoptions on a node** are in the control plane's
trace too: `stop shard` and `stop engine`, saying whether the engine had
to be killed; `drain shard`; and `adopt` when a registering node reports
engines already running, with how many nobody claimed. An engine install
is an `install engine` trace on the node.
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
`serve model` and `update model`, with a span per step — registry,
mirror, check, catalog, placement, smoke test, replicas — and from the
placement step a link to the replica's own start.
- **The agent has a `tracing` block** in `agent.yaml`, the same as
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
when unset). Point both at one collector. A node follows the control
plane's sampling for every start it is sent. Traces from a node carry its
id as `service.instance.id`, and a control plane's its host name.
- With no endpoint nothing changes: every span is the no-op one, and a
command carries no trace context.
### Quality suites, and new weights held to one
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
and scores its answers: per case, then the cases weighed into one score, 0
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
and one way to score it — `exact`, `contains`, `regex`, `number` within
a `tolerance` (the last number in the answer), `json_schema` (the answer
read as JSON, in a code fence or a sentence or not, held to a schema), or
`judge`: a rubric another model on the cluster grades 0 to 1. A
reasoning model's `<think>` is not part of its answer.
- **Run on the control plane**, through the router as its own requests,
as a replay is: past admission, quotas, accounting and the guardrails.
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
`deploy/eval/`, sized for small models. `--suites` lists them.
- **Kept** in the store (migration 0028): every run with its model, the
repository and revision of the weights that answered, the suite's name
and hash, its score and each case's answer and reason.
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
`model promote` take `--eval SUITE --min-score N`, or a model carries one
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
A promotion asks the canary first and leaves it in place below the score;
an update runs the new weights first as a canary given none of the
model's requests, asks it, and promotes it or takes it out again, so the
old weights serve until the new have passed. The job's `eval` step says
the score; `--skip-eval` moves once without the model's suite.
- **The console** has **Ask it a quality suite…** (the score counted in as it
comes), **Its quality runs** and **Hold its updates to a quality suite…**
on a model's menu.
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
handler cut off at 64 KiB, by the audit log's reading of it.
### MIG instances as GPUs of their own class
- **A card in MIG mode is placed as its instances.** Discovery reports a card
in MIG mode as its MIG instances instead of the whole card, each a device of
its own with its UUID (`MIG-…`), its own memory, and a class named from the
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
any other, and placement puts a replica on a free instance of it. Both
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
instances; `docs/install.md` has a MIG section on making them.
- **One instance a replica.** A process opens only one MIG device, and
instances have no NVLink or peer-to-peer between them, so a profile on a
MIG class asking for more than one GPU, tp or pp above 1, or more than one
node is refused when the model is added, with that reason; placement
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
for a model one holds and never several together.
- **Engines are given the instance by UUID.** Placement numbers card 0's
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
shows the device as `0/mig2`.
- **Memory per instance, no made-up utilisation.** Each instance's memory,
and the processes holding it, are its own — read per GPU and compute
instance from nvidia-smi's report, so a foreign process on one instance
keeps that instance out of placement and not its neighbours. Temperature,
power and wear are the card's. No driver reports utilisation per instance,
so none is published for one: the console says `n/a on MIG`, and
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
- **A card in MIG mode with nothing carved is kept out of placement**, with
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
is on.
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
of a card in MIG mode with no instances, of GPU instances with no compute
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
command for each.
- `ctl nodes` counts a node's devices by class when it has MIG instances or
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
GPU list shows instances under their card with their own memory and the
card's heat and power, and a node's page lists its classes with counts.
`ctl status` and the plan count each device under its own class, where a
node's cards were all put under its first class before.
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
### Engines and the mirror
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
installed it: name its interpreter in the node's `agent.yaml`
(`engines.trtllm.python`) and the node offers it while that interpreter
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
`--pp_size` and served name, through a few lines of its own that make it
demand the gateway's key on every path but `/health` (trtllm-serve has no
key option), turn on the Prometheus endpoint and iteration statistics, and
keep usage reports to NVIDIA off. The router reads it at
`/prometheus/metrics` — `trtllm_num_requests_running`,
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
GPU; one node per replica. Profiles may not give it an options file,
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
engine-check trtllm --bin <python>` checks an installed one. Written
against TensorRT-LLM's documented command line, server and metrics and
tested with stand-ins for them; not yet run on a datacenter GPU.
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
model's projector beside it, each checked against the sha256 Ollama named
its file by. With no name it lists Ollama's models; it finds Ollama's
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
service) or takes `--ollama-dir`. The version is the tag and the start of
the digest, since a tag moves in Ollama. It prints a model definition to
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
view does the same with **I**, for the mirror beside the control plane.
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
mirror machine into it the way a download is kept: whole or not at all, the
checksum beside it, a version never replaced, a `.gguf` checked to be one,
and `--sha256` checked when given.
### The portal in Turkish
- **The web portal speaks English and Turkish** — the landing page, sign-up
and sign-in, the account page and its editor setup, the claim page, chat,
the about page, and every administrators' page (cluster, nodes, NVLink and
fabrics, people, limits, usage, organizations, mirror, guardrails), and
the docs portal around its documents, which stay English. Flash
messages and form errors built by the server are translated too.
- **A switcher at the foot of every page** (English · Türkçe) remembers the
choice in a cookie; without one the browser's Accept-Language decides, and
without that the new `landing.language` (`en` or `tr`, empty is English).
An unknown code stops the server starting. `<html lang>` and
Content-Language follow the page.
- **Figures and dates in the page's language**: a Turkish page writes
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
Turkish. Form values, CSS and SVG coordinates are unchanged.
- **Ready for more languages**: the catalogue is JSON under
`internal/web/locales/<code>/`, and a language is a directory. Tests refuse
a key a page uses that some language lacks, a translation with different
arguments from English, and any template text left in English — so a new
page cannot quietly ship English-only.
- Messages that come from the control plane and other packages (node
warnings, engine errors, store refusals) are still shown as written, in
English; the API, CLI and console are English.
### Documentation
- **A docs portal at /docs.** The manual the binary carries — the pages
`llmgw docs` prints and the console's Manual shows, the configuration
reference among them — rendered on the client port in the portal's own
look: a sidebar of the documents in reading order, the open one's sections
under it, a link on every heading, and links between documents kept inside
the portal (links to files that are not in the binary, such as `deploy/`,
go to the repository). No script on the page, so the Content-Security-Policy
has nothing to allow. Public by default, like the landing page; set
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
the page carries the usual menu, which now has a **Docs** link.
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
needed — in Python, TypeScript, Go and curl: the base URL and key,
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
leads with it, and the landing page and the account page link to it.
- The console's Manual and the portal read one list of documents in one
order (`llmgw.Manual`), so the two cannot drift.
### Administration
- **An audit log.** Every change made through the admin API (`llmgw ctl`,
the console) or the pages under /admin is recorded: who, when, through
what, the fields it was given with passwords, keys and tokens taken out,
and how it ended — refusals included, in words. So are a person's own new
keys, revoked keys and password changes. Read it with `llmgw ctl audit`
(`--since`, `--actor`, `--contains`, `--detail`), `GET /api/v1/audit`, or
the console's **Audit** view. It moves with the store.
- **Staff roles** besides administrator: `viewer` looks at every
administration page and changes nothing, `operator` also runs the cluster
from them, `billing` sees usage and quotas. Set with `llmgw users role` or
the console's **Staff role…**.
- **Single sign-on through OpenID Connect** (`auth.oidc`). A "Sign in
with …" button sends people to the identity provider and back, with PKCE,
state and nonce. The ID token is verified, and the account is found by
the verified email address, or made when `create_accounts` is on. Groups
give staff roles (`roles`, set at every sign-in) and places in the tree
(`memberships`). `no_passwords` leaves the provider as the only way in,
except for administrators.
- **Provisioning through SCIM 2.0** (`auth.scim.token_file`, `/scim/v2`).
- **Users:** the identity provider makes, suspends and reinstates
accounts — the PATCH forms Entra ID and Okta send are both read.
Deleting a user suspends the account.
- **Groups:** joining and leaving the groups `auth.oidc` maps gives and
takes away staff roles and places in the tree.
- **Key lifecycle.**
- **Lifetime limits:** `auth.max_key_ttl` limits how long any key may
live, and `llmgw keys ttl team:Acme/ml 720h` sets shorter limits for an
organization, team or project. A key is made within the shortest that
applies, and one asked for with no expiry is given that longest — from
the CLI, the profile page and the editor connect page alike.
- **Rotation:** `llmgw keys rotate <id> --overlap 48h` issues a
replacement where the old key was, and the old one stops after the
overlap.
- **Listing:** `keys list --all` shows each key's kind — service (a
project's), member, personal — with service keys first, and when it
expires.
- **Usage export.** `llmgw ctl usage export --from 2026-09-01 --to
2026-10-01 --by project,model` writes CSV by hour, day or month and any of
organization, team, project, user, key and model.
- **Data:** read from the hourly and daily sums, so it reaches back past
the request records' retention.
- **Format:** cost is given in micro-units and units, and names that
look like a spreadsheet formula are quoted.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### Forgetting a person, and requests that stay in the cluster
- **`llmgw users forget`** erases a person for a request to be forgotten:
their usage kept under a random pseudonym so totals stay true, their kept
requests, personal files, batches and limits deleted, their name taken out
of the audit log, the account removed.
- **`llmgw keys residency --project P --in-cluster`**: no model served by an
outside provider for the project's keys, asked for, as a fallback or as a
canary.
- Both are on the console's menus.
### Secrets in Vault
- **Secrets from Vault or the environment.** Every setting naming a secret's
file also takes `env:VARIABLE` or `vault:mount/path#field` (HashiCorp
Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN).
- **The store's own secrets sealed** under `secrets.master_key`: Vault's
transit engine (`vault-transit:KEY`), or a 32-byte key read from a
reference. A store, or a backup of it, then does not hold what engine keys
are derived from. Values from before are sealed as they are next read.
### Scaling by the clock
- **`scaling.schedule`**: windows of the week with their own `min` and
`max` — more replicas through working hours, an on-demand model kept
loaded while people are at work — in `control.timezone`.
- **`scaling.prewarm`** opens a window early, so its replicas have loaded
when it starts.
### Token-level timing, and admission by SLO
- **Per-token metrics:** `llmgw_tpot_seconds`, `llmgw_inter_token_seconds`
and `llmgw_output_tokens_per_second` by model and pool, and TPOT and
tokens a second by tenant.
- **A model's `slo` counts in admission:** past its `ttft_p99` or
`tpot_p99` (the recent p99), it is treated as saturated, so the lower bands
wait. `llmgw_model_slo_seconds` and the `llmgw-slo` alerts make the
promise visible in Prometheus.
### Restoring a backup, and copies off the machine
- **`llmgw server restore ARCHIVE`** puts a backup back: checks it first
(whole, opens with `--identity`, not newer than this llmgw, where the store
goes), says so with `--dry-run`, then moves `/etc/llmgw` aside, restores it
with modes and owners, restores the store (SQLite, or PostgreSQL with
`pg_restore --clean`), and starts the server. A drill is in
docs/install.md.
- **`backup_copy_to`** copies each archive to a directory or over ssh as it
is written, keeping as many there as here; refused without
`backup_recipients`.
- Archives now record each file's owner by name, so a restore on another
machine gives the llmgw user its files back.
### Anthropic's Messages API
- **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat
completions and back, streamed or not: text, images, tools and their
results, the system prompt, stop sequences. Claude Code and Anthropic's
SDKs reach the models here; the key may come in `x-api-key`. Errors come
back in Anthropic's shape.
### More of OpenAI's API
- **Responses:** `text.format` asks for structured output, `input_image`
sends images, and `previous_response_id` continues a conversation the
gateway keeps as a thread (for a project that keeps conversations).
- **Passed through:** `/v1/rerank` and `/v1/score` for vLLM's rerankers, and
`/v1/audio/transcriptions` and `/translations` as the multipart form came,
for an engine serving Whisper.
- **`/v1/moderations`** is answered by the guardrails' judge model.
### Requests kept, and replayed
- **A project can keep its requests and answers** (`llmgw keys requests
--project P --retain-hours N [--sample 0.05]`, or **Keep requests…** in the
console): what the model was sent, after redaction, and what the client
got. Off until asked; never under `no_store`; pruned hourly.
- **`llmgw ctl request ID --body`** shows one; **`--replay MODEL`** sends it
to another model and shows both answers.
### Batches
- **OpenAI's Batch and Files APIs.** `POST /v1/files` (purpose batch),
`POST /v1/batches`, and reading, listing, cancelling and deleting them, so
OpenAI's clients send batches here unchanged.
- **Serving:** each line goes as a request of the key that made the batch,
in the lowest admission band, a few at a time; rate limits are waited
out. Answers and failures are written to files kept 30 days.
- **Accounting:** each line is an ordinary usage record tagged
`batch=<id>`.
- **Restarts:** answered lines are kept as they are answered, and any
gateway on the store carries a batch on under a lease.
- **Limits:** `router.batch` bounds the file, the lines and the
concurrency. Refused to a project under a `no_store` policy.
### New weights tried on a share of the traffic first
- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer
weights of a model's repository and runs them beside it as `MODEL-canary`,
given that share of its requests.
- **Routing:** a session stays on one side; `x-llmgw-variant` pins a
request to `canary` or `stable` and says which served it. Clients cannot
name the canary.
- **Rolled back by itself** when, over 50 of its requests or more, its
error rate is five points above the model's or its time to the first
token half again as long; `canary_rolled_back` is sent to the
notification webhooks.
- **Then:** `--percent` changes its share, `llmgw ctl model promote`
moves the model to it, `--abort` takes it out, and `llmgw ctl model
canaries` compares the two sides. All of it is also on the console's
model menu.
### Requests can be tagged
- **Tags and end users on every usage record.** OpenAI's `metadata` and the
`x-llmgw-tags` header (`feature=search, env=prod`) tag a request; OpenAI's
`user` field names the client's own end user. Both are kept with the
request's usage and shown by `llmgw ctl request`. Tags are not sent to the
engine. A malformed one is refused with 400.
- **Counted by tag, for the tags named.** `router.tag_labels` lists the tags
counted in `llmgw_tagged_requests_total` and `llmgw_tagged_tokens_total`,
the first 100 values of each apart and the rest as `other`.
### A hardened install by default
- **`server init` and the setup screens write a hardened configuration.**
- **Guardrails:** a `baseline` policy on every request. It refuses
secrets in prompts and answers, masks card numbers and records injection
attempts.
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
(about half a second for a short text). It refuses violence, crime,
weapons, child abuse, privacy violations, hate, self-harm and sexual
content. The policy fails closed: a request it cannot read within 5
seconds is refused (503), and `judge_unavailable` goes to the
notification webhooks.
- **Keys:** they live at most 90 days.
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
leaves the judge out and keeps the pattern checks. A `server.yaml`
written earlier is never changed.
- **What is left to you:** the closing text now names the two things
init cannot do — a certificate for the client port, and the keys
backups are encrypted to.
### Notifications
- **Webhooks for what needs a person** (`notify.webhooks`): a model parked,
a node lost, a budget spent, new weights in a model's repository — sent by
the gateway itself, with no Prometheus in between.
- **Formats:** JSON signed with HMAC-SHA256 (`X-Llmgw-Signature`) when
`secret_file` is set, or a Slack-compatible `{"text": …}`.
- **Delivery:** queued, never holding up what raised it; tried three
times; the same event about the same thing at most once an hour, and new
weights once per revision.
- **Checking:** `llmgw ctl notify test`.
- **A judge not answering** (`judge_unavailable`) is one of the events.
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
chosen.
### Guardrails
- **Hooks for guardrails on the request path** (nothing judges yet; B2 adds
the policies). A guard, when one is set, judges a request once it is
authorised, within its limits and has its stored conversation spliced
in, and before it is dispatched; and filters the answer, buffered or
streamed, before the client sees it. A blocked request is refused with 400
and `code: "content_policy_violation"` — the first error with a code — and
costs nothing; a blocked answer ends with `finish_reason: "content_filter"`,
and a stopped stream with that chunk and `[DONE]`, the engine cancelled.
The Responses API reports either as `incomplete` with
`incomplete_details.reason: "content_filter"`. `x-llmgw-guard` and
`x-llmgw-guard-policy` say what was done; the usage row records the
action, policy, category and time taken — never the text.
- **Guardrail policies** (`guard:` in server.yaml; see docs/guardrails.md).
- **What is found:** personal data, each kind validated before it counts
(email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits,
IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt
injection seen through disguises (look-alike letters, invisible
characters, base64/hex/URL encoding); deny patterns.
- **What is done:** flag, redact as `[EMAIL_1]`, or block. A request can
be shaped too: a pinned system prompt, the caller's own system messages
dropped, an output limit, tools stripped or refused.
- **Answers:** judged buffered, or a sentence at a time when streamed,
never releasing half a value; tool calls are checked against allow
lists, a count and the request's own schemas.
- **Modes:** `shadow` records what it would do; `no_store` refuses
conversations, notes and models outside the cluster.
- **Where they apply:** `guard.default` everywhere; others attached with
`llmgw ctl guard attach` to an organization, team, project, user or
key, and narrowed like limits.
- **A judge model on the cluster** (`role: guard` in the catalog, `judge:`
in a guard policy). It reads what patterns cannot — Llama Guard 3,
ShieldGemma or a JSON-answering model — and its verdict flags or blocks
input, output or both.
- **Calls:** past admission and accounting, bounded by a latency budget
(250ms by default), after which the policy's `fail` decides.
- **Caching and streams:** answers are cached for five minutes, and a
streamed answer is read every 600 characters.
- **Isolation:** a judge model is hidden from clients and refused to them.
It is critical by default, and cannot be loaded on demand.
- **Seeing what the guardrails do.**
- **Metrics:** `llmgw_guard_checks_total` (stage, policy, action,
category, mode), latency for checks and for the judge, judge errors,
fail-open, redactions by kind, stopped streams, and blocks per tenant.
- **Alerts:** judged traffic served unjudged, a slow or unreachable
judge, a tenant's blocks spiking, a secret in an answer.
- **Pages and commands:** `/admin/guard` shows counts by place, policy
and category and the recent requests by id; `ctl request` shows the
verdict; the profile page lists the policies in force where the person
works; the console's organization rows have **Guardrails…** and a
column.
- **The chat page** says when a request was refused or an answer stopped
by policy, instead of showing an error.
### Security
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
tests and vet.
- **Releases are signed, and only signed releases are installed.** The build
now carries the release signing key (`internal/release/trusted.pub`); the
control plane upgrading itself, an agent told to upgrade and a node being
added all refuse a release whose checksums are not signed with it. The CI
release job stops rather than publish an unsigned release. Adding a node
no longer has the node fetch the release and compare it with a checksum
from the same server: the control plane fetches it for the node's
architecture, checks the signature and the checksum, and hands it over.
Releases are fetched over https only (loopback excepted), and
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
- **The agent port is not opened without TLS by accident.** A control plane
with no `agent_tls` refused nothing: its agent port was plaintext, on every
interface, and any machine reaching it could register as any node — be
given replicas and so be sent prompts. It now refuses to start unless
`agent_insecure: true` is set, and then listens on loopback unless
`agent_listen` names an address. `server init` writes `agent_tls`, as
before.
- **A model definition cannot run code on its nodes.** A profile's `env` and
`args` reach a process the agent runs as root. The environment was checked
against a short denylist that missed NCCL's and CUDA's plugin and
injection libraries; arguments were not checked at all, so
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
name for a path, plugin, library, address or credential, and llama.cpp's
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
writes or loads files, loads code, or moves what the agent sets (host,
port, key, model). Both are refused where a model is defined and again on
the node before the engine starts.
- **Engines no longer run as root.** The agent starts them as the system
user `llmgw-engine` (in the video and render groups for the GPUs), with a
home and caches of its own under the agent's work directory. `llmgw node
add` makes the user; a node without it keeps running engines as root and
the agent says so in its log. `supervisor.run_as` in agent.yaml names
another user, or `root`.
- **The admin API wants a token.** It listened on loopback with no
authentication, so any process or user on the control plane could add a
model, upgrade the fleet, read the engines' keys or dump the heap through
pprof. Its `/api` and `/debug` paths now demand the token the server
writes, 0600, to its state directory (`/var/lib/llmgw/admin-token`) on
first start; `llmgw ctl`, the console and `node add` read it, so they run
with sudo, and `LLMGW_ADMIN_TOKEN` or `ctl --token-file` give it from
elsewhere. pprof is off unless `debug_pprof: true`. `admin_listen` off
loopback is refused unless `admin_remote: true`. The read-only dashboard
pages, `/metrics` and `/health` are unchanged.
- **The replicated store asks for a password and speaks TLS.** Patroni's
REST API took changes from anyone who reached port 8008 — rewriting the
access rules to `trust`, or moving the leader — and PostgreSQL traffic
between the machines was plaintext (`sslmode=disable`). Patroni now asks
for a password before a change (kept with the cluster's others in
`/root/llmgw-pg`, and on the control plane in
`/etc/llmgw/secrets/patroni-rest` for `server add`); each member has a
certificate from the cluster's authority, PostgreSQL has `ssl` on, every
remote access rule is `hostssl`, and the control planes connect with
`sslmode=verify-full` against the authority's certificate. A new cluster is
set up this way; an existing one is changed in place, by reload, the
cluster serving throughout, by running **Replicate the store** in the
console again — or `sudo --preserve-env=SSH_AUTH_SOCK llmgw server init
--patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node
--no-model` — on the control plane that holds the authority, then on each
other control plane so it reconnects by TLS. etcd is not yet given TLS:
keep ports 2379 and 2380 to the cluster's machines.
- **A capacity fallback respects the key's allowed models, and is charged
at its own price.** When the model asked for had no capacity, the next in
its `fallbacks` served the request even if the key was not allowed it, and
the request was priced as the model asked for. Fallbacks a key may not use
are now skipped, and a request is charged what the model that answered
costs.
- **Suspending an account stops its keys.** Suspension closed the person's
sessions and left every API key they had made working. A key whose owner's
account is not active — suspended, pending, or gone — is now refused as a
revoked key is, within `auth.cache_ttl`, and works again when the account
is reinstated.
- **A join code is a credential only for its hour, and only on its
address.** The setup checked a code's expiry on the joining side alone, so
a code stayed good for as long as the setup ran; it now refuses an expired
one itself. Its listener binds the address the codes name rather than
every interface, and a joined machine is known by the address it connected
from, not the one it claimed.
- **The setup's sudo grant cannot outlive it.** Given a sudo password, the
setup writes a temporary `NOPASSWD` grant and removes it when it
finishes; a setup that crashed or was interrupted left it for good. The
grant now schedules its own removal within the hour as it is written, and
the password is dropped once the grant stands in for it. An address given
to the system ssh can no longer be read as one of its options.
- **Made-up keys cannot fill memory or the database.** Every well-formed key
presented, real or not, was cached for good and cost a database read the
first time. The cache is now bounded, and keys not in it are looked up at
most 20 a second per client address (50 at once); beyond that the request
is answered 429 without a read.
- **Slow or idle connections cannot pile up.** The servers bounded only how
long headers took. Every listener now also bounds how long a connection
may sit idle and how large its headers may be, and reading a request
(5 minutes on the client port, a minute or less elsewhere — never on the
agents' long-lived streams); the client port holds at most
`max_connections` (10,000) at once, and a message from an agent at most
16 MB.
- **The sign-in and signup forms resist scripts.** Ten wrong passwords from
anywhere locked an address's owner out for the window; attempts are now
counted per address and client, so somebody guessing elsewhere cannot lock
the owner out, with a looser count per address alone so guesses spread
over many clients are still bounded. The sign-in form carries a token its
cookie matches, so another site cannot sign a visitor into an account of
its choosing. Posting either form is rationed per client address (sign-in
30 a minute, signup 5), and a password hash that cannot get a turn within
ten seconds answers "busy" instead of holding the request. `server init`
now writes `signup_open: false`.
- **Forwarding headers are believed only from the proxies named.**
`X-Forwarded-Host` and `-Proto` were taken from anyone, and decide the
gateway address written into the `/setup` scripts clients pipe to a
shell; behind a caching proxy one request could poison them for others.
They — and `X-Forwarded-For`, now used for the per-address limits — are
believed only from `router.trusted_proxies` (addresses or CIDR prefixes).
**Behind a reverse proxy, set it**, or the pages and scripts name the
address the proxy reached and every client counts as the proxy. `/setup`
scripts are served `no-store`.
- **Removing a node revokes its certificate.** `ctl node remove` only
dropped the node from the inventory; its certificate, valid for a year,
still connected and was given work. Now the removal is recorded in the
store and the agent port refuses any certificate issued for that node
before it — onboarding the node again issues one that works. A mirror
serving its management API with `--tls-ca` now requires `--api-client`
(the control plane's name), since the same authority signs every node.
- **Engines listen on the management address only**, the one the control
plane reaches them on, instead of every address the node has — a public
one included. (When that address is a name, or not one of the node's,
they still listen on all.) **SGLang's key is off its command line**,
where every user on the node could read it: it reaches the engine in its
environment, and a short launcher hands it to SGLang inside the process.
Both take effect as engines restart under an upgraded agent.
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
recipients or SSH public keys) has `llmgw server backup` encrypt each
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
and every secret in the store, and is meant to be copied off the machine.
Open one with `age -d -i <key>`.
- **PyTorch's index gives torch and nothing else.** vLLM's CUDA 12.9 build
was resolved with PyTorch's index beside PyPI and `unsafe-best-match`, so
either index could supply any package — how torchtriton was hijacked in
2022. Now that index is explicit, named only for torch, torchvision and
torchaudio; every other package comes from PyPI alone, and the lock is
PEP 751's (`pylock.llmgw.toml`), pinning each file's address as well as
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
- **The control plane's own fetches stay off link-local addresses.**
Checking a model's source and reading a GGUF's header went wherever the
URL said, cloud instance metadata (169.254.169.254) included. They now
refuse link-local, multicast and unspecified addresses, while still
reaching the cluster's network, where the mirror is. The mirror's fetches,
which refuse private addresses too, now also see through IPv6 addresses
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
the target's addresses are checked before the request is handed to it.
- **A CPU-only node reporting a problem no longer panics the agent
handler**, and **thread ids are bounded** to 128 letters, digits and
`- _ . :`. A thread is readable by every key of its project that knows
the id; the docs now say to use unguessable ids.
- **`/v1/models` lists only what the key may use**, rather than every
model the gateway serves to anyone.
- **Portal sessions end after a day unused** (still at most 14 days in
all), and expired or idle sessions and used or expired key claims are
deleted hourly — they were refused, but never removed.
- **Every page carries a Content-Security-Policy**: scripts only from the
gateway itself and the page's own inline ones, each named by its hash —
no inline handlers, no eval, nothing a model's reply or a model card
could slip into a page. The few `onchange`/`onsubmit` attributes moved
to `/_static/pages.js`, htmx runs with `allowEval: false`, and the live
pages' "only while visible" refresh no longer needs an evaluated filter.
Pages now carry their headers even when answered with an error status.
- **KaTeX 0.16.47** (from 0.16.11), past the advisories fixed in 0.16.21.
- **Smaller hardening.** The mirror no longer serves its bookkeeping files
(a file's `.source` URL and `.sha256`) on `:8081`. A GGUF header nesting
arrays more than 4 deep is refused rather than recursed into. Serve and
update jobs keep the last 100 finished, and the mirror forgets finished
downloads after a week, instead of remembering every one until restart.
- **A firewall for llmgw's internal ports.** `server init` (and the store
members it sets up with `--patroni`), `node add`, `server add` and
`mirror init` now keep the machine's internal ports — the agents' stream,
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
cluster's own networks, in an nftables table of llmgw's own loaded by
`llmgw-firewall.service`. Only those ports are touched: the table accepts
everything else, so other services on the machine and any firewall it
already has are left as they were. Extra networks go in
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
node lacks it.
- **etcd and Patroni's API speak TLS.** etcd's peers and clients use
certificates from the cluster's authority, and a client must present one;
Patroni reaches it with its member's certificate and serves its API over
TLS. `server init --patroni`, run again, moves an existing cluster: with
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
control plane asks the API over HTTPS, verified, and over HTTP only where a
member has not moved yet.
- **A cluster keeping its access rules per member got no `hostssl`.** The
TLS step looked only at the shared configuration, and on a cluster set up
before the rules moved there said the rules were `hostssl` while
plaintext was still accepted. Each member's own rules are now made
`hostssl` too.
### Fixed
- **An engine that died after an agent restart went unnoticed.** The watcher
that reports a ready engine dying was tied to the stream it was started
on, so after a reconnect nothing reported it, and an engine adopted from
the previous agent had no watcher at all: its replica stayed "ready" and
was routed to. Every connection now watches every ready shard, started or
adopted. Found on the PoC: a killed engine on node06 stayed ready for
minutes.
- `server init` rewriting the store's address no longer leaves
`/etc/llmgw/secrets` unreadable to the service when it was made 0700 for
the service user, which kept the gateway from starting.
### Everything that can be changed on a running cluster, from the console
`sudo llmgw` only watched the cluster and edited the configuration files;
changing a model's replicas still meant `llmgw ctl`. Now Enter on a row opens
what can be done to it, as forms and questions:
- **Models**: replicas (min, max, idle timeout), class, owner, resources, LoRA
adapters, restart, retry when parked, why it is placed where it is, remove;
A adds one from a definition file.
- **Nodes**: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its
agent, delete model files nothing uses, remove.
- **Replicas**: restart, replaced or in place.
- **Quotas** (new): capacity quotas, set and removed.
- **Mirror** (new): what it holds and which models use it; download into it,
remove from it.
- **People** (new): accounts — approve, suspend, reinstate, the cluster
dashboard, memberships, limits, a new password, delete, create; API keys —
issue (shown once), limits, revoke; organizations, teams and projects — add,
rename, move, remove, limits for the level and for each member, capacity
quotas, how long conversations and agent notes are kept.
- **Serve a model** (new view): GGUF models on Hugging Face — trending, most
downloaded, most liked or newest, or searched with / — then one of the
chosen repository's files, a name and a replica count, and it is fetched,
checked, added and started. The registry is mirror.registry's when set.
- **Do**: upgrade the cluster, restart every replica, back up now.
The people side runs the same commands as `llmgw users`, `keys` and `org`,
against the same store, so their checks and messages are the ones shown. A
sidebar taller than the terminal now scrolls with the current entry.
### New weights in a model's repository, found and moved to
A repository on Hugging Face can publish new weights under the same file
names — a fixed chat template, a better quantisation — and nothing said so. A
model served from the registry now records its origin: the repository, the
commit, and each file's checksum there; a model served before is worked out
from the mirror's record of where it fetched each file. Every
`mirror.update_check` (six hours by default; 0 turns it off) the control
plane asks whether those files changed. A new commit that leaves them alone
is not an update.
- `llmgw ctl model updates [--check]` lists what each repository has.
- `llmgw ctl model update MODEL` fetches the new files into the mirror beside
the old ones, checks them against the registry's checksums and the nodes'
llama.cpp, points the catalog at them, replaces the replicas one at a
time and asks one to say hello.
- `llmgw ctl model rollback MODEL` goes back to the files the update
replaced, which stay on the mirror.
- The console shows UPDATE on Models and offers all three on a model's row.
A model from `server.yaml` is reported but not changed: its source is edited
there.
### The monitor: the cluster drawn like btop
The console has a **Monitor** under Watch: CPU, memory, GPU busy and memory,
requests, tokens, time to first token, in flight and queued, network and
disk, as Braille charts filled and coloured by height, over 5 minutes, an
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
each card's busy, memory, heat and power, and each model's requests, as
sparklines. A node's and a model's own pages carry their charts too.
The history is read from the Prometheus in `monitoring.prometheus`, through
the admin API (`POST /api/v1/metrics/range`); without one, the charts draw
what the console has seen since it opened.
The control plane now lists the cluster's machines for Prometheus' service
discovery, at `/prometheus/targets` on its metrics port — nodes, itself and
the store's members, each labelled with its llmgw id — and `llmgw node add`
installs `prometheus-node-exporter` on the machine (`--no-node-exporter`
leaves it off). `deploy/monitoring/prometheus.yml` scrapes them as the
`nodes` job.
### Serving and updating run in the background
Serving a model or moving one to new weights from the console starts a job
on the control plane and leaves the console free at once, instead of
closing it to follow a download that can take an hour. A new **Jobs** view
shows each job's progress and steps, and the sidebar how many are running.
`llmgw ctl model serve --detach` and `llmgw ctl model update --detach` do the
same; `llmgw ctl jobs` lists the jobs and `llmgw ctl jobs ID` follows one.
### The file that suits the hardware, and how to spread it
`llmgw ctl model plan REPO` now plans the files the repository actually has
against the fleet as it is, before anything is downloaded: the model's shape
comes from one GGUF file's header, read from the registry, or from its
config.json. Every file is listed with the best way it runs here — one card;
several of a node's cards (llama.cpp splits by layer, vLLM by tensor
parallelism, dividing the KV heads); several nodes' cards over one fabric,
pipeline-parallel between them (vLLM); or a CPU node's memory — what each
card or node would hold and whether that hardware is free now. Best first:
on a GPU, the fewest cards, then the better quantisation up to eight bits;
whether the cards are free now only breaks a tie. Busy is not unfit: a model
for cards that are in use is added and starts when there is room, the job
saying so rather than failing. `--precision` keeps the old plan in the abstract.
`llmgw ctl model serve` without `--quant` takes the plan's file, and without
`--cpu` or `--gpu-class`/`--gpus`/`--nodes` the plan's way of running it —
a GGUF model on GPUs is now given a GPU profile rather than always the CPU.
`--fallbacks` writes the next best ways too, lower in priority, so placement
uses one of them when the first is busy. The console's Serve a model shows
the plan for each file, marks the best, and asks for the way and the
fallback.
### Models of several files: vision projectors, original weights
- A vision model's projector (`mmproj`) is fetched beside its GGUF weights
and loaded with them: `llmgw ctl model serve REPO --projector FILE`, or the
choice offered by the console. The model is marked as taking images. Nodes
load it once their agent is this version or later.
- `--format safetensors` serves a repository's original weights, with their
config.json, tokenizer and shard index, on vLLM across `--tp` GPUs of
`--gpu-class` (the fleet's one class when there is one). In the console,
**F** on Serve a model switches to them.
- The mirror keeps where it fetched each file beside it, and reports that
and the checksum after it restarts. The console's Mirror view shows one
row per version of a model, with its files on Enter.
- An update watches every file of a model, the projector and the tokenizer
included; one whose files the mirror cannot trace is found on the
registry by name, commit and size, or given with `llmgw ctl model origin`.
- `llmgw_model_update_available` is published, and `ModelUpdateAvailable`
alerts on it.
### The manual, in the binary
`llmgw docs` reads every document — installation, operation, the
configuration reference, the README, the changes — and a page for every
command, built from the commands' own help, like `info`: topics on the left,
T for a page's contents, / to find, n for the next match. Piped, it prints
the page. In the console it is **Help › Manual**, and **?** opens the new
page about the console itself from any screen.
## v0.11.1 — 2026-10-03
### The console's overview no longer spills over the screen
On a cluster with a GPU, `sudo llmgw` printed the overview's GPU table onto the
terminal under the console every three seconds, scrolling across the full
screen. The summary's tables were written to standard output whatever they
were given; they are now written where the summary is, which for the console
is its panel. The test cluster the console was tried on had no GPU, which is
why it was not seen there.
## v0.11.0 — 2026-10-03
A minor version: the store can be replicated and the control plane doubled,
both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
usage is read from sums kept as it is written. Released without candidates.
**Upgrading.** Migrations 0015 (usage sums, filled from the request table
already recorded — once, at the first start) and 0016 (control planes) run by
themselves. An agent older than this ignores the list of control planes and
keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
The GPU and fabric work below began from the field report in issue #2, after
a deployment on GPU nodes.
### GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.
- **New series**: `llmgw_gpu_memory_utilization_percent`,
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
sample.
- **The node page** shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
- **The cluster page** charts the fleet's GPUs by class: average utilisation
and memory in use.
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
- **The cluster page's GPU map**: every card a square, a row per node,
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
with what holds it and its figures under the pointer; free cards dashed.
- **Cards held and doing nothing**: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
- **What each model gets from its cards**, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
- **What wears a card out**, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
`GPUNVLinkErrorsClimbing`.
- **GPU time by owner** on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### Usage is read from sums kept as it is written
Every chart and total — the usage tab, quotas, the dashboard, a person's own
page, and budgets restored at start — summed the request table again on each
view: a row per request, scanned whole. Usage is now also summed as it is
written, in the same transaction, into five-minute slices, hours and days per
key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
0015 fills them from what is already recorded).
- **Charts** read the coarsest of the three their slices are made of: a day in
20-minute slices from the five-minute table, a week or a month from hours.
- **Totals and budgets** read the rows up to the next five-minute slice, then
slices, hours and days — exact, not rounded to a slice. On three million
requests over thirty days, a month's total per person went from 705 ms to
18 ms on PostgreSQL.
- **Latency and time to first token** are kept as histograms in the sums, so a
percentile per team or model can be read after the rows are gone.
- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
that, hourly. Totals, charts and budgets are unchanged by it — they read the
sums — but `ctl request` finds nothing older. Unset keeps every record, as
before.
### Alerts per tenant and per budget
- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
tokens and time to first token per tenant. The tenant is the level
`router.tenant_label` names — `team` by default, or `org`, `project`, or
`none` — named by its path, "acme/platform". One level rather than every
project, so a cluster of thousands of projects does not become a Prometheus
of millions of series.
- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
over the budget for every organization, team, project and person given one,
refreshed every minute.
- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
`BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
`TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
`TenantSlowFirstToken`.
- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
(uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
refused by a limit, first-token p95, every budget's share spent, and who used
most in the last day — with a tenant chooser at the top.
### How each team's requests fared, not only what they cost
- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
per organization, team, project, person or model, read from the histograms
kept with the usage sums: on the usage tab's table, and in a branch's or
person's panel on the people page, beside the error rate and requests
refused by a limit.
- **Refused requests are counted against their owner.** A request a rate,
token or budget limit refused was counted only in Prometheus, by model; a
team running into its limit looked like a team that had gone quiet. It is
now recorded as a 429 against its key, person and project — no tokens, no
cost — and shown as "Limited".
- **Usage by model** is a grouping of the usage tab, by the model that
answered.
### The console watches the cluster and holds every setting
`sudo llmgw` on a control plane is a console in three groups down the left:
- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
memory, GPUs free, disk and agent version, and a node's detail — load,
engines, each card's busy, memory, temperature and power; every GPU in the
fleet and which model holds it; models, with a model's profiles, warnings
and replicas; replicas and where they run; the last day's use by team,
project or organization; and a node's agent log.
- **Settings**: every key of `server.yaml`, block by block, with what applies
when the file sets nothing and what each key is for, from its comment in
the code. Enter changes a key or opens a block; lists and maps of blocks
(endpoints, models, prices) take entries added and removed. A save is read
by the same loader the service uses first — nothing is written that would
not load — keeps the file's comments, leaves the old one beside it, and
offers to restart the service.
- **Do**: add a node, replicate the store, leave.
On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
The keys come from the configuration structs themselves, through the
generator that writes `docs/reference.md`, so a setting added to the code is on
the screen without anyone listing it, and a test fails if the two drift.
That generator told blocks apart by name alone, so `tlsconf.Config`,
`engines.Config` and `supervisor.Config` were each taken for the agent's
`Config` and dropped: the reference had no TLS, engine or supervisor settings
at all. It now has 37 blocks, six more than before.
### A second control plane with `llmgw server add`, and the screens as windows
- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
control plane, makes another machine a standby: this llmgw copied over, a
certificate from the cluster's authority, the same configuration and store
address, the machine let into the store (for a store under Patroni, one
change through Patroni's API), and the service started. Every node is told
of it within half a minute and fails over to it untouched. `--vip` keeps a
floating address on whichever control plane leads, with keepalived on both.
Tried on new machines: switching the leader off, the standby led after 14
seconds and the floating address answered from it after 23.
- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
ones in their configuration.
- **A node notices a dead control plane within a quarter of a minute.** A
leader whose machine stopped dead left its nodes' streams open until the
kernel gave up retransmitting, about fifteen minutes; the agent now pings
the stream after ten quiet seconds, and bounds dialling too.
- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
open like `/ready` — what keepalived follows.
- **The store's access rules live in Patroni's shared configuration** for a
cluster set up by `server init --patroni`, so another control plane is let
in without touching each member.
- **The screens are windows**: full screen, a title bar, the steps or actions
down the left, the content in a bordered panel, the keys along the bottom,
questions and errors as dialogs over it — the setup, the console, adding a
node and one-machine init alike. Keyboard only.
### Machines nothing may log into: `llmgw join`
For sites that allow no ssh into a machine, the setup screens offer a fourth
way to reach one: a join code. The setup listens on port 7444 while it waits
and shows a code for each such machine, with the commands that fetch llmgw
onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
checks by the certificate fingerprint in the code that it reached the setup
that made it, admits itself with the code's secret, and runs what the setup
sends, printing each step, until the setup says it is done. A code is good
for one machine and an hour, and is under 90 characters so it can be copied
whole from one screen into another.
To the setup a joined machine is one more way of running a script there:
the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
machine joining by code and another logged into with a password, in one
setup.
### Setting up from nothing in screens: `sudo llmgw`
Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
setup on a machine with nothing installed, and a console on a control plane.
In a pipe or a script it prints its help, as before.
- **Setup** asks one machine or three. For three — this one as the control
plane and etcd's third vote, two more for the store under Patroni — it asks
how to reach the other two: the ssh agent, a key file, or a password, and a
sudo password where sudo asks for one. Passwords stay in memory for the run;
a sudo password grants the user passwordless sudo until the install
finishes, and the grant is removed after, failed or not.
- **Every machine is checked before anything is installed**: login, system
(Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
role, clock against this one, NTP, the ports its role needs, internet, and
whether it reaches the other machines. Failures stop it, with what to do;
warnings let it go on.
- **Then the plan, and the command line that does the same** from a script,
then the install with its progress, and the administrator's password.
- **The console** shows `ctl status`, refreshed every five seconds, and offers
to add a node or, for a store on one machine, to replicate it.
- Machines are reached with Go's ssh client rather than the system's, so a
password can be used; a host key is recorded on first sight and refused if it
changes.
Tried end to end on three new machines whose user logs in and sudoes only with
a password.
### A store that survives losing its machine
`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
under Patroni on two machines, its leader lock in a three-member etcd, and
llmgw given every member with `target_session_attrs=read-write`. Measured: a
planned switchover cost one failed lease renewal; switching off the primary's
machine promoted the standby in 24 seconds with the gateway serving
throughout. Example configurations are beside it.
**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
third vote — the role and database, and the store already in use moved in.
Every step looks first; run against a cluster that is up, it changes nothing.
Tried from scratch on three new machines (two minutes, then a failover by
switching the primary off) and against the reference cluster (nothing
touched).
**Watching it**: for a store of several hosts the control plane publishes
`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
failed: the flag was sent as a bool to an integer column, which SQLite takes
and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
by being copied from SQLite, which is why nobody had met it.
`llmgw server backup` reads a store address of several hosts. It took the
list for one host name and reached the database only by falling through to
the second; it now hands libpq the list, and dumps from a standby when there
is one.
### People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
page and its spend on the usage page, each linking to the others. They are now
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
each at the address it had before, so links and bookmarks still land. The menu
has one entry for them, People.
Choosing an organization, team, project or person in the tree, or a person in
the account list, opens a panel beside it with everything about that one
thing: its own limits and what it is held to from above, spend against its
budget, its per-member limit, its capacity quota, and its requests, tokens
and spend over the last week, drawn. Its limits can be set or changed there,
and the answer comes back to the same panel.
The administrator's menu no longer has an Organizations entry beside it: `/org`
and `/org/<id>` send an administrator to that organization's panel here. The
page stays as it was for an organization's managers, who cannot see this one.
### The chat page says whether a GPU or a CPU answered
A model can have replicas on a card and on CPUs at once, and the two answer
at very different speeds; the chat page gave no way to tell which one a reply
came from. The picker now says where each model runs — `GPU`, `CPU` or
`GPU + CPU` — and the line under each reply names what answered it: `GPU
6000-24g` or `CPU`, beside its tokens a second.
Every answer through the gateway carries the same as `x-llmgw-hardware`:
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
none.
### A card shared with something else is judged every heartbeat
A card with another process on it when its agent started was kept out of
placement, and stayed out — "5014 MiB already allocated by another process" —
after that process let go of it, until the agent was restarted. Found on a
node shared with ComfyUI. The other way round, a tenant arriving after
registration was never seen, and a card the agent's own engine held when the
agent restarted read as someone else's.
The agent now reads every card's memory per process on each heartbeat,
leaves out what its own engines hold, and sends the rest; the control plane
keeps a card out of placement on that reading, and lets it back in when it
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
agent too old to send it keeps the reading from registration.
### A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
where the loader looks), came up on the CPU, answered at CPU speed, and was
reported ready while its GPU sat reserved and idle. Three checks now stop it:
- **Installing** the CUDA build asks it which devices it sees
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
own `ldd` could not say it: the backend is opened at run time.
- **After the upgrade**, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
- **Starting** a shard on GPUs is ready only when its process holds memory on
each of them (`nvidia-smi --query-compute-apps`, its process group
included). One that does not is stopped and reported failed: "placed on GPU
0 but computing on the CPU", with the devices the engine lists. This covers
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
when the engine sees no GPU, and otherwise prints the devices it computes on.
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
node with two kinds of GPU, a shard could get a different card from the one it
was given.
### A model of many files, mirrored whole for vLLM
vLLM loads a directory — shards, config.json, the tokenizer — and does not
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
shards alone, a node could be handed a version still downloading or missing a
shard, and a repository's page given as a model's source was fetched as one
file of HTML.
- **The explorer fetches a safetensors set with what the engine reads beside
it**: config.json, generation_config.json, the tokenizer files, the shard
index, the quantisation config. Never code: a `.py` would be the engine
running what the repository ships, which is to be chosen on purpose.
- **A version is offered to nodes only whole.** The mirror refuses its
manifest (409, with what is missing) while a file is still downloading, or
while a shard its `model.safetensors.index.json` names, or a part of a
split (`-00002-of-00004`), is absent. A node reports "not complete on the
mirror yet" and is tried again.
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
- **`llmgw mirror add` takes several URLs** into one version.
- **A Hugging Face repository page given as a source** is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
such a model, as before.
### Signing up with an address that has an account
On PostgreSQL, signing up with an address already registered showed
"something went wrong; try again": the store recognised a duplicate by the
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
code 23505. It now knows both, and the address is answered as on SQLite —
with the same page as a new sign-up, so the page cannot be used to find out
who has an account. That page now also says what to do if the address had
one: sign in, or ask an operator to reset the password.
An error the page does not explain says the account was not created and
gives a reference (`err-…`), logged beside the error, for an operator to find
it; the database's own words stay in the log.
### An adapter no fabric uses does not keep a node out
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
added for single-node inference, with no fabric given, that refused it for
RDMA nobody had asked for.
- Those are failures only for ports a declared fabric uses, and only when a
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
- Which fabric an undeclared InfiniBand port is on, and whether it has a
subnet manager LID, are notes rather than warnings; `node add` still warns
once that adapters are up with no fabric given.
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
written afterwards.
### A restart with no room beside it
`ctl restart` places each replacement beside the replica it replaces. With
no room for it — one GPU, every card that fits held by the replica itself —
the old replica was stopped only after the drain grace period, five minutes
of "restart replacing" with nothing saying why, which read as a restart that
never happened; scaling to zero and back was the way out. And a replacement
that fit nowhere even with the old replica gone was stopped for all the same,
leaving the model with nothing. Now:
- **The replica is the only thing in the way**, found by placing the
replacement on a copy of the inventory without it: stopped at once, the
replacement taking its place.
- **The replacement fits nowhere even then**: the restart is given up and the
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
- **Room being made** — a lower class preempted, an idle on-demand replica
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
### Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.
| | llama.cpp | vLLM |
|---|---|---|
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
| No NVIDIA GPU | the CPU build | none |
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
- **The install computes on the GPU before it counts**: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
- **The build can still be chosen per node** on the cluster page (auto, CUDA
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
at once.
- **Whether an engine computes on its GPUs is read from every process
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
`CUDA_VISIBLE_DEVICES`.
### Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly: `node add` refused a
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
and `--ignore-checks` passed over every other problem with it.
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
node joins without it and serves what it does run. `node add` checks the
vLLM the control plane installs, not the binary's own default.
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
- **The requirements follow the version**, as the table above sets out; the
threshold had been CUDA 13's for every version.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
### Serving from the mirror says where a model will run
The serve form put every GGUF model on the CPU: it offered no choice, and its
"Profile name" field, pre-filled "cpu", read as though renaming it would move
the model to a GPU. Placement is decided by `gpu_class`, which the form never
set. Now:
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
cards is chosen.
- **GPUs per replica** sets how many cards a replica takes on one node
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
how many a llama.cpp replica serves at once.
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
typed.
- A line under the form says what the model will run as — engine, CPU or how
many of which GPU, memory, context pool and how it is divided — and turns
red when the CPU is chosen with GPUs free. The message after adding repeats
it.
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
anyway** is ticked.
### llama.cpp on several GPUs of one node
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
placement reserves N cards of its `gpu_class` on one node, the engine sees
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
(`--split-mode layer`, in proportion to each card's free memory, which for
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card. It is now read as
`gpus_per_node`, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
`nodes` above one, is refused.
## v0.10.6 — 2026-10-01
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
format check and has no binaries.
Tried on the PoC cluster before release: LoRA adapters asked for by name
through the gateway, two on one model, each answering as trained and the model
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
requests on the preferred replica, 4 on the other); and a bronze request
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
live: a model served from an external provider, which needs a provider key.
### Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks as `llmgw ctl quota`.
- **A model's card** on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said.
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
### LoRA adapters
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (`--lora-scaled FILE:0`); a request that names an
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in `/v1/models`, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.
### Finding and attaching adapters from the mirror page
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
held model's card, *find adapters on Hugging Face* — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original.
- **Opening a repository** reads the first bytes of each GGUF file and marks it
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says.
- **Held adapter files** are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
lists and changes them from the command line, through
`POST /api/v1/models/{name}/adapters`, with the same checks.
### Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
model, api_key_file}` and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model's `fallbacks`, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md). `ctl models` and the cluster page mark it
external.
### Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.
### Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it. `admission.queue_capacity_total` now bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted as `displaced` in the admission rejection metric.
### Releases can be signed
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
public key in `internal/release/trusted.pub`, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.
### Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
## v0.10.5 — 2026-09-30
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
format check and has no binaries.
After upgrading:
- **Declare what CPU models need.** A CPU model added from now on must say its
memory; one stored before this still loads and is flagged in `ctl models`
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
are then reserved per replica and become the engine's thread count, for
replicas started after the change: `llmgw ctl restart --model MODEL`.
- **Upgrade the mirror on its own host**, now with one command:
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
mirror older than this). `llmgw ctl upgrade` says so at the end.
### CPU cores are reserved, and memory has to be declared
The first step of resource allocation for larger clusters: what a replica
holds of a node is reserved, not assumed.
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
node holds as many CPU replicas as it has cores for, and two on one node no
longer spin against each other. Agents report their CPU count; a node that
does not, and a profile that declares no cores, constrain nothing.
- **Memory is required of new CPU models.** A model with no GPU is all host
memory, and one declaring none looked to placement as if it fitted
everywhere, any number of times. Adding one without `min_memory_gb` is now
refused. Models stored before this still load, and say what is missing.
- **Declared against measured.** Each model shows what one replica reserves
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
cluster page — with a warning when nothing is declared or when it uses more
than a tenth over what it reserves.
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
`ctl model serve` and the mirror's serve form fill in cores — half the
smallest node the model fits on — beside the memory they already estimate.
Running replicas keep their old figures until restarted.
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
rc.7 the first was briefly `llmgw ctl resources`.
### Capacity classes, and preemption
A model has a class — `critical`, `standard` or `opportunistic` — saying what
its capacity is worth against another's when there is not enough for both.
Unset, an on-demand model is opportunistic and every other model standard,
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
sets it.
When a model cannot be placed, replicas are stopped to make room for it, by
these rules:
- A lower class gives way to a higher one.
- Within a class, only replicas above a model's minimum — room it grew into
under load — and only for a model still below its own minimum.
- A critical model's minimum is never taken, nor a replica that has only just
loaded, nor one spanning several nodes.
- Idle on-demand replicas go first, then what lower classes grew into, then
the same within the class, then lower classes' minimums; the longest unused
first within each, and of equally idle ones the smallest.
- The set is tried on a copy of the inventory first, at most three replicas,
and nothing is stopped unless it makes the room. Of a set that does, each
replica whose stopping turns out not to be needed is left running, so an
idle replica tried first on the wrong node is not stopped with the one that
made the room.
A replica stopped this way is drained and given 30 seconds to finish what it
is serving. Its model is held back for up to five minutes, or until the model
it gave way to has its replicas, so it does not take the room straight back.
`ctl models` shows each model's class and "gave room to X" while held; the
cluster page shows the class and how many replicas were lost; the metric is
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
### Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
one with no owner is shared and counts against nobody.
- **A guarantee is kept.** A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one.
- **Beyond it is borrowing.** Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum.
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
by scope and resource; alert: CapacityAtLimit.
### Sized by what a replica was measured to serve
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
replica served while meeting its SLO, and its throughput there — now drives
decisions rather than only being printed:
- **Scaling up** when requests per replica pass what was measured, not only
when the engines' queues fill: an engine takes requests into its slots long
after latency has gone.
- **Scaling down** only when the replicas left would stay under 70% of what
they were measured to hold, so a replica taken back does not put the rest
straight over.
- **Choosing a profile.** Among profiles of equal priority, the one measured
to serve the most per GPU — or per core — is tried first. Priority set by an
operator still wins.
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
replicas that takes, how many run, and whether the rest fit, with the reason
when they do not; `--apply` sets it if they do. Headroom reports what one
more replica would serve.
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.
### Upgrading a machine the control plane does not reach
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
the mirror's host above all, whose service runs unprivileged and cannot replace
its own binary as an agent does. The release's checksums are checked, the old
binary kept, the unit restarted and watched until it stays up.
### Nodes report their own heat and power
The first step of facility-aware placement (docs/facility-aware-placement.md),
observe-only: nothing is decided on these yet. Every agent reads, at most every
ten seconds, and sends with its heartbeat:
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
it, power draw and the enforced limit, and what is holding the clocks down —
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
at registration; it is now current.
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
the package temperature, and the board's inlet air sensor where it has one.
- **CPU power** from RAPL's energy counters.
A sensor a machine does not have is not reported, never shown as zero. New
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
table of temperature, power against its limit, and what is throttling it.
### The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
explorer and the fetch form, one under another, with every visit waiting on
Hugging Face for a list it might not show. It is now three tabs — Held (with
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
URL — each with its own address. Only the tab shown does its work, and only
Held reloads while something downloads. A download started from the explorer
links to Held, where its progress is; a fetch the mirror refused comes back on
the form it was sent from.
### A held model opens to where it runs, and how many
Clicking a model under Held opens a card beneath it: each replica with its
node, state, age and the CPU and memory its engine uses, and how many more
replicas the cluster has room for now, on which nodes, and why not more. The
replicas it always runs and the most it grows to under load are changed there.
Room is found by trying the scheduler itself on a copy of the inventory — the
same decisions the next passes would make, not an estimate beside them — and a
number of always-running replicas it cannot place is refused with the reason,
rather than accepted and left short. A ceiling above the room is taken, and
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
API.
Each row also says how its model runs without being opened: replicas ready of
those it should have, how many are still starting, and the nodes they are on —
in amber when short, in red when parked or when it does not fit. It is one line
per model beside its mode, and nodes drop the part of the name every node
shares (node05 for llmgw-node05, the full name on hover).
### One line per record, in every table
The same rule across the admin pages. Headers do not wrap; a record is one
line; free text is cut to its column with the whole of it on hover rather than
given a row of its own; a row across the whole table is only a note or an open
form. On the cluster page that means a node is one line (its address and logs
beside its name, not under it), a model's description is a column rather than a
second row, node names drop the prefix they all share, and the GPUs, classes,
fabrics and fallbacks columns appear only when some row has something in them —
on a CPU-only cluster they were three columns of dashes pushing the table past
the page. On People, a person in many teams shows two and a count, the rest on
hover.
## v0.10.4 — 2026-09-30
Published first as release candidates v0.10.4-rc.1 to rc.9.
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
thread count below only when they start.
### CPU replicas no longer fight over the CPUs
llama.cpp computes on every CPU of the machine unless told otherwise, and the
agent did not tell it. Two busy CPU replicas on one node then spin against each
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
0.8 each with both busy, in two rounds of the same test. The agent now starts a
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
node — all of them suits a node that only ever runs one CPU replica — and a
profile's own `--threads` still wins. Measured on VMs whose host is
oversubscribed; the proportions, not the numbers, are the point.
### Tracing
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
request is traced over OTLP/HTTP: a span for the request, carrying its
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
it. A caller's own `traceparent` is continued, and the engine is sent one for
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
without a decision. Off, it costs nothing.
### Restarting replicas
`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
ones. Each replacement is started beside the replica it replaces, which keeps
serving until the new one is ready; replicas of one model go one at a time.
Where no second copy fits anywhere, the old replica is stopped after the drain
grace period and the new one starts in its place. This is how running engines
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
version.
### A model that keeps failing is parked, and everybody is told
A model whose replicas failed at every start was retried forever: kev-0.5b,
whose architecture the nodes' llama.cpp did not know, was started 155 times in
eight hours. After five failures in a row a model is now parked — not placed
again until its definition changes or `llmgw ctl retry MODEL` is run — and
`ctl status`, `ctl models` and the admin API say so with the last error.
A request for a model that will not come up — parked, or fitting on no node —
is answered at once with the reason, instead of "the model is loading; retry
shortly", which a client would retry forever. Each request still makes the
control plane try placing it afresh, so a model that fits once room is made is
not refused on an old answer. "no placement for model" is logged when its
reason changes or every ten minutes, not on every pass: it was 250 identical
lines in two days for one model.
### Metrics and alerts for what goes wrong
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
`llmgw_model_replicas_ready` is now published at 0 for every model that should
be running; it was published only while a replica was ready, so
ModelHasNothingServing could never fire. New alerts: ModelParked,
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
`ctl status` counts a model against the replicas it should have now rather
than its minimum, so an on-demand model that was asked for and fitted nowhere
no longer reads as satisfied.
### Backups
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
secrets, and the certificate authority's key — to one archive readable by root
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
nightly. The control plane publishes the newest archive's time as
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
more than 26 hours old or there has never been one. docs/install.md now says how
to restore, including why running engines survive it: the engine-key secret
comes back with the store.
### Replicas follow load between min and max
A model's `max` was a ceiling nothing reached: every model ran exactly its
minimum however long its queue. Now, when requests are waiting at the engines —
every slot of every replica busy — at one or more per replica for a minute, one
replica is added, up to `max`; ten minutes with nothing waiting takes one of
those away, down to `min`. One step at a time, and not while the previous one
is still starting, so a spike that is over before a replica loads does not
leave several behind. Each decision is logged with the load that made it.
`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
### A model from the registry in one command
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
mirror fetch it and checks what arrived against the registry's published
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
header, adds it to the catalog, waits for a replica — stopping early with the
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
is shown as it happens. It runs in the control plane as a job, so a download of
hours does not depend on a terminal staying open. `--on-demand` adds it without
starting a replica.
### Upgrading from one command
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
machine and then every node in turn. Each agent is told the version over its
existing stream, fetches the release for its own platform, checks it against
the release's checksums, keeps the old binary beside the new and restarts,
while its engines keep serving. A node that does not come back on the new
version, ready, with its replicas, stops the upgrade before the next one.
Running it again resumes. Agents from before this have to be moved by hand
once.
### Following a request
Every response carries `X-Request-Id` — the client's own when it sends a plain
one, otherwise one the gateway makes — and the engine receives it too. One
access log line per request records it with the model asked for and served,
the replica, status, time queued, time to the first token, total time,
tokens, and the key, project and user. Usage records keep the id, replica,
queue time and time to first token, and `llmgw ctl request ID` reads one back:
the answer to "why was this slow?".
### Rate-limit headers
Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
tenancy tree will refuse first, so SDKs and editors can slow down before they
are refused.
### Memory sized from the model
The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
weights, plus the KV cache for the context pool from its layers, KV heads and
head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
to 5.25 GB across the models on the live mirror, which the old "weights plus
half again plus two" could not see.
### Security
- **Alertmanager answered anyone on the network**, including requests to
silence alerts. The monitoring README now binds it to loopback, where
Prometheus beside it is its only client.
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
64 MiB of argon2, with no bound on how many ran at once, and the per-address
throttle grew by one entry per address tried. Hashes now run a few at a time
and the rest wait; the throttle expires old entries, and when it is still
full it refuses new addresses rather than dropping live ones — which would
have let a flood of made-up addresses reset the throttle on a real one.
- **Any key holder could grow Prometheus without bound** by asking for made-up
model names, each of which became a metric label. Requests are counted under
the catalog's name for their model, or "unknown".
- **A SQLite backup left a full copy of the store readable** by local users
while it ran. The snapshot is now made in a private directory, and
llmgw-backup.service sets `UMask=0077`.
- **An engine archive could write outside its directory through chained
links.** Archives are unpacked through `os.Root`, and a link may only point
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
follows that rule.
- **Onboarding made every node trust the cluster's root for everything.** It
was added to the system store, so a certificate the control plane signed
would pass for any site with every TLS client on the node. The agent trusts
it through its own configuration only; onboarding no longer adds it, and
takes it out of a node that an older version onboarded.
- **Any node could manage a mirror running mutual TLS**, since one root signs
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
the certificate with that common name, and the mirror warns when mutual TLS
runs without one.
- **The administrator's password stayed in `init.log`.** It is shown once, on
the screen, and the log keeps a placeholder.
- **Auth off on a public address was silent.** The server now warns at start
when the API listens beyond loopback without keys.
- **The actor recorded with a change was whatever the caller sent**, line
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
records the person who ran sudo rather than root.
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
reaches it. It is off. The head itself still has no authentication, and
SGLang still takes its key as a flag; docs/install.md says what to do about
both. The monitoring README says how to keep Prometheus's queries to the
control plane.
### Fixes
- Tests that slept a fixed time and then asserted now wait for their
condition. They are the likeliest cause of CI's intermittent failures on a
loaded runner.
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
`ctl restart` did on its first real use.
- The status each failure is answered with — 499, 502, 503 with and without
Retry-After, 504, 429 — is now tested case by case.
### Releases
Small fixes are published as release candidates of the next version —
v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
notes come from this section, and they are marked as pre-releases. Onboarding
treats a release candidate as a release and downloads it.
## v0.10.3 — 2026-09-30
Security fixes from a review of v0.10.2. Three of them were confirmed on the
live cluster before being fixed.
### Engines demand a key only the router holds
Engines listened on every address of their node with no key. A request from a
neighbouring node reached the model directly, past the gateway's
authentication, quotas and accounting. Each replica's engine is now started
with a key derived from the replica id and a secret in the store. The router,
its metrics scrape, the agent's readiness check and `ctl bench` send it.
llama.cpp and vLLM receive it through their environment, not their command
line. Engines started before this keep running without a key until their
replica is next started.
### The mirror's management API answers only who it was told to
Without `--api-allow`, the management API answered anyone who could reach it,
including every node. It now answers this machine only; the control plane is
let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
add `--api-allow <control plane address>` to its unit.** Downloads started
through the API go to http and https only, and never to loopback, private or
link-local addresses (`--allow-private-sources` to permit that). Model,
version and file names can no longer be `..` or start with a dot.
### The admin listener refuses what a browser page could send
The admin API has no login and trusts whoever reaches it. A page open in an
operator's browser, reaching it over an ssh tunnel, could post a model
definition. The listener now requires Host to be an address or localhost,
refuses cross-site requests, and takes changes only as `application/json`.
A profile's `env` may no longer set variables that make an engine run other
code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
They are refused where a model is defined and dropped again before an engine
starts.
### Fixes
- The agent installs only engine versions that look like one, whatever the
control plane sent; a version of `../x` could have installed outside
`/opt/llmgw/engines`.
## v0.10.2 — 2026-09-29
- After `server init`'s screen closes, only the administrator's password and
the next steps are printed, not the whole run again under the screen that
just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
only, and printed in full when init fails.
- The screens' spinner and text field are llmgw's own rather than from
bubbles, one dependency fewer. Every llmgw command still asks the terminal
for its colours as it starts, which bubbletea v1 does on purpose; a terminal
that never answers waits for its timeout.
## v0.10.1 — 2026-09-29
### `llmgw ctl fabric-test`: two nodes, rail by rail
`llmgw doctor` checks a node on its own, and every check can pass while two
nodes still cannot reach each other: a cable into the wrong switch, a
partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
one rail at a time, through the agents — no SSH between nodes, and perftest is
installed the first time — and grades each against its link's rate: ok at 80%,
slow at 50%, fail below or when nothing got through.
Agents now say which requests they answer beyond the usual (`features`), so
one too old for the test is refused at once instead of waited for.
Written against perftest's real output and tested with it, but not yet run
over real InfiniBand or RoCE: this cluster has neither.
### The mirror on its own machine, in two commands
`llmgw mirror init --control-plane <address>` sets a machine up as the model
mirror: the user, `/srv/models`, and a service serving files to every node on
:8081 and the management API on :8082. `server init --mirror
http://<mirror>:8082` uses it instead of running one beside the control plane.
It took a unit written by hand and an edit to `server.yaml` before.
The management API can fetch and delete models; `mirror serve --api-allow`
makes it answer only the addresses given and the machine itself, and `mirror
init` gives it the control plane's. With `--no-node`, the starter model is
fetched and catalogued anyway, unpinned, and starts on the first node added.
Tried on three empty VMs: mirror, control plane and a node, and a request
answered by the starter model on the node.
### `server init` asks, and `node add` takes several nodes
- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
administrator's email, the other names nodes will reach the machine by
(showing the addresses it already found), SQLite or PostgreSQL, and whether
to run the mirror, a node and the starter model here. Then its steps fill
in as a list, downloads updating in place, and the administrator's password
stays in a box until the end — and is printed again after, with the whole
run, so it is in the terminal's history. The last screen says how to add a
node and to back up the authority's key. `--plain`, or no terminal, prints
lines as before.
- **`llmgw node add a b c`** adds each in turn and ends with which were added.
A fabric named for one node's adapters is recognised on the next by what the
adapters report, and not asked again.
Both tried on empty VMs: init from the form to a starter model answering, and
two nodes added to it in one command in 40 seconds.
## v0.10.0 — 2026-09-29
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
installed by the nodes themselves; people in many places at once; and a chat
page, editor setup and a model explorer for the people using it.
### Independent InfiniBand fabrics, told apart and drawn
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
with several independent fabrics, two given one name would have had a replica
spread across both — started, then hung on its first collective. Now:
- **Identity.** Each active InfiniBand port reports the master subnet manager
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
asked once a minute through one port per fabric. Ports whose managers are
listed together are one physical fabric, so a failover does not split one in
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
installs `infiniband-diags` where there are adapters and no `sminfo`.
- **Placement** divides a declared fabric by that identity and forms a replica
on one part only. A node whose adapters for one name are cabled to two
fabrics is refused; a node that cannot say is kept apart from those that can.
- **Subnet managers.** The cluster page lists each fabric's master, its
priority, its standbys and its master changes over the last day, and warns
when a fabric has no standby, when its ports disagree on the master, when a
port is up with no subnet manager (INIT), and when one name covers two
fabrics or two names one fabric.
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
each heartbeat: state, physical state, rate, error and data counters. A port
going down takes its fabric off the node within seconds, and back when it
returns — it used to wait for the agent to restart.
- **Maps.** Every five minutes one node on each fabric
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
the fabric: switches in their tiers, adapters under their leaves, cables
coloured by rate, degraded ones — trained below the rest of their kind —
marked in red, rail alignment, and the cluster's own adapters live, with
their traffic running along the cables. It zooms, pans, highlights what a
switch or adapter touches, and opens detail on a click. Placement prefers
the nodes of one leaf and puts nodes with a degraded cable last.
### NVLink, on every build and across nodes
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
which was the only source of NVLink peers: on every NVLink machine the
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
peers, each card's NUMA node, and how near each network adapter is.
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
within it, then — for a multi-node shard — nearness to the fabric's adapters,
then NUMA nodes spanned. It used to grow one card's peers, which took cards
joined to that one but not to each other.
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
found from each GPU's fabric state, ClusterUUID and CliqueId, with
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
domain first, on nodes whose every card is joined and whose IMEX is up, and
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
### Multi-node launches fixed before they meet hardware
- **vLLM launches multi-node its own way by default.** An empty
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
it now means `mp`, every node running `vllm serve` with `--nnodes`,
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
documents. The `mp` path used to give workers no rank and the leader's
address as their own. Set `launch_mode: ray` to keep Ray.
- **NCCL bootstraps over the node's own interface.** The agent's default of
`eth0` meant it never looked for the interface holding its management
address, and the control plane sent one name for every node. A node on
`ens3` or `bond0` had NCCL bind nothing.
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
address; NCCL's default is a v1 link-local one that does not route.
- A Ray worker now waits for its head before joining.
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
### `llmgw doctor`
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
wrong and how to fix it, most serious first: NVSwitch without the fabric
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
shard on each fabric would be given. `--json` writes every fact it read, to
send back; it exits non-zero when something fails.
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
model, where llama.cpp runs several times slower and nothing else looks wrong —
and whether vLLM can be installed on the node, and why not.
### vLLM installs itself on GPU nodes
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
that can run it, with nothing done by hand. A Python 3.12 environment is built
by uv — itself a GitHub release checked against its checksum — from PyPI,
resolved as PyPI stood a day after that vLLM was published and installed with
`--require-hashes`, so every node gets the same files whenever it is built. The
resolved list stays in the environment as `llmgw-requirements.txt`.
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
25 GB free while it installs. A node that lacks one says which on the
cluster page; a node with no GPU skips vLLM without a word.
- gcc is installed with it, since Triton compiles its kernel launcher on the
first model load.
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
version.
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
itself was not started there.
A node is asked only for engines its agent says it can install. An agent older
than this release says nothing, and is asked for llama.cpp alone, so a cluster
half way through its upgrade does not show every node as failing vLLM.
### `llmgw node add`: a node in one command
Adding a node was a bash script from the repository, not installed anywhere,
that needed the node's name, the control plane's address and a dozen flags.
It is now part of the binary:
```
sudo llmgw node add ubuntu@10.0.0.21
```
run on the control plane. The node id is the machine's hostname and the
address its agent dials is this machine's on the route to it, checked against
the server certificate. It installs llmgw on the node — the release for its
architecture, fetched and checked there, which works on RHEL too — runs
`llmgw doctor` there and stops on a problem, issues and installs the
certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
the node to join and install llama.cpp.
In a terminal it is a checklist that fills in, and asks what a script cannot:
which fabric each group of active adapters is on — grouped by the subnet
manager or RoCE network they report — and whether to go on past a problem.
Elsewhere it prints one line per step. Running it again on a node is safe.
`llmgw node remove` forgets a node that is gone for good, which otherwise
stays listed as lost until the control plane restarts; only a lost node, or an
unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
longer lists vLLM as skipped on every CPU node.
Tried on an empty VM, from the terminal screen and plain: joined with
llama.cpp ready in under a minute, and 17 seconds to add it again.
### People belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the
gateway before that. Now:
- **Memberships.** A person can be a member of any number of organizations,
teams and projects, or of none. A member of a team or organization may use
every project in it. `llmgw users join/leave`, and the people page adds and
removes them from either side. Approving no longer requires a project.
- **Personal keys.** A key can belong to a person with no project. On
`/profile` a person chooses what a new key spends against: one of their
projects, or Personal. `llmgw keys create --for <person>` does the same.
`auth.personal` sets default limits for personal use, or turns it off.
- **Limits for people.** `--scope user` limits a person across every key,
project and the chat page. `--per-member` on an organization, team or
project gives each member their own share there. Both are on the quotas
page, and are enforced alongside the tree, each in its own pool.
- **The chat page** asks what to spend against when a person has more than one
choice, and checks it on every message.
- **Spend is recorded with who spent it**, so a person's budget survives a
restart, personal use and the chat page included.
Leaving a project revokes the keys the person made there, unless another
membership still reaches it. A personal key keeps no conversations or notes:
those are kept per project.
### An organization's own page
A member of an organization can be made its **manager** (`users join --org …
--role manager`, or the star on the people page). A manager gets `/org`: the
organization's usage over time by team, project or person, its teams and
projects with limits and spend, and its members — read-only, and only what was
spent in its projects. Administrators can open every organization's page.
### The chat page
- A model that is not running shows a progress bar while it loads — waiting
for a node, downloading, starting — and the message is sent once it is up,
rather than failing after 45 seconds with "the model is loading".
- A finished reply is rendered as markdown, by the server under the model
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
- Stop, while a reply streams; its tokens, speed and time to first token under
it; a system prompt and a temperature; and the conversation kept in the
browser across a reload — the gateway still keeps none.
### Connect an editor, and the Responses API
- The account page makes a key for **Continue**, **Codex** or **Cline** and
shows what the tool needs, filled in: Continue's assistant file, Codex's
provider and profile, Cline's four settings. Continue and Codex get a one-line
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
no secret — the key comes from the environment — keeps a copy of whatever it
replaces, and leaves the rest of the file alone.
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
chat completions and back — instructions, messages, function and custom
tools with their calls and outputs, usage, streamed or not — down the same
path as any request.
### Your own use
The account page (was "Your keys") shows the last thirty days of a person's own
use and, for each place they spend, a bar for every budget that applies — their
own, or a project's shared with others, said as such.
### The mirror page
- **Explore models** opens on what is trending on Hugging Face, with views for
new and most downloaded, filters for task and size, and each repository's
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
base model and license. "Only what these nodes can run" is on by default.
- A repository opens beneath its row with its model card, and a download
started there starts in place, its progress followed in the table above.
- What the mirror holds is grouped by model family.
- An importance matrix published beside a model's quantisations is no longer
offered as a model.
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
demand, or stopped — and it can be changed there, without the CLI. The serve
form asks the same: on demand loads the model when somebody asks for it and
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
mode and says to change it there, because the file is read on every start.
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
was downloaded, catalogued and placed, and failed on every node it was tried on.
The architecture was in the first kilobytes of the file all along. Now:
- The mirror page's registry listing shows each file's architecture and, for
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
it. A vision projector (`mmproj`) is marked as not a model on its own.
- A fetch reads the file's header first — a range request, not a download —
and refuses one that cannot be served, unless "fetch anyway" is ticked.
- Serving from the mirror and `llmgw ctl model add` refuse the same.
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
not refused, only not checked.
### Each node's CPU and memory, and a page per node
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
used and available, and each engine's cores and resident memory. The cluster
page's node table shows CPU and RAM as bars; each node has a page with its
current load, its engines, its fabric ports, its GPUs, and an hour of charts
from Prometheus.
### The admin pages
- The cluster page leads with capacity, nodes, models and replicas, then usage;
it refreshes in place instead of reloading, and its usage table names people
and projects instead of ids.
- The usage page's legend gives each line's total, not its last hour.
- The people page keeps an account on one line and its rarer actions in a menu.
- Every page fits a phone: a wide table scrolls in its own box.
- Static files are named by their content, so a deploy's new styles and
scripts reach browsers at once instead of a day later.
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
order, the page you were on sometimes listed and sometimes not. Every signed-in
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
left for an administrator, Chat, Account and Sign out on the right for
everyone — with the current page marked. The chat and key pages name the
person's project by its path rather than its id.
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
its organization held it to sixty requests a minute. Each limit now shows the
value it is held to from above, muted and marked ↑, with where it comes from on
hover; a larger value set below a stricter one says what caps it, for rate and
token limits as well as budgets. Spend and budget share a column with a meter
that turns red past 90%. The tree is indented like the people page, opening a
form or saving one no longer reloads the page, and the explanations moved under
"How limits combine". Projects are placed under their team by id rather than by
the start of their path, which misplaced them when two organizations shared a
name.
### The project's website, in every gateway
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
the reverse proxy configurations, and the release's files to download. It is
served without a key like `/`, whose footer links to it, and says nothing about
the installation. Its fonts are in the binary, so it makes no request elsewhere.
The published website is rendered from the same template.
### Metrics
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
### Fixes
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
rewriting read it as the start of a string. Comments are now skipped.
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
- Usage by team, project or organization, and the spend on the quotas page,
left out every request made on the chat page: they were joined through the
key, and the chat page has none. They now count where a request was spent.
- `/org` was answered with "no api key was provided" for a signed-in browser;
a test now drives every session page through the authentication middleware.
### Upgrading
Upgrade the control plane first, then the agents: an older agent is served as
before and simply reports none of the new measurements. Existing agent units
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
that set `mgmt_interface` keeps it; one that did not now uses the interface
holding its management address rather than `eth0`. Multi-node vLLM replicas
start with `mp` unless `launch_mode: ray` is set.
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
attributes past usage to people where a key had an owner. On SQLite it rebuilds
the keys table; it was run against a copy of a production PostgreSQL store
before release. `llmgw users move` is gone: use `join` and `leave`.
vLLM is installed on every GPU node that can run it, once its agent is
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
replaced by `llmgw node add`; it still works, for a machine holding the
authority but not this llmgw.
## v0.9.2 — 2026-09-28
### A new control plane serves a model by itself
`llmgw server init` now also runs a model mirror on the same machine, makes the
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
and every step is skipped on a rerun when it is already done.
Not yet run end to end on a real machine.
## v0.9.1 — 2026-09-28
### The mirror page is one line per model
What the mirror holds used to take two table rows per file — seven columns and
the address under them — so a sharded model filled the screen. Now each version
is one line: model and version, the file (or "4 files" for a set), total size,
and which catalog models use it. The address moved into a "copy address" button;
the checksum is in the file's tooltip. A version already in use is not offered
for serving again, and removing one warns that the catalog uses it. The header
counts versions rather than files.
## v0.9.0 — 2026-09-28
### Managing people and the tenancy tree
Organizations, teams and projects can now be renamed, moved and removed, and an
approved person can be moved to another project or removed — none of which
could be done at all before, from the web or the CLI.
- **`/admin/users`** shows the whole tree, each project opening onto its people
and keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level. A search
over every account sits below it. Changes update the page in place (htmx,
embedded in the binary) and it still works with scripts off.
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
- **`llmgw users move`** and **`llmgw users delete`**.
- **`llmgw keys list --all`** lists every key with its project and owner.
- Every command that takes a project, team or organization accepts an id, a
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
be omitted when there is only one. Users are named by id or email.
Moving a person takes the keys they made with them; keys an operator issued stay
with the project. Only empty things are removed, and a project that ever had a
key is never removed, because usage history refers to it.
### Changes
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
built from the web has no project of that name. With one project it is
chosen; with several the command lists them.
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
rather than micro-units. `--budget 50000000` now means fifty million units;
divide existing scripts' figures by 1 000 000.
### Fixes
- `keys init` could silently move an existing team or project to another
organization or team by redeclaring it. It now refuses.
- An account rejected while pending and then reinstated became active with no
project, and failed at its first key. Reinstating one is refused; approving
gives it a project. Rejecting a signup now removes it.
## v0.8.8 — 2026-09-28
### Fixes
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
executable file not found". The PostgreSQL project's own packages (PGDG)
provide `postgresql-server` too, so dnf can answer that name with
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
reads the data directory from it, and runs `initdb` itself when the directory
is empty. A PostgreSQL that was already installed is used as it is.
### Known gaps
Not yet run on a real machine. Three arrangements are not handled: a
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
and says so), one listening on a port other than 5432, and several PGDG
versions where the highest is not the one on 5432.
## v0.8.7 — 2026-09-28
### PostgreSQL in one command
```
sudo llmgw server init --postgres
```
Installs PostgreSQL with the machine's own package manager when there is none,
creates the `llmgw` role and database with a generated password kept in
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
already running on SQLite it stops the gateway, copies the store across so
every account and key survives, points `auth.store` at PostgreSQL and starts
the gateway again; if the move fails, the gateway is started again on SQLite.
Given on the first run, there is never a SQLite store.
`server.yaml` is edited in place with its comments intact, and the previous
file is kept beside it. A PostgreSQL that init did not install is left sized as
it was, since something else may depend on it.
The PostgreSQL installation itself has not yet run on a real machine; the move,
the configuration edit and the rerun behaviour are tested.
### Fixes
- **A new control plane with no models did not start.** A configuration with
no static endpoints and no models was refused as having nothing to serve, so
under systemd the unit restarted every two seconds and nothing listened on
8080. It now runs as a control plane with an empty catalog.
## v0.8.6 — 2026-09-28
### Reverse proxy configurations
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
port, and both are attached to every release beside the binaries. They do not
buffer streamed replies, their read timeout outlasts a model's wait before its
first token, their body limit matches the router's, and they set
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
cookies `Secure` from the first and shows the second as the base URL on its
landing page.
## v0.8.5 — 2026-09-28
### A control plane in one command
```
sudo llmgw server init --admin you@example.com
```
Creates the llmgw user and its directories, the certificate authority, the
control plane's certificate for the machine's hostname and every address it
has, `server.yaml`, the default project, and an administrator whose password is
generated and printed once — then installs the systemd unit, starts it and
waits until it answers. What used to be two steps of install.md is now this.
Each step is skipped when its result exists, so running it again finishes an
install that stopped and changes nothing on one that did not. `--host` adds an
address the machine cannot see of itself (a DNS alias, a NAT or floating
address) and reissues the certificate keeping the old ones. It never overwrites
`server.yaml` or an authority, and certificates issued by an authority kept on
another machine are left as they are.
### Fixes
- **The configuration install.md showed could not start.** An empty catalog
without `control.enabled: true` is refused as having nothing to serve. The
documentation and the file `init` writes both set it.
## v0.8.4 — 2026-09-28
### Node logs on the admin page
Each node on the cluster page links to its logs — the agent's own, and the
output of every engine running there — with an option to follow them. `llmgw
ctl logs NODE` reads the same. They come over the agent's existing stream, so
no ssh, jump host or root is needed to see what a node did. The agent keeps its
last 2000 lines in memory; each engine its last few hundred.
## v0.8.3 — 2026-09-28
Found by building a node from nothing with v0.8.2.
### Fixes
- **A new node's first engine install failed on apt's lock.** cloud-init and
unattended-upgrades run apt in a machine's first minutes, and
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
package command held up by a lock now waits for up to ten minutes, and says
so; a failed install is retried after a minute rather than ten.
- **A new node reported no disk** until its first download created the weights
directory. It is created when the agent starts.
- **Onboarding took one control plane address.** It takes several, comma
separated, so a new node fails over like the rest.
## v0.8.2 — 2026-09-28
A node no longer runs whatever engine somebody once put on it. It installs its
own.
### Engines are installed by the node
`engines:` in server.yaml names the engines every node has and their versions,
defaulting to the llama.cpp build this release was checked against (b10991).
Each agent converges on it: it picks the build for its hardware — CUDA with its
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
packages that build needs with the machine's own package manager, downloads the
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
anything that would land outside its directory, and runs it. `ldd` names a
missing library or a system too old for the build ("libstdc++.so.6 is too old
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
again every ten minutes.
An engine is reported only once installed and seen to run, or when the agent's
configuration names it on purpose. A node that has installed nothing yet is not
a candidate for any model, where before it was a candidate for all of them.
`ctl nodes`, the cluster page and placement's refusals say where each
installation stands, and warn about a CPU without AVX2.
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
installed this way and must still be named in the agent's configuration.
### Onboarding
Onboarding installs llmgw from the CI release for the node's architecture,
checked against the release's checksums, and writes an agent configuration of
only the control plane's addresses and the TLS files.
### Fixes
- A race on the weights fingerprint, which two heartbeat loops touched during a
failover.
## v0.8.1 — 2026-09-28
Nothing here changes a configuration format: every file that worked with
v0.8.0 still works. What changes is how much of it needs writing, and how much
of running the cluster can be done from a page rather than a shell.
### A node describes itself
An agent file used to name its node, its address, the path to llama-server and
its weight directories — different on every node and kept true by hand. The
agent now finds these out: its id is the hostname, its address is the one on
the route to the control plane, llama-server is looked for on PATH and in
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
Memory is re-read on every heartbeat and installed engines once a minute, so a
VM given more RAM or a node given an engine becomes a candidate without an
agent restart; the control plane logs both. The agent file shrinks to the
control plane's addresses and the TLS files, the same on every node.
### Models from the mirror, without writing YAML
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
and serve a mirrored version from its row — name, source, engine and an
estimated memory requirement filled in. A model can say what it is for, and
`/v1/models` shows it. Placement no longer hands the same memory to two models
or puts an engine on a node that lacks it, and says when a replica runs an
edited profile.
### Tenancy, quotas and usage on pages
The tenancy tree, quotas and budgets are managed from the admin pages; an
organization's budget bounds its teams, a group's quota counts against the
group, and a budget that cannot be spent against says so. `/admin/usage`
charts requests, tokens and spend over time by organization, team, project or
person, and a request is priced by the model that served it.
### Watching the cluster
The cluster page has its own charts, backed by a year of Prometheus history,
with alerts that were made to fire before being trusted — including for the
disk the metrics share with the models.
### Chat
The picker offers each model once rather than once per alias, never asks for
more tokens than the model allows, and offers a model that loads on demand —
sending it a message is what loads it.
## v0.8.0 — 2026-09-27
The store stops being a file on one machine's disk. PostgreSQL, a way to move
an existing installation onto it without reissuing every key, and the two
things a second control plane needs so that running one is safe rather than
clever.
### A second control plane, and a way to fail over to it
Leadership is a lease in the store, so two control planes against one database
cannot both place work — which they otherwise would, both seeing a model
wanting replicas and none running, leaving twice the replicas each holding GPUs
the other believes are free.
A standby also refuses agent streams. Commands travel down the stream the agent
opened, so a standby holding one is a node the leader cannot command, and the
leader's placements would fail with "no agent connected" for a node that is
plainly connected. Agents therefore take a list of addresses rather than one,
trying each in turn: a standby refusing, a stopped control plane refusing the
connection and a crashed one never answering are the same thing from a node.
Measured on the cluster. Leadership moved three seconds after the leader was
stopped — it released the lease on the way out rather than letting it lapse —
and all five agents had registered with the standby six seconds later. The
standby adopted every replica rather than restarting any: six replicas, all
ready, engines that never stopped serving.
Everything is unset-means-leader, so a single control plane and every
deployment that existed before this behave exactly as they did.
What this does not do is move client traffic. The standby served inference
correctly on its own port while the public address returned 502, because
nothing in front of them moved — that needs a load balancer or a shared
address, and it is the remaining piece.
### PostgreSQL
The store runs on PostgreSQL, chosen by what the address looks like: a path is
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
address out of a file so a DSN carrying a password stays out of a configuration
that gets copied and pasted into issues. Every existing installation keeps
working untouched.
One set of queries rather than one per database, because the SQL here is almost
entirely portable and two copies would drift one query at a time. The rewrite
of placeholders is a wrapper around the handle rather than a change at sixty
call sites — which found six places reaching past it, including all three
manual transactions and two prepared statements, every one of which worked on
SQLite and would have failed on PostgreSQL.
Exactly one query is not the same sentence in both: the note search, MATCH
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget. It uses the 'simple' configuration
rather than English — these are an agent's own notes, in whatever language it
chose, and stemming English rules over Turkish would quietly stop finding
things.
`llmgw keys move-store` copies an existing store rather than starting a fresh
one, because keys are stored hashed: a migration that drops them cannot
recreate them, and every person and every script would need a new one.
Testing it against a real database found three things nothing else would have.
`schema_migrations` is created in Go rather than in a migration file, so it
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
and a millisecond timestamp does not fit. The copy order followed the order the
migrations create tables in, and `users` is created after `api_keys` even
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
difference pgx cannot see, so it arrived as NULL.
The driver is pgx, which is pure Go: the whole binary is built with
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
and a cgo driver would take that away from every deployment, including the ones
staying on SQLite.
## v0.7.0 — 2026-09-25
What a cluster needs to be lived with rather than demonstrated: somewhere to
keep a conversation, somewhere to keep a note, and an honest answer about the
disk underneath both.
### Memory notes for agents, and threads without API keys
Notes an agent writes for itself and reads back in later conversations, offered
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
`forget`). A note is a key, text and tags, searched by words or tag, private to
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
tools into chat requests: an agent that is not given the server never sees
memory.
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
and the retention is required, as it is for threads.
On a gateway with no authentication, a `threads:` block (`store`,
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
nothing refuses it with an explanation instead. The shared pool is for one
trusted machine: anyone who can reach the port and knows an id can read it.
### Conversations that can move between models
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
llmgw replays the stored history to whichever model is serving, which is what
lets a conversation begun on a large model continue on a small one — including
when a spent budget moved it there and the caller never chose.
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
stores no message content anywhere, as it never has, and that default is worth
more than the feature. The retention is required rather than optional, because
a conversation store with no expiry is a transcript of everything a team ever
typed kept forever by omission, and it is measured from the last turn so a
conversation in progress does not expire underneath it.
History that does not fit the serving model is trimmed oldest-first and the
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
are never dropped — losing one silently changes what the model is for while it
carries on answering — and neither is the turn the caller just sent.
Threads are scoped to a project, and reading, writing or deleting one that
belongs to another answers exactly as one that never existed: two answers would
let a caller map which ids are real in projects they cannot read.
Capturing the reply is opt-in and bounded, so a request without a thread
accumulates nothing on the hottest path in the system, and one with a thread
stops at a cap rather than growing to it.
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
operator: the person who said the thing should not have to ask someone to read
it before it can go.
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space in the same struct was discarded, so the only way to find
out why a node was filling up was to ssh to it.
Nodes now report it, per filesystem rather than per configured directory: free
space belongs to the disk, and two model directories on one disk are one pool
of bytes. Shared filesystems are kept out of a node's own figure, because space
on a filer belongs to everyone mounting it and twenty nodes would otherwise
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
Prometheus, and a node that reported nothing reads as unknown everywhere — an
older agent must not look like a full disk.
A profile may declare `min_disk_gb`, and a node without the room is no longer a
candidate, with a reason `explain-placement` prints. Skipped on a node that
already holds the weights: it needs no room for them, and without that the node
best placed to run a model is the one most likely to be rejected.
`ctl weights` lists model files llmgw downloaded for models no longer in the
catalog, and `--collect` deletes them. The design is mostly about what it will
not do. Only files llmgw fetched are candidates, and that is a calculation
rather than a heuristic — the cache name is a hash of the source, so the wanted
set is computable exactly. Anything an operator put on a node is left alone
however unreferenced it looks. Each node checks every path again before
unlinking it, refusing anything outside its cache directory or without the
cache name, so a control plane that is wrong cannot talk a node into losing a
model that took a week to copy.
### Fixes
- **A capability was lost at an indirection.** The controller is handed a
forwarding dispatcher, and reaches past its interface by type assertion to
find what lies beyond scheduling. Deleting weights was implemented on the
agent service and not on the forwarder, so it was simply absent at runtime —
with the whole suite green, because the dispatchers the tests use do not
implement it either and its absence there is the expected case.
- **`ctl tool-check` blamed the engine for what the model did.** It had one
verdict for a failed attempt and told operators not to try another model.
Measured: the engine renders the tool prompt correctly, the coder models do
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
the same engine returns a populated `tool_calls` array. A failed attempt is
now two verdicts, told apart by whether the reply used the wrapper.
## v0.6.0 — 2026-09-18
Phase 4: a replica can be bigger than one machine, and the fabric it spans
decides where it is allowed to go. Around that, everything a cluster needs
before anyone other than its operator is let near it — keys, budgets, a way in
through a browser, and enough visibility to answer "why is nothing scheduling".
### Multi-node replicas
A replica is now a set of shards with ranks, a leader and a bootstrap address,
rather than one process. Two invariants are enforced rather than documented: a
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
several times slower, and nothing says which.
A replica missing a shard is not ready. It was previously derived from the
shards that had reported, so a two-node replica with one node up looked
servable and took traffic into a model that could not answer.
Adoption after a restart happens one agent at a time, because that is how
agents reconnect. Merging them as they arrive is what keeps a multi-node
replica whole instead of being torn into fragments that are then killed as
strays — and an adopted engine is health-checked before it is called ready,
rather than being assumed alive because its process exists.
### Fabrics are distribution domains too
An isolated fabric is not only a compute boundary: weights present in one are
not present in the other. Node weight inventories are reported and summarised
per fabric, so "the model is on the cluster" is never mistaken for "the model
is reachable from the nodes that would run it". A fabric name that matches
nothing a node actually has is refused at registration rather than becoming an
unschedulable replica hours later.
### Who may use it
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
still no cgo). Rate limits and budgets are enforced on the request path; usage
is written off it, in batches, so accounting never adds latency to inference.
Budgets survive a restart — counters are rebuilt from the store, because a
limit anyone can clear by bouncing the process is not a limit.
A spent budget is now served a cheaper model where the catalog offers one,
rather than refused. For a team trying models out, being dropped to something
smaller is a far better answer than a wall, and the response says which model
answered.
People can sign up, be approved, and issue and revoke their own keys. A key is
shown once; only its hash is kept.
### The browser surface
A landing page that says how to connect, a read-only cluster dashboard, and an
in-browser chat that never puts an API key in the browser. The dashboard shows
who is using the cluster and what they are running — per person and per model,
with requests, tokens, errors, spend and latency.
All of it is served from the client-facing port behind a session, deliberately
not from the admin port, which also serves pprof — and pprof can dump the heap
that holds every API key in plaintext.
### Models without a restart
The catalog is live. Models can be added and removed from the command line or
the API, take effect immediately, and outlive a restart; configuration still
wins over the store, and the whole catalog is validated before anything is
written.
A model is only listed once it can actually serve. It was listed as soon as it
was known, so the first thing a new user did was ask for a model that returned
503.
### Loading on demand, and letting go
A model whose scaling allows it is started when someone asks for it, and
stopped when nobody has for a while. An idle replica is evicted to make room
for one that was asked for, with guards that exist because each is a way to
lose work quietly: only on-demand models, only single-node replicas, never one
that has never been used, and never one younger than its minimum lifetime.
Unknown is not idle. A replica with no recorded use is left alone rather than
treated as idle since the epoch.
### Knowing whether it will run before it runs
- `ctl plan` works out whether a model fits the cluster and how it would be
split, in the precision the chosen engine can actually serve — recommending
a quantisation the engine cannot load would cause the failure the command
exists to prevent.
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
installed. Flags drift between releases, and the alternative is finding out
during a model load.
- Placement now knows a model's context length and memory footprint, and node
RAM, so a CPU profile is not placed on a machine that cannot hold it.
- Placement waits for the fleet to settle after a restart rather than deciding
against a half-reported cluster.
### Weights
Nodes report the model files they can see, without llmgw owning the storage:
a shared filer, an image exported over the fabric, rsync or a copy made by hand
all look the same to placement, which only cares whether the files are there.
Reporting nothing means "not asked to look", not "has nothing".
Where the simple arrangement is not enough, a node can fetch a model by URL —
single-flight, resumable, written to a temporary name and renamed atomically,
so a killed download never leaves something that looks like weights.
### Seeing the fleet
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
scrape time rather than from a background goroutine, and served on their own
endpoint rather than through the one that also exposes pprof. The same view is
in the dashboard, so nothing else needs installing to see it.
### Onboarding a node
Adding a node is one command. There are systemd units, a private CA and a
from-nothing install guide that starts with a machine that has no CUDA and no
binary.
### llama.cpp, and CPU profiles
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
API without a GPU, so placement, supervision, scraping, scheduling and admission
can all be exercised on ordinary machines.
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
engines that can actually serve without a card may omit it — for vLLM an omitted
class stays an error, because otherwise a typo becomes a replica placed on a
machine with no hardware and diagnosed minutes later during model loading. CPU
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
allocation to stop two of them sharing a machine's cores.
### Metric mappings are per engine
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
than one for the cluster. A mixed fleet has two names for the same three numbers,
and a single global mapping left half of it unmeasurable.
A mapping that leaves a metric unnamed now means the engine does not publish it,
and the field stays zero. Naming one the engine does not serve is still a contract
violation. `Running` and `Waiting` can never be omitted.
### Mutual TLS on the agent stream
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
certificates on both ends, and the control plane refusing any agent whose claimed
node id does not match its certificate's common name. Public authorities cannot
supply these: they stopped issuing the `clientAuth` usage.
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
agent turn it on.
### Fixes
- **An agent told to stop did not stop.** Cancelling the context is not enough —
HTTP/2 watches it while the request body is being written, and a bidirectional
stream spends its life blocked on the read side. Measured on a deployed node:
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
- **An agent could not explain why it was refused.** The session cancelled its own
context and then asked that context whether it was shutting down, so every
stream error read as a clean close: no message, and the backoff reset.
- **A broken metric contract was invisible.** It was logged per scrape at debug
level, so a cluster could look healthy while every routing decision was made
blind. It is now a warning, once, with a matching line when it clears.
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
over about 2 KB was undercounted, so tokens-per-minute limits let through far
more than they were set to.
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
bfloat16 does not fit two 80 GB cards. It does.
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
already written the response header, which is the one point at which headers
stop being changeable. A client had no way to tell a fallback from the model
it asked for.
- **Fleet metrics published nothing.** They were registered after the registry
had been built, so the collector existed and was never scraped.
- **GPU discovery broke on older drivers.** A field added for utilisation was
required rather than optional, so a driver that does not report it produced no
inventory at all instead of an inventory without that number.
- **Replica ids and ports were reused** while the processes holding them were
still alive.
## v0.5.0 — 2026-09-16
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
control plane places replicas, and `ctl` makes all of it visible.
This release covers both phases because their exit criteria depended on the same
piece — the agent stream — and neither was demonstrable without it.
### ctl
```
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
```
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
the answer when a node looks idle but nothing schedules onto it.
- `explain-placement` prints every candidate considered and the reason it was
rejected. "Unschedulable" on its own says nothing about which of six
constraints bit.
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
is placed elsewhere, and the original is stopped only once the replacement is
serving. Bounded by a grace period, because draining the only node that can
host a model would otherwise never finish and leave it undrainable.
- `cordon` is the weaker operation: no new placement, nothing disturbed.
The management API is on the admin listener, not the client-facing one. The two
have different audiences: one serves inference to whoever holds an API key, the
other lets a caller cordon nodes.
`ctl logs` is not here. It needs a request/response pair the agent protocol does
not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
### Phases 2 and 3, in detail
Written as the work landed, which is why it reads as a log rather than as
notes. Everything here shipped in v0.5.0.
### Added
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
without IB is not an error — it is a single-node island.
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
would force the whole binary, server included, to be built with a C
toolchain. Health flags anything that should keep a card out of placement:
uncorrected ECC errors, and memory already held by another process.
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
always match the marketing name: an L40S is sold as 48GB and reports 44.99
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
box, because the number decides whether a model fits.
- **Engine adapters** for vLLM and SGLang: command and environment
construction, kept free of process handling so it is testable without GPUs.
vLLM supports both Ray and multiprocessing launch modes, because which one
works depends on the version.
- **NCCL environment generation**, the part plan §5.3 marks critical.
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
NCCL pick an adapter on another fabric where the peer is unreachable, and the
job hangs rather than failing. Bootstrap is pinned to the management
interface for the same reason.
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
compute capability 8.9 rather than silently falling back to a slower path.
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
process group and are stopped as a group, because engines fork workers that
hold GPU memory and outlive a signal sent only to the leader. Bounded log
tail, a state machine that separates Starting from Loading (a 70B model takes
minutes, and an operator needs to know that is normal), and orphan adoption
through pid files fingerprinted with the command line — pids get reused, and
adopting an unrelated process is worse than leaking the original.
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
says the weights loaded. Routing on `/health` alone sends traffic into a model
that is still loading.
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
as a table or JSON. Run it before adding a node: it is the fastest way to find
a card another process is using, or a port with no subnet manager.
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
a tensor-parallel group off the slow path inside a node. `Discover` falls back
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
topology. CI compiles the tagged build so it cannot rot unnoticed.
### Not verified
The NVML path compiles but has never run against a driver — there is no GPU
machine to try it on. Treat the NVLink domains it reports as unverified until
someone runs `llmgw agent inventory` on a real node.
- **Control plane wiring** (`internal/control`): Ready replicas become router
endpoints, with no static configuration involved. The router keeps live state
for endpoints that survive an update — inflight counts, breaker state, drain
rate — so a replica that was already serving does not start from cold because
another one moved. The sink is an interface, so when the router and control
plane split into separate processes this becomes a watch stream and nothing
above it changes.
### Phase 3 exit criterion, measured in `test/e2e`
A node hosting a live replica is killed while four clients stream continuously.
The control plane marks it unreachable, then lost, rebuilds the replica on a
survivor, and republishes. Across three runs: **~370 requests completed, zero
failures**, and no replica remained on the dead node.
This proves the control logic, not the transport: the agent fleet is simulated,
and the real path still needs the stream.
### A real bug the race detector caught
`startReplica` read the replica's fields after publishing it to the map, and
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
status reports, which arrive from another goroutine and mutate exactly those
fields. Under the simulated fleet the reports came back fast enough to hit the
window every time. Everything is snapshotted under the lock now and the
functions work from values.
### The agent stream
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
carrying registration, heartbeats and shard status upward, and start, stop and
drain commands downward. The agent dials out, because GPU nodes sit behind
firewalls far more often than a control plane does and an outbound connection
needs no inbound rule on every node in the fleet.
- **Registration carries the whole inventory**, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it — they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
the reconciler's dispatcher: it holds the open streams, so it is the only
thing that can deliver a command to a node.
- **Agent** (`internal/agent`) validates a profile before launching anything —
an engine that refuses a profile does so after minutes of loading — then
supervises the process and reports every transition, including the engine's
output tail on failure.
- **Reconnects forever with jittered backoff.** A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock the control plane over again. Engines keep serving throughout: the
router talks to them directly.
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
any model declares a profile. The static endpoint list remains for
router-only runs against sim-engines.
### Verified with real binaries
Server and agent as separate processes over a real stream: the agent registers,
a replica is placed, an engine is launched through the vLLM adapter's actual
argument construction, streaming traffic is served through the router, and
killing the agent moves the node through unreachable to lost with placement
explaining exactly why it can no longer run anything.
### Surviving a control plane restart
Not persisting replica state left one hole that had to be closed before it
mattered: restarting the control plane orphaned live engines. They kept holding
GPUs the inventory believed were free, so the next placement failed to allocate
with an error naming the new process rather than the old one.
The fix is on the agent, not in a database, and it cannot disagree with reality
the way a stale row can:
- The supervisor now writes the full shard spec, not just a pid and command
line. A pid identifies a process; the spec is what lets it be described back
to the control plane and matched to a replica.
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
waited on — it is not this process's child — so it is watched by polling, and
stopped by signalling its group directly. Its earlier output is gone, and the
log says so rather than presenting emptiness as silence.
- Every `Register`, not only the first, reports what the agent is running. A
reconnect may be a control plane restart and has to recover the same way.
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped: nothing will
route to it and it is holding cards.
- An engine the agent cannot identify — a corrupt state file — is killed. A
process holding GPUs that nothing can describe is worse than no process,
because placement will hand those GPUs out again.
One mechanism covers agent restarts and control plane restarts.
### A test that was passing for no reason
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
tracking hijacked connections. The agent never reconnected and the assertion —
on the connection map, which still held a stale entry — passed anyway. Tests now
sever connections through their own listener and count opened streams.
### Multi-node launch
Placement and environment generation had landed early, which made multi-node
look further along than it was: a multi-node replica could not come up at all.
Three things were missing, and all three came down to two assumptions that do
not hold once a replica spans nodes.
**Not every rank serves HTTP.** The agent health-checked every shard at its own
port, but only the leader answers. Workers timed out, reported Failed, and took
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
worker is judged by staying alive instead. That is weaker evidence on purpose —
the replica's real readiness is the leader answering, which only happens once
every worker has joined.
**A shard is not always one process.** `Engine.Commands(spec)` returns a
sequence: every command but the last is a sidecar kept alive for the shard's
lifetime and torn down with it, and the last one's exit is the shard's exit.
vLLM under Ray is why — the leader runs a Ray head and then the server against
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
had been written and tested but never called, so `launch_mode: ray` would have
run `vllm serve` against a cluster that did not exist.
A command can also require the rendezvous point to be listening before the next
one starts, bounded by a timeout so a head that never appears fails the shard
rather than hanging it. Sidecars are killed with the shard: a Ray head left
behind holds the port the next replica needs.
**Start order was backwards.** Workers were started first, on the theory that
the head should wait for them. Both engines put the rendezvous point on rank 0,
so a worker started first has nothing to connect to and spends its startup
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
needs, so the leader can start immediately and waits on its own.
None of this is verified against real engines. It is tested with stub processes;
the Ray sequencing and which launch mode a given vLLM version wants can only be
confirmed on hardware.
### Still missing for phase 2
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
network.
### Phase 3: placement
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
their fabric attachments, and what is allocated where. A re-registering agent
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
without doing so either. A GPU with an ECC fault or another process on it is
not counted as free. Replicas are released whole, because they fail whole.
- **Placement** (`internal/control/placement`): picks the highest-priority
profile that fits and explains every candidate it rejected. Two constraints
are enforced structurally rather than checked afterwards — there is no code
path that can combine nodes from two fabrics, and only GPUs of the profile's
class are ever considered.
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
large models that cannot be split. Multi-node replicas get the opposite
treatment: they are the least movable thing in the cluster, so they take the
emptiest machines.
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
domain where the topology is known. Without the NVML build it falls back to
the lowest free indices, which is usually right but only by accident of how
vendors number slots.
- **Property-based invariant tests** (plan §13): 400 randomly generated
clusters, checking on every placement that a multi-node replica lives on one
fabric, that each shard receives only that fabric's adapters, and that no
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
The table-driven tests cover the cases someone thought of; this covers the
ones nobody did.
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
rather than at request time — a typo in a fallback should stop a deployment,
not surface during the outage when the fallback is finally needed. The
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
load instead of minutes into model loading.
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
loop. It places one replica per pass, because placement depends on what is
already allocated and starting several from one snapshot would hand them the
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
vLLM under Ray needs its workers attached before the head serves.
- **Replica failure is whole-replica failure.** One dead shard makes the others
useless — they wait on collectives that will never complete while holding
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
exponential backoff carried on the model rather than the replica it just
destroyed.
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
blip must not tear down replicas that are still serving, and rebuilding a
multi-node replica costs tens of minutes of model loading.
- **Scale-down drains before stopping**, so in-flight requests finish instead of
being cut off mid-stream.
## v0.3.0 — 2026-09-16
Flow control. A batch workload no longer starves interactive traffic.
### Added
- **Admission control** (`internal/router/admission`), inserted between quota
checks and scheduling. It answers *when* a request may proceed; the scheduler
still answers *where* it goes. Keeping them apart is the point: an engine
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
no amount of endpoint selection rescues the request behind it.
- **Saturation detector.** One number per pool, the worse of queue depth and KV
occupancy. Either signal alone misleads — a pool can have an empty queue and a
nearly full cache, or the reverse.
- **Priority bands.** Each band has a saturation ceiling; above it, that band
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
"flow up to the healthy limit"; above 1 means never wait.
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
one caller sending a thousand requests occupies one slot in the rotation
rather than a thousand.
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
(504), because sending the GPU work the client has already abandoned helps
nobody. Cancelled requests leave the queue instead of holding capacity.
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
`x-llmgw-queue-ms` when a request waited.
### Phase 1.5 exit criterion, measured in `test/e2e`
While 500 concurrent bronze requests run continuously, interactive p99 in the
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
With admission switched off the same load pushes it to **55–60x**. Both
directions are asserted, so the test fails if admission regresses *and* if the
load generator ever stops being a real load.
### Two lags, two different fixes
Getting this right took longer than the queueing itself:
- Engine metrics are scraped, so they lag by an interval. The router's inflight
count does not — but it includes *running* requests, and treating those as
queue depth makes a healthy engine look overloaded every interval. What is
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
the metric contract, so it is learned as the high-water mark of observed
`running`.
- Between admission granting a request and the router dispatching it, the
request is invisible to every signal. A simultaneous burst therefore passed
the check en masse — exactly the batch arrival pattern this layer exists to
survive. The controller now counts its own outstanding leases against
dispatched inflight.
### Known limitations
- The band comes from a trusted header until authentication arrives in v1.0.0.
Anyone who can reach the port can promote themselves to platinum.
- The tenant key is the session id, for the same reason.
- Displacement (dropping queued low-band work to make room) is not implemented;
the plan defers it, and bounded queues plus TTL cover most of the need.
## v0.2.0 — 2026-09-16
The router serves traffic. `llmgw server` is real: it fronts a static set of
engine replicas with an OpenAI-compatible API.
### Added
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog.
- **Retry and fallback.** Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it via `x-llmgw-served-model`.
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched.
- **Usage accounting that does not depend on the client.** The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way.
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time.
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
### Phase 1 exit criterion, measured in `test/e2e`
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
### Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing.
- The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
Reference in new issue View Git Blame Copy Permalink