CI / test (push) Successful in 9m31s
The same inputs place a replica on the same node and card, so an on-demand model that failed to load there was started there again after every backoff until it was parked. The reconciler remembers where a model's replicas failed and places the next away from them when anything else fits (Scheduler.Away); with nowhere else, there again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5398 lines
312 KiB
Markdown
5398 lines
312 KiB
Markdown
# Changelog
|
||
|
||
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
||
is only released once its **exit criterion** is met, not when its code lands.
|
||
|
||
| Version | Phase | Scope | State |
|
||
|---|---|---|---|
|
||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
|
||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
|
||
| v0.7.0 | — | Conversations, agent memory, disk management | released |
|
||
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
|
||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
|
||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs; v0.16.2: standbys that answer clients, FreeToken, a model fetched once per node; v0.16.3: replicas per profile, a served model's settings on the mirror page, the CPU's speed before serving) |
|
||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||
|
||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||
APIs change between minors without a deprecation period.
|
||
|
||
## Unreleased
|
||
|
||
### Added
|
||
|
||
- **An organization's admin.** Besides member and manager, a membership of
|
||
an organization may now be `admin` (`llmgw users join … --org Acme --role
|
||
admin`, the role chosen on a membership on the people page, the console,
|
||
the admin API). An admin sees the organization's page as a manager does and
|
||
also changes the organization from it: adds, renames and removes its teams
|
||
and projects, adds people with an approved account by their address and
|
||
takes them out, makes its members managers or admins, and revokes keys spent
|
||
in its projects. Limits, approving and suspending accounts, the
|
||
organization's name and the rest of the cluster stay with the cluster's
|
||
administrators; an admin cannot take away their own role. Every change is in
|
||
the audit log.
|
||
|
||
### Changed
|
||
|
||
- **Forms only for those who may send them.** A viewer sees the people,
|
||
limits, mirror and cluster pages without their forms, an operator the
|
||
mirror's and a node's engines but not the rest; before, they were drawn and
|
||
answered "not found".
|
||
- **Smaller forms.** A label sits close to its field, a field is as wide as
|
||
what goes in it rather than the page, buttons are smaller, and the short
|
||
forms — a new key, a password, connecting an editor, a capacity or a limit —
|
||
sit on one line or two instead of a screen.
|
||
|
||
### Fixed
|
||
|
||
- **The serve forms start at the longest context that fits.** The mirror
|
||
page's form began every GGUF model at 4096 tokens, and the console's at
|
||
4096 or the plan's figure, whatever the model was trained for. Both now
|
||
start at what it was trained for, as long as its KV cache fits where the
|
||
form places it beside the weights, else at the most that does, in whole
|
||
thousands; the mirror page's llama.cpp slots are as many of that, up to
|
||
four, as the room left holds, and its memory figure is for that pool. A
|
||
plan says what the model was trained for (`trained`). Strata stays at
|
||
its 32768; vLLM's form is unchanged.
|
||
- **A model that failed to start was started in the same place again.**
|
||
Placement given the same inputs chooses the same node and card, so an
|
||
on-demand model whose replica failed to load was retried there, past each
|
||
backoff, until it was parked, with another card or node free all along.
|
||
The control plane now remembers where a model's replicas failed (node and
|
||
cards, up to eight places) and places the next one elsewhere when anywhere
|
||
else fits; with nowhere else it is tried there as before. Coming up, a
|
||
change to its definition or `ctl retry` forgets the places. The log says
|
||
`placing a model away from where it failed to start`, with both places.
|
||
- **Deleted model files came back after a control plane restart.** An agent
|
||
registers with the files its startup scan found, and sends the list again
|
||
only when it changes; a file deleted after the agent started was listed
|
||
again by a control plane that restarted, until the next change. A new
|
||
stream now gets the list on its first heartbeat.
|
||
- **`ctl weights` offered models in use for deletion.** A model of several
|
||
files (a set, its address ending in `/`) is fetched into a directory named
|
||
from the address without the slash; the sweep named it from the address
|
||
with it, so every set in the catalog looked like nobody's. On the PoC that
|
||
was 330 GB of six catalog models, the qwen3.8-flash-next Q8_0 among them;
|
||
`--collect`, and `ctl model remove --purge` of a set, would have deleted
|
||
them. Both now compute the name the fetch does.
|
||
- **A sign-in took an organization's manager role away.** A group from single
|
||
sign-on, SCIM or Active Directory that puts someone in an organization wrote
|
||
the membership again at every sign-in or sync, as a plain member: a manager
|
||
made one on the people page was one only until they next signed in. A
|
||
membership the directory names is now added if missing and otherwise left
|
||
as it is.
|
||
- **The limits form on the people page's panel** was laid out on one line, its
|
||
title squeezed into a column.
|
||
- **Deleting the model files nothing uses said what it was asked to free,
|
||
not what it freed.** The console's "Delete model files nothing uses" and
|
||
`ctl weights --collect` added up every file the nodes were asked to
|
||
delete as soon as the request was sent, and a model of several files —
|
||
fetched into a directory of its own — was refused by the node as "not
|
||
empty", so a sweep said tens of gigabytes freed and nothing went. The
|
||
node now deletes such a directory whole (a link only as a link), and the
|
||
sweep waits for the nodes to list their files again: freed is what they
|
||
no longer list, and what they kept is said beside, with where to read
|
||
why. Removing a model with its files reports the same way.
|
||
|
||
## v0.16.3 — 2026-11-03
|
||
|
||
Replicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
|
||
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
|
||
the threads it is given rather than the whole machine's memory.
|
||
|
||
**Upgrading.**
|
||
- No migrations. `llmgw ctl upgrade v0.16.3`: the control planes carry the replicas per profile, the model card and
|
||
the speed in the serve form and in `ctl model plan`; the agents carry the measurement of one core's memory rate,
|
||
without which the speed is reckoned the old way, from the whole machine's.
|
||
|
||
### Added
|
||
|
||
- **Replicas counted per profile: seven on cards and four on CPUs, under one
|
||
name.** `llmgw ctl model replicas <model> gpu=7 cpu=4`, the console's
|
||
Replicas per profile… and the mirror page's model card set how many of a
|
||
model's replicas run on each of its profiles, rather than every replica
|
||
on the best profile that fits. For a GGUF model with no profile on CPUs
|
||
one is added (`--cpu-memory`), preferred below the cards, so the router
|
||
fills the cards first and sends to the CPUs once they are full. A change
|
||
starts the replicas short before stopping those over: the model never
|
||
runs fewer. `ctl models` shows the counts; a scaling of another total
|
||
ends them.
|
||
- **Every setting of a served model from the mirror page.** A served
|
||
model's row has a Settings link, and its card now changes the context
|
||
and longest answer, the cards a replica takes, the replicas on each
|
||
profile (with the CPU's speed beside), stops and starts it, restarts its
|
||
replicas, and takes it out of the catalog with its files or without —
|
||
beside the replicas, class, owner, memory and adapters it held before.
|
||
Stopped on the row is now the model stopped and kept, started again as
|
||
it was, rather than its scaling set to nothing.
|
||
- **How fast a model answers on the CPU, before it is served.** The serve
|
||
form says, as the cores are changed, about how many tokens a second one
|
||
reply gets on the CPU and on one card of the fleet's fastest class, and
|
||
warns when it is slow (under 15) or very slow (under 5, a reply of 500
|
||
tokens taking minutes). Very slow on the CPU is refused until "run on the
|
||
CPU anyway" is ticked, GPUs free or not. `llmgw ctl model plan` and the
|
||
console's Hugging Face screen show the same figure for every way a model
|
||
can run, and `ctl model serve --cpu` says it in its log.
|
||
- **The speed follows the threads a replica is given.** Each agent now
|
||
measures what one core reads beside what all of them read: a replica of
|
||
four threads on a large server reads at about four cores' rate, not the
|
||
server's, which the estimate used to assume and overstated several times.
|
||
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
|
||
and a model of experts reads its experts' share, from the GGUF header.
|
||
Agents need upgrading to report one core's rate; until then the old
|
||
estimate is used.
|
||
|
||
## v0.16.2 — 2026-11-02
|
||
|
||
Standby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
|
||
engine for models larger than the card; a served model's longest answer changed in place; and fixes found putting
|
||
the router under load and fetching large models: a multi-file model fetched once per node however many replicas
|
||
start there, downloads that carry on where they were cut off, and a router that does twice the work on two cores.
|
||
|
||
**Upgrading.**
|
||
- No migrations. `llmgw ctl upgrade v0.16.2`: the control planes carry active-active routing (off until
|
||
`llmgw ctl config set router.active_active true`) and the router fixes; the agents carry the set fetched once
|
||
per node and the set's progress and checking; the mirror carries the resumed downloads.
|
||
|
||
### Added
|
||
|
||
- **Standbys that answer the client API themselves.** With
|
||
`router.active_active: true` in the cluster's settings, a standby control
|
||
plane routes `/v1` to the leader's replicas instead of passing every
|
||
request to the leader, so more machines carry the traffic. It hears the
|
||
replicas from the leader every two seconds over the agent port (no engine
|
||
keys travel) and reports back what it used, so the leader does not evict
|
||
a replica busy behind a standby, and the on-demand models it was asked
|
||
for. Rate limits are the cluster's: each router's request and token
|
||
counts reach the others through the leader every two seconds, so a key
|
||
spread over several control planes gets its limit, not that many times
|
||
it. Budgets are brought up to the store's every 30 seconds. Off by
|
||
default; the portal, `/admin` and `/mcp` still go to the leader.
|
||
- **A served model's longest answer, changed in place.** `llmgw ctl model
|
||
context <model> --max-output N` (and `max_output` on `POST
|
||
/api/v1/models/{name}/context`, and the console's Context length… form)
|
||
sets the longest answer on its own: the catalog's limit, in force at once,
|
||
with no replica started again. With a context it is set with it. Strata's
|
||
context can now be changed too (its setup's `--context`), so neither
|
||
needs the model removed and added again.
|
||
- **FreeToken, an engine for models larger than the card.** Mixture-of-experts
|
||
models several times a card's size, from one card, with the experts in RAM
|
||
and streamed over PCIe — read from Hugging Face safetensors checkpoints as
|
||
published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
|
||
MiniMax), with batching. Installed like vLLM on every node it runs on: a
|
||
card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
|
||
toolkit it compiles kernels with; older cards are passed over (tried on a
|
||
Turing card, where 0.2.1 has no kernels). Profiles are one card
|
||
(`engine: freetoken`); the key goes in its environment, and flags that read
|
||
files or fetch are refused. Readiness now also waits for an engine whose
|
||
`/health` answers 200 while saying `loading` in its body. FreeToken is
|
||
followed release by release: a weekly workflow fails when PyPI has a version
|
||
newer than `DefaultEngines` installs.
|
||
- **How much of each GPU works, beside whether anything does.** The busy
|
||
figure is nvidia-smi's `utilization.gpu` — the share of time any kernel
|
||
ran, which a kernel on one SM fills — and it is now said to be that: on
|
||
the dashboard, the node page, the metric's help and Grafana. Beside it,
|
||
each card's draw against its power limit (the efficiency table, the fleet
|
||
and node charts, the console's GPU work chart), and where the card's
|
||
profiling counters can be read, its SMs, SM occupancy, tensor cores and
|
||
memory interface: `llmgw_gpu_sm_active_percent`,
|
||
`llmgw_gpu_sm_occupancy_percent`, `llmgw_gpu_tensor_active_percent`,
|
||
`llmgw_gpu_dram_active_percent`. The agent reads them from NVML's GPM on
|
||
Hopper and later, and from a dcgm-exporter on the node elsewhere
|
||
(`discovery.dcgm_exporter`, default `http://127.0.0.1:9400/metrics`, `off`
|
||
for none). The GPU map colours by this "work" by default — SMs active
|
||
where read, power against the limit otherwise — and keeps busy as a mode.
|
||
On cuda01 decode read 92% busy with the SMs at work 69% of the time.
|
||
|
||
### Fixed
|
||
|
||
- **Several replicas of a multi-file model on one node fetched it once
|
||
each, over one another.** Four replicas placed on a node's four cards
|
||
wanted the same safetensors set at the same moment, each wrote the same
|
||
`.part` files, and all but one failed to start ("rename … no such file")
|
||
until a retry found the set finished. A set is now fetched once per node
|
||
and every replica waiting on it gets it, as single files always were.
|
||
- **A model of several files showed one file's download as the whole.** A node fetching a set reported the part it
|
||
was on — "59% of 46.1 GB" for a 188 GB model — instead of the set: it now reports the set's bytes against its
|
||
whole size.
|
||
- **A set's files checked with nothing said, one at a time.** After the last byte arrived a node checked each file
|
||
against the mirror's checksum, 195 GB on one core for eight minutes, while the replica still read "fetching 100%".
|
||
It now checks four at once and says so: `checking 40% of 195 GB` in `ctl replicas`, "Checking the weights" in the
|
||
chat page; and a check that takes long is not taken for a stuck download.
|
||
- **A mirror download the registry cut off started again from nothing, or not at all.** Hugging Face's CDN ended a
|
||
60 GB transfer at 23 GB (`stream error … CANCEL`), the mirror deleted what had arrived, and `ctl model serve`
|
||
failed. The mirror now tries again up to six times, carrying on from where it got to (`Range`), keeps what arrived
|
||
when it gives up, and carries on from it after a restart too.
|
||
- **A model from an outside provider took three or four requests at once.** Admission read every request in flight to
|
||
an endpoint it does not measure as a queue, so an external model (OpenAI, another llmgw cluster) was held to what a
|
||
saturation of 0.85 allows of "in flight / 4"; the rest waited and timed out with 504. An outside provider queues for
|
||
itself and says 429 when full, which its breaker already hears: it no longer counts towards saturation.
|
||
- **A valid key told to slow down under load, every thirty seconds.** When a busy key's cached entry expired, every
|
||
request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
|
||
hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
|
||
at once now share one read and one allowance.
|
||
- **A streamed answer was one write to the client a token.** Each event was flushed alone, a system call each: a
|
||
third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
|
||
waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
|
||
and the first token at 128 at once from 91 to 45 ms.
|
||
|
||
## v0.16.1 — 2026-10-30
|
||
|
||
The mirror page's Strata tab, Strata's ready-made packs served with Strata fetching them, and two fixes found
|
||
serving Strata's models: repositories that keep each size in a directory, and DeepSeek on cards its engine does not
|
||
run on.
|
||
|
||
**Upgrading.**
|
||
- No migrations. The control planes carry the tab, the plan and the serving; the agents carry the refusal of a
|
||
Strata profile its card cannot run (`llmgw ctl upgrade v0.16.1`). Mirrors need nothing new.
|
||
|
||
### Added
|
||
|
||
- **The mirror page's Strata tab.** Every family and size Strata runs, with its download, the RAM it keeps and where
|
||
it would run here; a GGUF size fetched into the mirror in one press (checked against Hugging Face's checksums),
|
||
and the packs Strata fetches itself — NVFP4, DeepSeek V4.1 — served at once, their repository the source.
|
||
`ctl model plan` and `ctl model serve` take those repositories too, and the explorer names a `strata-ple`
|
||
repository for what it is: part of a pack Strata knows, or one Strata 0.2.7 does not.
|
||
|
||
### Fixed
|
||
|
||
- **A repository that keeps each quantisation in a directory of its own** (ISTA-DASLab's `IQ2_XS/`, `IQ3_S/`) listed
|
||
no weights: its file tree is now read recursively, by the explorer, the plan and `ctl model serve`.
|
||
- **DeepSeek V4.1 offered on a card its engine was not built for.** Strata's DeepSeek engine runs on RTX 30 (compute
|
||
8.6) and newer; on a Turing card the tab and the plan offered it, and Strata's setup refused it once placed. Now
|
||
neither offers it there, the tab says why, and the agent refuses such a profile before starting it.
|
||
|
||
## v0.16.0 — 2026-10-30
|
||
|
||
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
|
||
Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
|
||
model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
|
||
are, and do not fail a node that came back on the version.
|
||
|
||
**Upgrading.**
|
||
- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
|
||
(`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
|
||
- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
|
||
fetches as before.
|
||
- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
|
||
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
|
||
they are.
|
||
- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
|
||
- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
|
||
|
||
### Added
|
||
|
||
- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
|
||
card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
|
||
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
|
||
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
|
||
RAM a size keeps its experts in.
|
||
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
|
||
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
|
||
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
|
||
of a vector down and up again, some thirty times more. `llmgw-kb` does so with a gateway that says it can
|
||
(`server_embeds` in a collection's answer); `--embed-here` keeps the old way.
|
||
- **Weights pinned to a commit, and checked against what Hugging Face published.** A model added from Hugging Face
|
||
(`hf:`, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at
|
||
then — the file's URL at that commit, or `--revision` — and each file's published sha256 goes in its origin. A node
|
||
refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for what `ctl model
|
||
serve` fetches, and `llmgw mirror add` for a Hugging Face URL (`--sha256` for others). Nodes need this version to
|
||
check; older ones fetch as before.
|
||
|
||
### Fixed
|
||
|
||
- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
|
||
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
|
||
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
|
||
- The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
|
||
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
|
||
- The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
|
||
control plane waits longer than the download may take; a second request while one runs is refused.
|
||
- A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
|
||
is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
|
||
- The console's node upgrade waits as long as the control plane, and offers no development version.
|
||
- systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
|
||
|
||
- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
|
||
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
|
||
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
|
||
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
|
||
of one. The path now reaches the leader as it was sent.
|
||
|
||
## v0.15.3 — 2026-10-29
|
||
|
||
A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
|
||
|
||
**Upgrading.**
|
||
- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
|
||
- The mirrors carry the fixes: on each mirror's machine, `sudo llmgw upgrade v0.15.3 --service llmgw-mirror`.
|
||
A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
|
||
|
||
### Added
|
||
|
||
- **A model stopped and kept, and a model removed with its files.** `llmgw ctl model stop|start MODEL`: none of its
|
||
replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
|
||
administrator stopped it. `llmgw ctl model remove MODEL --purge` (`DELETE /api/v1/models/{name}?purge=true`) also
|
||
deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
|
||
stopped; what another model uses stays. Both are in the console's model menu.
|
||
|
||
### Fixed
|
||
|
||
- **`llmgw mirror add-peer` on a mirror with a machine of its own.** It set the second mirror up and then
|
||
failed for want of the control plane's server.yaml; now it prints the `llmgw ctl config set mirror.peers`
|
||
to run on a control plane.
|
||
- **Models fetched before checksums were kept reach the second mirror.** A mirror kept in step copies only what it
|
||
can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
|
||
- **A file removed with `llmgw mirror remove` is gone from the mirror's answers too.** The running mirror kept
|
||
reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
|
||
model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
|
||
|
||
## v0.15.2 — 2026-10-29
|
||
|
||
A served model moved to more cards, or fewer, while it is served.
|
||
|
||
**Upgrading.**
|
||
- No migrations, and nothing changes until the command is used. The control
|
||
planes carry it; the nodes need nothing new.
|
||
|
||
### Added
|
||
|
||
- **A served model moved to more cards, or fewer.** `llmgw ctl model gpus
|
||
MODEL N [--profile P] [--in-place]`, `POST /api/v1/models/{name}/gpus`, and
|
||
GPUs… in the console's model menu. How many cards of one node each replica
|
||
uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
|
||
the model's KV heads, and the layers split for llama.cpp. Refused for a
|
||
count no node has, or whose cards would not hold the model and its
|
||
context, with the figures; its replicas are then started again on it, one
|
||
by one, beside the old — or, with `--in-place` when there is no room
|
||
beside them, each stopped first.
|
||
|
||
## v0.15.1 — 2026-10-29
|
||
|
||
A 125B model on one card, a second mirror, and weights that run code
|
||
refused. Strata runs as an engine: a mixture-of-experts model with its
|
||
experts in RAM, served from one 24 GB card. A second model mirror keeps in
|
||
step with the first, so an HA install has no single machine left. Pickled
|
||
weights are refused unless a model says otherwise. A served model's context
|
||
can be changed in place. The explorer says how fast a model would answer
|
||
here before it is fetched, and a card in the wrong PCIe slot is said so.
|
||
|
||
**Upgrading.**
|
||
- No migrations. Upgrade the control planes before the nodes, for the
|
||
pickle refusal below.
|
||
- A model whose weights are in a pickle format (`.bin`, `.pt`, `.pth`,
|
||
`.ckpt`, `.pkl`) stops being placed: its node refuses the source. Give it
|
||
its safetensors or GGUF, or add it again with `allow_pickle: true`
|
||
(`llmgw ctl model add -f`) if its files were checked. Look at the
|
||
models' sources before upgrading the nodes: one ending in
|
||
`.ckpt` or `.bin`, or a mirror version holding `pytorch_model.bin`. Upgrade the control planes first: a node on
|
||
this version refuses those weights until its control plane can say
|
||
`allow_pickle`.
|
||
- Strata is opt-in: nothing installs it until `engines` or an
|
||
`engine_overrides` entry names a version (`0.2.7`).
|
||
|
||
### Added
|
||
|
||
- **Strata as an engine: a 125B model of experts on one card.** Strata
|
||
(github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones
|
||
used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves
|
||
Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client;
|
||
more clients share about that, its slots buying fairness rather than
|
||
throughput on one card. It runs a few families only — Qwen3.8-Flash-Next
|
||
and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a
|
||
profile's args with `--family` and `--size`
|
||
(`--family qwen --size IQ2_XS --context 32768`). The source is the
|
||
directory of the model's GGUF files, or the repository, which Strata then
|
||
downloads itself, pinned. The agent installs it on an NVIDIA node of
|
||
x86-64: its source pinned by digest, in a Python environment, and its
|
||
release's engine checked against the digest GitHub publishes — the CUDA 13
|
||
build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal
|
||
to Ada, a V100 included, on driver 575. A replica prepares its model the
|
||
first time (minutes; its draft layer is a 5 GB download) and loads in
|
||
about a minute and a half after. One card of one node per replica, for
|
||
now; not in an offline bundle, its packages coming from PyPI.
|
||
|
||
- **A served model's context, changed.** `llmgw ctl model context MODEL
|
||
TOKENS [--slots N]`, `POST /api/v1/models/{name}/context`, and Context
|
||
length… in the console's model menu. The catalog, the engine's arguments and
|
||
the memory change together, checked against what the model was trained for
|
||
and what its cards hold; its replicas are then started again one by one. The
|
||
models API shows `context_length` and `max_output`.
|
||
|
||
- **A second model mirror, kept in step.** In a three-machine install the setup screens put one on a database machine
|
||
(`--second-mirror 1|2`). An install with one mirror gets one with `sudo llmgw mirror add-peer user@host`.
|
||
- **In step:** each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
|
||
removes what the other removed, even if it was down at the time (`llmgw mirror serve --peer`).
|
||
- **Control planes:** they name both (`mirror.peers`) and use whichever answers.
|
||
- **Nodes:** while one mirror is down, a node is given a model's file on the other.
|
||
|
||
This takes away the last single point of an HA install: with the first machine off, models are still added and
|
||
fetched.
|
||
|
||
### Fixed
|
||
|
||
- **An engine that computes in a process of its own group was not its
|
||
replica's.** A GPU replica read as running on the CPU, and the memory it
|
||
held on its card as somebody else's, which placement then kept free of
|
||
other work. A process the replica started, at any remove, is now the
|
||
replica's. And what a replica computes on, once sent, was never sent
|
||
again: a wrong reading kept from before an agent's restart stayed wrong.
|
||
It is sent again when it changes, and read again when a GPU replica was
|
||
kept as the CPU.
|
||
|
||
- **VS Code's sign-in lost its place through the sign-in page**, so the
|
||
prompt to confirm came every time; a link to the same gateway by plain
|
||
http could take an https setting down to http; and a new build of the
|
||
extension was not installed over an old one (the command now says
|
||
`--force`).
|
||
- **The install guide's recordings were not checked for secrets**, only
|
||
the page; every frame is now.
|
||
- **MCP server fields could not be emptied**, an address could carry a
|
||
password, the cluster's own server could not be changed from the
|
||
console, and a grant to a key that does not exist was said done.
|
||
- **A guardrail service two policies named was asked as the first said**,
|
||
not the stricter; a `null` answer read as safe; and with `restore` the
|
||
request log kept the answer with the redacted values back in it.
|
||
- **What an engine downloads by a Hugging Face id was not held to the
|
||
pickle refusal** — an adapter by its id, a TensorRT-LLM model's; a model
|
||
with either now needs `allow_pickle`.
|
||
- **`server init` run again on a machine already set up was refused its
|
||
memory**, as on an 8 GB install fixing its certificate with `--host`, and
|
||
judges that would not be installed (no mirror) were counted.
|
||
- **Removing an MCP server left its sealed credential in the store**, and
|
||
replacing a credential under another header kept the old one sent. A
|
||
server's secrets now go when it no longer reads them.
|
||
- **A cloned repository could point the VS Code extension at another
|
||
gateway.** `llmgw.gatewayUrl` is a machine setting now, never taken from a
|
||
workspace.
|
||
- **A node's memory bandwidth could be measured in its cache** on a busy
|
||
node, and was 0 after a reconnect until the next heartbeat. The buffer is
|
||
at least eight times the processor's last cache.
|
||
- **A project kept in the cluster could have its questions embedded outside
|
||
it.** The similar-question cache now asks the embedding model as the
|
||
question's project would, external fallbacks refused, and skips a model
|
||
that is itself outside. Its comparing no longer holds the cache's lock,
|
||
which every project's lookups waited on.
|
||
- **Restored values were left as placeholders in streamed tool calls**, in
|
||
completions and in reasoning, and a value with a quote in a whole answer's
|
||
tool-call arguments broke their JSON.
|
||
- **An outside guardrail service read what `redact` had taken out**, and a
|
||
streamed answer was sent to it twice. The docs now say a service stops a
|
||
streamed answer only under `stream: buffer_all`.
|
||
- **A replica found slow once could stay unused.** Its time to first token
|
||
is forgotten after five minutes without an answer, so it is tried again.
|
||
- **Weights that run code got past the refusal in four ways.** A mirror
|
||
version already on a node was used without being looked at again; a
|
||
`?query` or `#fragment` on an `hf:` source hid the file's name; `.pt2` was
|
||
not on the list; and a profile could write `--load_format`,
|
||
`--trust_remote_code` or an abbreviation such as `--trust-remote`, which
|
||
vLLM and argparse read as the flag, or give SGLang `--load-format`. All
|
||
refused now. TripoSR's preset says `allow_pickle`, without which it could
|
||
not be added, and a shared ComfyUI environment takes its first member's.
|
||
- **A temperature written as a string was refused.** Since the GenAI traces
|
||
read it, `"temperature": "0.7"` got a 400 before reaching the engine; it
|
||
is read loosely again, for the trace only.
|
||
- **A reused answer could carry another caller's personal data.** With
|
||
`restore: true` and the answer cache on, two callers' requests that
|
||
redacted to the same text shared one answer — with the first caller's
|
||
address put back in it. Such requests are now never answered from the
|
||
cache nor kept in it.
|
||
- **Granting a server's tools could take others away; revoking could close
|
||
a level to every tool.** A level with no tool list sees what the levels
|
||
above grant; giving it its first list made it see only that list. And a
|
||
level left with an empty list was closed to every tool, which the screens
|
||
did not show. Now a first list starts from the nearest list above, taking
|
||
a server away leaves the rest or goes back to no list, the screens say
|
||
when a level above keeps granted tools out of reach and which levels are
|
||
closed, and the console reads and writes the level at once rather than
|
||
from what it loaded.
|
||
- **Similar questions about different pictures got each other's answer.**
|
||
A question with a picture, a sound or a file in it is reused only when it
|
||
is the same request.
|
||
|
||
### Added
|
||
|
||
- **A card in the wrong PCIe slot is said so.** Each agent reads how every
|
||
card is linked — the fastest generation card and slot allow together, and
|
||
the lanes in use — against the card's own most, from the kernel (the
|
||
driver's figure is what card and slot allow together); a card on fewer lanes or
|
||
an older generation is a node warning, with what it carries and what it
|
||
could, on the dashboard, the console and in `llmgw doctor`.
|
||
- **And on a web page, `/admin/mcp`.** For administrators: the same list,
|
||
adding a server with its credential sealed, testing it and listing its
|
||
tools, granting them in the tree beside that level's other grants and
|
||
limits, revoking, changing, replacing the credential, turning it off,
|
||
removing; and the line that points an agent at this gateway's `/mcp`.
|
||
In English and Turkish; every change in the audit log, the secret not.
|
||
- **The company's MCP servers in the console.** A screen of its own: each
|
||
server with its state, who has its tools, its rate, the names of the
|
||
credentials it is sent and its address; add one with a bearer token or
|
||
a header kept sealed, test it and list its tools, grant them to a team,
|
||
project, organization, person or key beside that level's other grants,
|
||
revoke, change, replace the credential, turn off, remove.
|
||
- **And on the cards, as this cluster would lay it out.** The explorer's
|
||
**GPU tok/s** and a repository's files give one reply's speed for each
|
||
layout the planner offers here — one card, tensor parallel across a
|
||
node's cards, llama.cpp's layer split, a pipeline across nodes on a
|
||
fabric — the fastest of them, with the layout in the tooltip. From each
|
||
card's memory bandwidth by its model name, NVLink or PCIe between a
|
||
node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed
|
||
classes is judged class by class. A model of experts reads 1.6 times its
|
||
active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144
|
||
tok/s; qwen3-vl-8b Q8: 62).
|
||
- **How fast a model would answer on a CPU, before it is fetched.** Each
|
||
agent measures how fast its node's memory reads when it starts (`Host`
|
||
gets `mem_bandwidth_gbs`, the node's view `mem_bandwidth_gbs`); the
|
||
explorer's **CPU tok/s** column and a repository's GGUF files say about
|
||
how many tokens a second one reply gets on the fastest node that holds
|
||
the model — the bandwidth over the bytes a token reads, its active part
|
||
for a model of experts. Calibrated on the PoC against llama.cpp's own
|
||
timings: within 15% from 1.5B to 14B.
|
||
- **The explorer lists what this cluster can run.** Judged by the nodes
|
||
as they are now — every card's memory and compute capability, every
|
||
node's RAM — through the same planner as serving a model, rather than a
|
||
size picked from a list. A model too big for any node, in FP8 without a
|
||
card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and
|
||
counted; the page says what it judged by. With cards, safetensors repositories are
|
||
listed beside GGUF. On a repository's page a file too big for the
|
||
cluster is shown with what it needs and no Fetch button.
|
||
- **Weights that run code are refused.** Python pickles — `.bin`, `.pt`,
|
||
`.pth`, `.ckpt` — run whatever they hold when loaded, which is how the
|
||
malicious models on Hugging Face work. The node refuses such a source,
|
||
or a mirror version with one in it, before fetching it; adding such a
|
||
model is refused; vLLM and SGLang downloading a model by its id read its
|
||
safetensors only; the mirror page marks the file. `allow_pickle: true`
|
||
on a model lets it through on purpose. The mirror's explorer no longer
|
||
lists a repository with nothing a node loads (no GGUF, no safetensors)
|
||
while "runs here" is ticked, and a repository's pickle files are not
|
||
offered, only counted. BACKLOG M lists what is next:
|
||
checksums against Hugging Face everywhere, its own scan, trusted
|
||
publishers, engines without the internet.
|
||
|
||
## v0.15.0 — 2026-10-25
|
||
|
||
The company's tools and editors through the gateway, and guardrails that
|
||
can lean on others. Its own MCP servers sit behind llmgw's `/mcp` under the
|
||
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
|
||
extension the gateway serves. A guardrail policy can ask an outside
|
||
service, and can put redacted values back in the answer. Chat goes to the
|
||
replica answering soonest. A similar question can get the answer already
|
||
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
|
||
GenAI conventions, and Kong is written down from trying it. Also an install
|
||
guide with every screen, at `/install`.
|
||
|
||
**Upgrading.**
|
||
- Migrations 0049 (`mcp_servers`), 0050 (their rate and breaker) and 0051
|
||
(`projects.cache_similar`), applied when the control plane starts.
|
||
- `llmgw server init` now refuses a machine that would serve models without
|
||
the memory for them: the starter model and the judges need 12 GB on one
|
||
machine, where 8 GB was said before. `--no-model`, `--no-guard-model` or
|
||
`--no-node` make it fit. A running cluster is not checked again.
|
||
- Traces carry the GenAI conventions' `gen_ai.*` attributes beside the
|
||
`llmgw.*` ones, which stay; docs/install.md, "Tracing", lists them.
|
||
- Nothing else changes until it is turned on: `services` and `restore` in a
|
||
guardrail policy, `keys cache --similar` with
|
||
`router.cache.embedding_model`, `cached_per_1k` in a price.
|
||
|
||
### Added
|
||
|
||
- **Answers to similar questions.** `llmgw keys cache --similar 0.95` also
|
||
gives a chat question the answer already given to one like it in the same
|
||
conversation, compared by `router.cache.embedding_model`, as
|
||
`x-llmgw-cache: similar` with the similarity. Held in memory beside the
|
||
exact answers; nothing written to the store but the setting.
|
||
- **A cached prompt costs the cached rate.** `cached_per_1k` in a price
|
||
charges the prompt tokens an engine or provider read from its cache (the
|
||
answer's `cached_tokens`) at their own rate; without it they cost what any
|
||
prompt token does, as before.
|
||
- **Chat goes to the replica answering soonest.** The router keeps each
|
||
replica's time to first token, smoothed over its last answers, and the
|
||
`chat` profile scores by it beside the queue and the KV cache: of two
|
||
replicas of one model, the one on the slower card or the busier host gets
|
||
less, and a replica not yet measured is tried.
|
||
- **llmgw and Kong, written down** ([docs/kong.md](docs/kong.md)): Kong 3.9
|
||
in front of llmgw by a plain route or its `ai-proxy`, each team's llmgw
|
||
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
|
||
no completion tokens of a streamed answer, so bill from llmgw's usage.
|
||
- **A guardrail that is someone else's service.** A policy's `services`
|
||
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
|
||
HTTP: a body template, keys from files, dotted paths to where the answer
|
||
says unsafe and why, on input, output or both, block or flag, and the
|
||
policy's `fail` when it does not answer.
|
||
- **Redacted values put back in the answer.** `restore: true` on a guardrail
|
||
policy's input: the model reads `[EMAIL_1]`, the client reads its own
|
||
address in the reply, streamed or not. Only the request's own values, held
|
||
in memory for its life; the output rules read the answer as the client
|
||
will.
|
||
- **Traces in the OpenTelemetry GenAI conventions.** An inference request
|
||
and each attempt at an engine carry `gen_ai.operation.name`, the requested
|
||
and the answering model, `max_tokens`, `temperature` and `top_p` when set,
|
||
the session as `gen_ai.conversation.id`, and the input, output and cached
|
||
tokens, and an attempt the engine that answered as `gen_ai.provider.name`.
|
||
Tools that read the conventions show llmgw's calls as model calls with no
|
||
dashboard of their own. No prompt or answer text is ever in a span.
|
||
- **llmgw for VS Code.** An extension of llmgw's own, served by every gateway
|
||
at `/setup/vscode/llmgw.vsix` with that gateway's address in it, made from
|
||
the binary itself: plain JavaScript, no dependencies, no build, and so no
|
||
marketplace needed at a site with no internet. It offers the gateway's chat
|
||
models to VS Code's language-model API — the chat's model picker, agent
|
||
mode, and every extension that asks VS Code for a model — with the sizes
|
||
and the tool and vision capabilities the catalog says, streamed, with tool
|
||
calls and their results both ways. **VS Code** is a tool on the account
|
||
page's **Connect your editor**: a key, the one-line install, and an **Open
|
||
in VS Code** link that hands the key to the extension, which keeps it in
|
||
VS Code's secret storage. **llmgw: Sign In** opens that page with a state
|
||
of its own; a link without it is confirmed first, the gateway named.
|
||
- **An install guide with every screen, at `/install`.** From three bare
|
||
Ubuntu machines to a first answer: what each machine needs and how to
|
||
check it, the release fetched and checked, the setup screens one by one,
|
||
the system's state, and a first request with a key from the account page.
|
||
The screens are a real three-machine install, recorded and kept as
|
||
terminal frames — sharp at any size and a few kilobytes — with each
|
||
session played back on request; the administrator's password and every
|
||
key were masked before it was kept. `/about` says more of what llmgw does
|
||
(documents, guardrails, sign-in, high availability, pictures and 3D), and
|
||
its install steps are the ones of today, leading to the guide.
|
||
- **The company's MCP tools under the guardrails, a rate and a breaker.**
|
||
- **The answer:** the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
|
||
ticket is withheld as a pasted one would be, and personal data is redacted.
|
||
- **The arguments:** they are read before they leave, so a secret pasted into one does not reach the company system.
|
||
- **Rate:** `ctl mcp add --rate N` allows each caller N calls a minute to a server.
|
||
- **Breaker:** five failures in a row take a server out for a minute, its tools off the list.
|
||
|
||
- **How fast the judge is, measured.** `llmgw ctl guard judge check` (and `POST /api/v1/guard/judges/check`) works
|
||
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
|
||
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
|
||
and what to do: more cores, more judges, a GPU judge, a longer wait, or `fail: open`. `server init` runs it once the
|
||
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.
|
||
|
||
- **The company's own MCP servers through llmgw's `/mcp`.** `llmgw ctl mcp add jira --url … --secret-file
|
||
Authorization=FILE` registers one; `/mcp` then lists its tools as `jira.search` beside llmgw's own. Only keys granted
|
||
them see them: `llmgw keys limits --tools 'jira.*'`, patterns at any level of the tenancy tree. None granted anywhere
|
||
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
|
||
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
|
||
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
|
||
admin API has `/api/v1/mcp/servers`.
|
||
- **The cluster as a read-only MCP server.** Registered as `llmgw ctl mcp add llmgw --url llmgw:admin` and granted
|
||
as `llmgw.*`, its tools say what `llmgw ctl` does:
|
||
- `cluster_status`, `list_nodes`, `list_models`, `list_replicas`, `explain_placement`, `node_logs`;
|
||
- nothing that changes the cluster.
|
||
|
||
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
|
||
The admin API serves the same at `POST /api/v1/mcp`. `ctl mcp add --bearer-file` keeps a token, sent as
|
||
`Authorization: Bearer`.
|
||
|
||
- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's
|
||
`--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model
|
||
that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each
|
||
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
|
||
shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that
|
||
a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on
|
||
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.
|
||
|
||
- **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after
|
||
the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled
|
||
`llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
|
||
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
|
||
of their own, 2 were. A node with an `llmgw.role` label takes only profiles whose node selector asks for that role,
|
||
so no other model is placed on the judges' machine.
|
||
|
||
### Fixed
|
||
|
||
- **A machine too small for the models it would serve is said so first.**
|
||
The control plane that is also a node, serving the starter model and two
|
||
judges, needs about 10.5 GB. On 8 GB the install finished every step and
|
||
then had no room for the starter. Now the setup's checks fail its memory
|
||
with what it would serve, and `server init` refuses before it installs
|
||
anything, naming `--no-model`, `--no-guard-model` and `--no-node`.
|
||
|
||
- **Turkish pages are Turkish to the end.**
|
||
- **Errors:** those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
|
||
catalog", the settings changed meanwhile, and others) are now in the page's language.
|
||
- **Percent:** a percentage is written the language's way: `%93` in Turkish, `93%` in English.
|
||
- **Footer:** the time at the foot of a page is in the language's date form.
|
||
- **Stale forms:** they are refused with a sentence in that language rather than "bad request".
|
||
|
||
- **`llmgw keys limits` no longer clears the limits it was not given.** Setting one limit at a scope wrote the whole
|
||
row, so `--tools 'jira.*'` alone took away that scope's rate, budget and model list. Only what is given changes now.
|
||
`--replace` writes every limit, as the console's limits form does.
|
||
|
||
- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run
|
||
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
|
||
offered, as is one that divides them.
|
||
- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so
|
||
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
|
||
The timeout now runs from when a node's download last moved.
|
||
|
||
- **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the
|
||
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
|
||
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
|
||
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.
|
||
- **A judge the leader serves by itself is sized as init's is.** When a guard policy names a judge the catalog lacks,
|
||
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
|
||
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and `fail: closed` refused the
|
||
request. It now gets what `server init` gives its judges: 2 GB, two cores each (fewer on a small node), and no
|
||
prompt cache. Every judge served through `modelserve` runs with `--cache-ram 0`.
|
||
|
||
## v0.14.9 — 2026-10-19
|
||
|
||
Sign-in set up from the console, with Active Directory tried before it is
|
||
kept. A node's console now shows what the machine is doing, and a node's web
|
||
page shows what it runs. Also fixes found trying Active Directory against a
|
||
Samba domain controller on the test cluster. Sign-in through it, nested
|
||
groups and a disabled account suspended by the sync all worked there.
|
||
|
||
**Upgrading.**
|
||
- No migration.
|
||
- `auth.ldap` and `auth.oidc` are in the cluster's settings, as before. A
|
||
block written into `server.yaml` on a cluster with a store was never
|
||
read; the log said so. Put it in with `llmgw ctl config edit`, or set it up
|
||
again from the console's Sign-in screen.
|
||
- A Samba DC's own autogenerated certificate is refused (a negative serial
|
||
and no SAN). Give it one from a CA.
|
||
|
||
### Added
|
||
|
||
- **Sign-in set up from the console.** Under People → Sign-in, `sudo llmgw` shows how people get in: single sign-on,
|
||
Active Directory and SCIM, each on or off. It also shows the directory's last sync. **E** sets up Entra ID from the
|
||
tenant and client IDs. **D** sets up Active Directory, and the directory is tried before it is kept: each server
|
||
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
|
||
password, client secret and SCIM token are kept in the store as `store:auth/…`, sealed by the master key. The
|
||
domain's CA goes into the settings (`auth.ldap.ca`), so no file has to be put on each control plane. The admin API
|
||
has the same: `GET /api/v1/auth`, `POST /api/v1/auth/ldap/test` and `PUT /api/v1/auth/secrets/{name}`.
|
||
- **A node's page says what it runs.** It now shows the agent's version, marked when the control plane runs another,
|
||
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
|
||
GPUs.
|
||
- **`sudo llmgw` on a node opens on what the machine is doing.** A new Overview screen shows the agent's state and
|
||
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
|
||
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
|
||
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
|
||
|
||
### Fixed
|
||
|
||
- **A broken `auth.ldap` no longer stops every control plane.** It is the cluster's setting. A CA file that one
|
||
machine could not use made `llmgw server` exit on every control plane, and no `ctl config` could then put it right.
|
||
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
|
||
- **A second `llmgw server` on a machine stops before it competes to lead.** `llmgw server check` (no such command)
|
||
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
|
||
control planes stood by. `server` now refuses words it does not know and takes its ports before anything else.
|
||
- **The single sign-on and Active Directory guides said `server.yaml`.** On a cluster with a store, `auth` is in the
|
||
cluster's settings and the file's block is not read. They now say `llmgw ctl config edit`, and what certificate a
|
||
domain controller needs.
|
||
|
||
## v0.14.8 — 2026-10-18
|
||
|
||
Sign-in with an on-premises Active Directory, and Entra ID made safe for
|
||
people in many groups. Also several keys for one outside provider, and a
|
||
measure of what the gateway adds to a request.
|
||
|
||
**Upgrading.**
|
||
- Migration 0048 adds the `user_directory` table, applied when the control
|
||
plane starts.
|
||
- Nothing changes until `auth.ldap` is set; docs/install.md, "Active
|
||
Directory", has an example. It has been tested against a fake directory,
|
||
not yet against a real domain controller.
|
||
- A judge installed by an earlier `server init` is refitted by the leader
|
||
by itself (below); the steps given for v0.14.7 are no longer needed.
|
||
|
||
### Added
|
||
|
||
- **Sign-in with Active Directory** (`auth.ldap`). People sign in on the gateway's form with their AD username or mail
|
||
and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
|
||
staff roles and places in the tree, as with OIDC. Accounts are tied to the `objectGUID` and made at first sign-in
|
||
when `create_accounts` is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
|
||
suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
|
||
more than half the accounts is refused. See docs/install.md, "Active Directory".
|
||
|
||
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
|
||
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
|
||
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
|
||
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
|
||
|
||
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
|
||
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
|
||
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
|
||
It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
|
||
gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
|
||
|
||
### Fixed
|
||
|
||
- **Entra ID's groups past 200 no longer take a role away.** For someone in more than 200 groups Entra sends none in
|
||
the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
|
||
sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
|
||
registration. docs/install.md has an Entra ID section.
|
||
- **An account made by SCIM or OIDC could not be reinstated.** Setting it active was refused as "never approved".
|
||
- **An upgrade refits the judge an earlier `server init` installed.** Upgrading to v0.14.7 left such a judge as it was:
|
||
no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
|
||
fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
|
||
prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
|
||
replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
|
||
and the log says so once.
|
||
- **A restart is no longer refused for room another restart is giving back.** With every core of a node reserved, two
|
||
models restarted together, one to need more and one less (as after `ctl model resources` on a judge and a starter),
|
||
had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
|
||
its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
|
||
replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
|
||
another no longer wait out the five-minute grace each.
|
||
|
||
## v0.14.7 — 2026-10-16
|
||
|
||
The judge installed by `server init` sized for what it does. On a machine
|
||
of fewer than eight cores v0.14.6 left every judgment past its budget, so
|
||
`fail: closed` refused every request. A single-machine install of v0.14.7
|
||
on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.
|
||
|
||
**Upgrading.**
|
||
- No migration.
|
||
- **Upgrading does not change a judge already in the catalog.** v0.14.6
|
||
and earlier installed two judges, reserving no cores or one, and 2 or
|
||
4 GB. To give them what they need:
|
||
`llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2`, then
|
||
`llmgw ctl restart --model qwen3guard-0.6b`. The two then need four cores
|
||
on a single machine; below that, a longer `latency_budget` or
|
||
`fail: open`.
|
||
- On a node whose cores are all reserved, `ctl restart --model` cannot
|
||
start the new replica beside the old one and waits. Stop the old engine
|
||
process, and the replica comes back with the new figures.
|
||
|
||
### Fixed
|
||
|
||
- **Both judges fit an 8 GB machine again.** v0.14.6 had `server init` reserve 4 GB a judge replica. That figure came
|
||
from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
|
||
which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
|
||
nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with `--cache-ram 0`, uses about 1.6 GB and
|
||
stays there, and reserves 2.
|
||
- **One machine runs one judge, on up to four cores.** A second replica on the same machine shares its cores and is
|
||
lost with it, so it bought nothing but half the speed of each. `server init` on a single machine now serves one judge
|
||
on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
|
||
on two cores each, as in HA setups. `--judge-replicas` and the setup's Copies still choose.
|
||
- **A judge gets the two cores it needs.** v0.14.6 gave each judge replica one core on machines under eight. A judgment
|
||
of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
|
||
machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
|
||
cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
|
||
shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
|
||
`latency_budget`, where requests must not wait.
|
||
|
||
## v0.14.6 — 2026-10-16
|
||
|
||
Setup that says what each part of an install is and lets every one be
|
||
chosen and set; three machines that each run the gateway, so any one can
|
||
be lost; a cluster that loses its store's primary without losing a
|
||
request; one ComfyUI for an environment's models; pictures, 3D and
|
||
workflows as MCP tools; alerts asked for at setup; extended thinking.
|
||
|
||
**Upgrading.**
|
||
- No migration.
|
||
- **Control planes**: with `control.lease_ttl` unset the lease is now 10 s
|
||
(was 15). Upgrade the standbys first and the leader last; a mix of
|
||
versions meanwhile is fine.
|
||
- **The store** (PostgreSQL): each member is dialled for 3 s before the
|
||
next is tried. A store so far away that connecting takes longer needs
|
||
`connect_timeout=10` in its address.
|
||
- **Nodes** notice a control plane that died without closing its
|
||
connection in 8 s (was 15) once they run this release.
|
||
- **Judges installed by an earlier `server init`** keep reserving no cores
|
||
and 2 GB. To give them their own:
|
||
`llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2`, then
|
||
`llmgw ctl restart --model qwen3guard-0.6b`.
|
||
- **Patroni** replaces a lost primary within 20 s on a new cluster. On an
|
||
existing one, run `server init --patroni ...` again from the machine with
|
||
the authority; it changes only that.
|
||
- **A floating address** keeps VRRP router id 51 until `server add --vip`
|
||
runs again, which moves it to one of its own; run it for every standby.
|
||
- A request whose judge is not there yet waits up to the policy's
|
||
`latency_budget` for one before `fail: closed` refuses it.
|
||
|
||
### Added
|
||
|
||
- **Setup lets every part be chosen, each with its settings, and says what each is.** `sudo llmgw` on a new machine lists
|
||
the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model,
|
||
pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline
|
||
bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens
|
||
its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or
|
||
any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's
|
||
categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the
|
||
default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is
|
||
left out, with the same install as a command. Every setting is a `server init` flag too (`--starter`,
|
||
`--judge-fail`, `--media`, `--max-key-ttl`, ...).
|
||
- **Three machines are high availability for the gateway, not only the store.** With three machines, setup makes the two
|
||
database machines control planes too, standing by while the first leads, with an optional floating address
|
||
(keepalived) that follows whichever leads. Any one machine can now be lost. By command: `server init --patroni ...
|
||
--standbys --vip 10.0.0.160/24`. The mirror's management API answers the three control planes, so a standby that
|
||
comes to lead can still fetch models.
|
||
|
||
- **Pictures uploaded to work from**: `/v1/files` takes `purpose=vision` or
|
||
`user_data` besides `batch`, kept as long as job results, so a picture is
|
||
sent once and named by its `file-…` id in the MCP tools and a workflow's
|
||
inputs.
|
||
- **One ComfyUI for an environment's models**: ComfyUI models in one
|
||
`comfyui.environment` (two or more) share one replica — the catalog
|
||
makes it a hidden entry, `comfyui-env-<name>`, on the first member's
|
||
profiles with the most memory any member needs, carrying every member's
|
||
weights and custom nodes. Asking for any member starts it; it answers for
|
||
each with that member's workflow and checkpoint, and the members get no
|
||
replicas of their own. A picture model and a 3D model on one card stop
|
||
handing it to each other with a cold start each way.
|
||
- **Pictures, 3D models, vision and workflows as MCP tools** at `/mcp`,
|
||
beside the memory notes, each offered only where the gateway has a model
|
||
for it: `generate_image` (with `concept`, `side` and `texture` styles),
|
||
`generate_mesh` (from a picture or a prompt), `review_images` (a vision
|
||
model, to a JSON schema), `run_workflow`, `job_status` and
|
||
`gateway_status`. Each makes its request under the caller's key, so
|
||
allowlists, limits, licences and budgets apply and it is charged like
|
||
any request; what it makes is kept as the caller's file and answered by
|
||
id and download URL. A client that takes a stream gets
|
||
`notifications/progress` while a mesh is made and waits up to ten
|
||
minutes; one that does not, 50 seconds, then the job id.
|
||
- **Setup asks where alerts go**: `llmgw server init --webhook URL`
|
||
(repeatable) and a **Notification webhooks** field on both setup screens.
|
||
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
|
||
any other the event as JSON (`slack:` or `json:` in front says which).
|
||
They are added to the cluster's `notify.webhooks` beside those already
|
||
there, and each is sent a test once the control plane runs on them.
|
||
|
||
- **Extended thinking on `/v1/messages`**: `thinking: {type: "enabled"}`
|
||
tells a model that thinks by its chat template to (`enable_thinking`),
|
||
and its reasoning comes back as `thinking` blocks before the text,
|
||
streamed as `thinking_delta` events; `disabled` tells it not to. Not
|
||
asked for, the reasoning is left out as before.
|
||
- **How long on-demand models hold a card** is a setting, in force on the
|
||
leader within seconds: `on_demand.evict_after_idle` (2m unless set) and
|
||
`on_demand.min_lifetime` (5m). On one card shared by a picture model and
|
||
a 3D model, 1m each brings the wait between a picture and its mesh to
|
||
about a minute.
|
||
|
||
### Fixed
|
||
|
||
- **The judges and the starter model divide a machine's CPUs instead of each taking all of them.** `server init` gave the
|
||
judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter
|
||
on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a
|
||
node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge
|
||
replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A
|
||
judge also reserves 4 GB, not 2: it uses about 3.2.
|
||
|
||
- **A lost store machine stops costing requests most of a minute.** The store's connections try each member for 3 seconds,
|
||
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
|
||
keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
|
||
5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
|
||
VMs with the primary's power cut: about 45 seconds of hanging requests before.
|
||
- **Leadership stays put while the store's primary is replaced.** Replacing a lost primary takes longer than the
|
||
leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
|
||
took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
|
||
round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
|
||
the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
|
||
8 seconds (was 15), so it is on its way to the new leader as that one takes over.
|
||
- **A judge replica that hangs no longer costs the request.** All the tries of a judge request shared the policy's
|
||
budget, so a replica that hung used it all and the try on another replica had none left: under `fail: closed` the
|
||
request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same
|
||
question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut
|
||
short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken
|
||
are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before.
|
||
On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge
|
||
on a GPU.
|
||
- **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
|
||
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
|
||
so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
|
||
judge that is slow is not asked again, and an alarm is raised once per request, not once per try.
|
||
|
||
- Node discovery for Prometheus listed only the control plane asked and the
|
||
store, so a machine that is both was a control plane to one and a store
|
||
member to the other — scraped twice — and a standby asked alone lost every
|
||
node. Every control plane lists all of them alike; deploy/monitoring's
|
||
prometheus.yml says to scrape and ask each.
|
||
- A model whose weights and context pool did not fit its card was added
|
||
and then failed to make its context on every start, taking the card from
|
||
others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on
|
||
24 GB). `ctl model serve` shrinks the pool to what the card holds, saying
|
||
so, and refuses weights that do not fit at all; the portal's serve form
|
||
refuses such a pool and says the largest that fits.
|
||
- Removing weights from the mirror that a catalog model still loads is
|
||
refused, naming the model: removed from the mirror alone, the model stayed
|
||
in the catalog and its replicas went on loading the node's copy.
|
||
- Jobs ran one at a time, so a 3D job waiting for the card held up another
|
||
model's job that could have run. They run side by side, one per model at
|
||
a time and eight in all, on the leader as before.
|
||
- `llmgw ctl model serve` served an embedding model as a chat model: no
|
||
`--embedding`, four slots, a 512-token batch, `embeddings` unset and a
|
||
"say hello" smoke test it could not pass. A model whose GGUF says it
|
||
pools (or `--embeddings`) is served for embedding: 32 slots of 1024
|
||
tokens unless sized otherwise, a batch that holds a whole passage, and
|
||
one vector asked for as its smoke test.
|
||
- `llmgw ctl upgrade` checks every node's version once the waves are done
|
||
and fails naming any not on the release, and says when the control plane
|
||
has no nodes registered; `llmgw upgrade` on a control plane says how many
|
||
nodes it left behind and the command that upgrades them. A site's nodes
|
||
stayed a release behind with nothing to say so.
|
||
- The cluster and node pages swapped their live part every few seconds,
|
||
forms and all, so filling one in raced the clock: what was typed, an
|
||
open choice and the focus were lost. The refresh now waits while a field
|
||
in it has the focus or a form in it has been typed into and not sent;
|
||
the mirror page's reload during a download waits the same way.
|
||
|
||
## v0.14.5 — 2026-10-14
|
||
|
||
Picture and 3D models that add themselves where a card fits them, with a
|
||
card on the portal and a screen in the console to see and add them; a
|
||
mesh's colour brought to the picture it was made from; and the whole
|
||
licence written into every mesh.
|
||
|
||
**Upgrading.**
|
||
- The store gains a table (migration 0047: preset_state); nothing to do.
|
||
- **`presets.auto` is on**: once upgraded, the leading control plane adds
|
||
flux-schnell and triposr, on demand, where a node's card fits one, the
|
||
cluster installs ComfyUI (`engines.comfyui`) and the catalog makes
|
||
nothing of that kind. Nothing is fetched until somebody asks for one.
|
||
Turn it off with `llmgw ctl model presets auto off`.
|
||
- **Nodes** need this release for `albedo_match` (it is in the save node
|
||
the gate puts in each replica); a ComfyUI replica takes it when it
|
||
starts again.
|
||
|
||
### Added
|
||
|
||
- **A mesh's base colour brought to its picture's**: `LLMGWSaveMesh` takes
|
||
`albedo_match` with the picture and its mask, and scales the texture in
|
||
linear light to the object's brightness in the picture. Stable Fast 3D
|
||
takes the light out of what it sees and read a dark basalt (0.17) as a
|
||
light stone dimly lit (0.67); matched, it is 0.17. On sf3d it is the
|
||
`albedo_match` input (off unless asked).
|
||
- **Picture and 3D models by themselves**: once a node has a card that fits
|
||
flux-schnell or triposr and the catalog makes nothing of that kind, the
|
||
leading control plane adds the preset on demand — so a new cluster's
|
||
portal has its Images and 3D pages without a definition written — where
|
||
the cluster installs ComfyUI. `presets.auto` turns it off; a preset
|
||
somebody removes is not added back. `GET /api/v1/presets` says how each
|
||
stands and why one is not added; `POST /api/v1/presets/{name}` adds one
|
||
(and sets `engines.comfyui` if the cluster has none); `PUT
|
||
/api/v1/presets/auto`. Each automatic addition is in the audit log.
|
||
- **Presets on the portal and in the console**: the cluster page's
|
||
**Pictures and 3D models** card lists each preset — what it makes, the
|
||
classes it fits here, whether and by whom it was added, why the leader
|
||
has not — with an **Add** button that says first when it would install
|
||
ComfyUI and restart the control planes, and the switch for `presets.auto`.
|
||
The console's models screen shows the same; **A** adds one, **P** flips
|
||
the switch. `ctl model add PRESET` and `ctl model presets` go through the
|
||
API (and `ctl model presets auto on|off`), falling back to the entry as
|
||
v0.14.4 sent it on an older control plane.
|
||
|
||
### Fixed
|
||
|
||
- A mesh carried only its licence's name, URL and notice; it carries the
|
||
whole licence as `/v1/models` gives it — id, commercial use, excluded
|
||
regions, revenue cap, attribution — for a pipeline that sorts assets by
|
||
licence.
|
||
- A workflow's `{{name|false}}` default filled a BOOLEAN input with the
|
||
word rather than a boolean.
|
||
|
||
|
||
## v0.14.4 — 2026-10-13
|
||
|
||
Any ComfyUI workflow as a job, with typed inputs; picture and 3D models
|
||
added by name; profiles that fit whatever card a site has, and ComfyUI on
|
||
a V100; licences a machine can match and a person can leave out; a job
|
||
waiting for the one card says what holds it. Many fixes to on-demand
|
||
models taking turns on one card.
|
||
|
||
**Upgrading.**
|
||
- The store gains a table (migration 0046: user_excluded_licences);
|
||
nothing to do.
|
||
- **Nodes** need this release for ComfyUI on Volta or Pascal cards (the
|
||
cu126 build), for a driver from NVIDIA's open kernel module to be read,
|
||
and for a node that loads its weights by repository name (sf3d); a
|
||
ComfyUI already installed is kept.
|
||
- **Control planes before nodes**: an entry using the built-in `triposr`
|
||
workflow, or a `gpu:` need instead of a class, is refused by an older
|
||
control plane.
|
||
- Models added with `llmgw ctl model add <preset>` take the preset's name;
|
||
remove an entry of that name first.
|
||
|
||
### Added
|
||
|
||
- **The client API described**: OpenAPI 3.1 at `/v1/openapi.json`, no key
|
||
needed, generated from a route table a test holds to the router's own;
|
||
and `/docs/api`, Swagger UI (served from the gateway) over it and, for
|
||
staff, over the admin API's document.
|
||
- **A job waiting for the card says so**: `progress` is
|
||
`{"state":"waiting_gpu","held_by":…,"eta_seconds":…}` while another
|
||
on-demand model holds it; a direct request's 503 carries
|
||
`X-Llmgw-Waiting-For` and `X-Llmgw-Eta-Seconds`.
|
||
- **Any ComfyUI workflow, as a job**: `POST /v1/workflows/runs` with
|
||
`{model, inputs}`. A model's catalog entry declares its inputs
|
||
(`comfyui.inputs`: text, int, float, bool, choice, image, mask, audio,
|
||
video, file, with bounds and defaults) and its output (`comfyui.output`:
|
||
pictures, meshes, video, audio or files); a request is checked against
|
||
them before a job is written, file inputs are base64, a data URL or a
|
||
`/v1/files` id, and `/v1/models` lists each model's `workflow` so a
|
||
client can build its form. A PNG result carries the licence in an
|
||
`iTXt` chunk, a GLB in `asset.extras` as before.
|
||
- **Environments**: `comfyui.environment` names ComfyUI models that agree
|
||
on their custom nodes and may share one replica; the catalog refuses
|
||
one that cannot.
|
||
- **Profiles that fit the hardware**: a profile may say `gpu:
|
||
{min_compute_capability, min_memory_gb}` instead of `gpu_class`, and
|
||
stands for every class in the cluster that meets it, as
|
||
`<profile>@<class>`; a ComfyUI profile may carry its own `comfyui`
|
||
workflow, checkpoint and nodes, and any profile its own `source`. One
|
||
catalog entry then runs fp8 where the card has it and fp16 on Turing.
|
||
- **Custom nodes that compile**: `nodes[].build` names directories of a
|
||
node that are C++/CUDA packages; the gate builds each as a wheel for the
|
||
card the replica runs on, keeps it on the node, and installs it beside
|
||
the replica. `comfyui.huggingface_repo` shows a model's weights to a node
|
||
that loads them by repository name.
|
||
- **A licence a machine can match**: `license.id`, `commercial`,
|
||
`excluded_regions`, `revenue_cap_usd`, `attribution`.
|
||
- **On-demand models in `/v1/models`**, with `loaded` saying whether one is
|
||
up now; a picture is cut out and put on grey before TripoSR sees it
|
||
(`LLMGWPrepareForMesh`, rembg when it has no alpha).
|
||
- **Licences a person leaves out**: on the profile page, by licence, or by
|
||
rule — every licence forbidding commercial use, every licence granting
|
||
nothing in a region — so a model under one is neither listed to nor
|
||
served for that person's keys and sessions (403 `licence_excluded`).
|
||
- **Presets: a picture or 3D model added by name.** `llmgw ctl model add
|
||
flux-schnell` or `triposr` adds an entry that comes with llmgw — its
|
||
workflow (TripoSR's now built in), custom nodes, packages and licence,
|
||
and a profile by card need rather than class — and has the nodes install
|
||
ComfyUI when the cluster's settings name none. `llmgw ctl model presets`
|
||
lists them; the console's **A** on the models screen offers them beside
|
||
a definition file.
|
||
|
||
### Fixed
|
||
|
||
- A model waiting for another on-demand model to give up the card failed at
|
||
once as "cannot be placed"; it now waits, and a job with it.
|
||
- A model evicted for being idle was placed again on the next pass, ahead of
|
||
the one it made room for.
|
||
- A job or batch leased by a standby control plane, whose router has no
|
||
endpoints, waited on "loading" for good; they are served only by the
|
||
leader. A job told its model fits nowhere asks again for five minutes
|
||
before failing, which an agent re-registering makes true for seconds.
|
||
- An on-demand replica adopted after a restart was never idle enough to
|
||
evict, so a model waiting for its card waited out its whole idle timeout;
|
||
it counts as used at adoption.
|
||
- A picture in a prompt was counted as its base64 by the context check, so
|
||
one 1280x720 PNG was "138 thousand tokens" and a vision model refused it;
|
||
a picture or a sound now counts as about a thousand.
|
||
- An on-demand model of a higher class was evicted, idle, for a lower-class
|
||
one two minutes into a lull; it keeps the card through its idle timeout,
|
||
and a model waiting on it is told so.
|
||
- A model removed from the catalog stayed wanted, so added again it loaded
|
||
unasked.
|
||
- A node that loads its weights by repository name (sf3d) went to Hugging
|
||
Face for a gated repository and got a 401: the agent did not hand the
|
||
gate `comfyui.huggingface_repo`, so the weights on the node were never
|
||
linked into its cache.
|
||
- **ComfyUI on a V100.** torch's CUDA 12.8 and newer builds have no Volta
|
||
kernels, but its cu129 build was taken to run from compute capability
|
||
7.0, so on a V100 both ComfyUI builds were installed, about 10 GB each,
|
||
and failed with "no kernel image". Cards older than Turing now get
|
||
ComfyUI on PyTorch's CUDA 12.6 torch (build `cu126`); a Turing or newer
|
||
node keeps the build it has.
|
||
- **The open NVIDIA driver read by its version.** With the open kernel
|
||
module, the default since 560, the agent read the driver as "for" and so
|
||
as unknown, and tried every engine build whatever driver it needs.
|
||
|
||
## v0.14.3 — 2026-10-10
|
||
|
||
3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
|
||
with their progress, their licence written into the mesh, a price per mesh
|
||
and a page to make them. Also: a profile's own weights, a band for bulk
|
||
work, and fixes for control planes behind a balancer and agents that
|
||
restart.
|
||
|
||
**Upgrading.**
|
||
- The store gains tables and columns (migrations 0043–0045: notify_sent,
|
||
jobs, usage meshes); nothing to do.
|
||
- **Nodes** need this release for their engines' output to survive an
|
||
agent restart and for ComfyUI's custom nodes and progress; engines
|
||
started by an older agent keep its pipe until they are restarted.
|
||
- A 3D model needs a model that makes them (see install.md, "A model that
|
||
makes them"); none is added.
|
||
|
||
### Added
|
||
|
||
- **Jobs: 3D models asked after, not waited for.** `POST
|
||
/v1/3d/generations` answers 202 with a job id; `GET
|
||
/v1/3d/generations/{id}` says its status, progress and, when done, the
|
||
file its result is in (`/v1/files/{id}/content`); `POST .../cancel` stops
|
||
it, in the engine's queue too. The request is replayed by whichever
|
||
gateway holds the job's lease through the same path a client's request
|
||
takes — key, allowlist, limits, admission, guardrails, accounting — so
|
||
it is charged like any request, tagged `job=<id>`, and a model still
|
||
loading is waited out for as long as the job has (24 h) rather than
|
||
refused. A job survives the gateway that took it; results are kept for
|
||
`router.batch.retain_days`. Migration 0044.
|
||
- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The
|
||
picture is uploaded to the replica's ComfyUI and the workflow's
|
||
`{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned
|
||
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
|
||
stronger ones can be added the same way.
|
||
- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in
|
||
each replica's own `custom_nodes` by the gate, with its packages
|
||
installed beside the replica, never into ComfyUI's environment.
|
||
- **A job's progress**: the gate listens to ComfyUI's websocket and the job
|
||
says queued and how many ahead, then the node running, its step and
|
||
percent.
|
||
- **A model's `license`** (name, url, notice), in `/v1/models`, in the job,
|
||
on the portal and in the GLB's `asset.extras`.
|
||
- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count
|
||
`meshes`.
|
||
- **`/3d` on the portal**, beside `/images`: a picture chosen, the job
|
||
followed, the mesh in a viewer with a download and its licence; and
|
||
**`llmgw ctl 3d list|get|cancel`** for every owner's jobs.
|
||
- **A profile's own `source`**: the weights it loads instead of the
|
||
model's, so an embedding model can run on vLLM from its repository on a
|
||
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
|
||
nodes holding each profile's weights, `ctl model add` checks every
|
||
source, and `ctl weights` collects none a profile loads.
|
||
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
|
||
long its requests may wait for room, instead of the admission's 30 s. A
|
||
band for bulk work — a knowledge-base import — waits behind people rather
|
||
than being turned away.
|
||
|
||
### Fixed
|
||
|
||
- **An engine's output no longer breaks when its agent restarts.** Engines
|
||
wrote to a pipe of the agent's, which an agent that restarted and adopted
|
||
them left with no reader; an engine's next large write failed, and in a
|
||
Python engine took the request it was reporting on with it. Each process
|
||
now writes to its own file in the agent's work directory, and an adopted
|
||
engine's log keeps what it wrote before the restart.
|
||
- **Single sign-on behind a balancer.** The cookie that carries a sign-in
|
||
across the identity provider was signed with a key of each process's own,
|
||
so with two control planes a callback that reached the other one was
|
||
refused. The key is now a secret in the store, the same on every control
|
||
plane.
|
||
- **Notifications sent once by two control planes.** Each kept its own
|
||
hour of quiet per event, so a budget spent on a team both served was
|
||
announced twice. The control plane that claims an event in the store
|
||
(migration 0043) sends it; with the store unreachable it is sent anyway.
|
||
- **A request turned away for a busy cluster says when to come back.** A
|
||
request that waited as long as it may (504) or found the queue full (503)
|
||
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
|
||
of sending it again; both now say 10 seconds.
|
||
- The usage sums and exports count pictures (`images`), as the request
|
||
records did; pictures made before are added to the sums.
|
||
|
||
## v0.14.2 — 2026-10-10
|
||
|
||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||
cluster's own cards. The agents install ComfyUI themselves, the gateway
|
||
prices pictures by the picture, and the portal has a page to make them.
|
||
Also: on-demand models that survive a failover, and knowledge-base
|
||
embeddings fixes.
|
||
|
||
**Upgrading.**
|
||
- The store gains a column (`usage.images`); nothing to do.
|
||
- **ComfyUI is installed only where asked**: `llmgw ctl config set
|
||
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
|
||
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
|
||
or run engine `comfyui`.
|
||
|
||
### Added
|
||
|
||
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
|
||
key, allowlist, limit, admission and guardrail checks as any request. A
|
||
model with profiles on engine `comfyui` is placed on a GPU node like any
|
||
other. A ComfyUI server outside the cluster can be an external model with
|
||
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
|
||
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
|
||
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
|
||
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
|
||
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
|
||
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
|
||
that leaves takes its workflow off the queue. `/v1/models` says `images`
|
||
for such a model.
|
||
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
|
||
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
|
||
gate listens on the replica's port, demands the replica's key, and lets
|
||
through only what the router asks. The model's `comfyui:` block names
|
||
the workflow and, for a source that is a set of files, the checkpoint.
|
||
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
|
||
checked against a digest llmgw holds, since GitHub publishes none, and a
|
||
version without one is refused. Its requirements are resolved as PyPI
|
||
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
|
||
CUDA 12.9 one for an older driver, and the result is checked on the card.
|
||
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
|
||
agent.yaml.
|
||
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
|
||
prompt's tokens. A model priced only per picture is not free, so a spent
|
||
budget still stops it. Each usage row keeps how many pictures were made,
|
||
shown by `llmgw ctl request`.
|
||
- **`/images` on the portal**, beside the chat, for the models that make
|
||
pictures. The chat no longer offers them. The page offers size, how many,
|
||
seed, steps and a negative prompt, and shows each picture with its seed
|
||
and a download link. Nothing is kept but the usage.
|
||
- **An embedding model's `query_prefix`**, put before the question a
|
||
knowledge-base search embeds and never before documents, for Qwen3-Embedding
|
||
and models trained like it.
|
||
|
||
### Fixed
|
||
|
||
- **An awake on-demand model survived no restart or change of leader of the
|
||
control plane.** A control plane that had heard no requests read every
|
||
on-demand replica its nodes reported as wanted by nobody, and drained them
|
||
all. An adopted replica now gets a whole idle timeout.
|
||
- `ctl models` said `not placed` for an on-demand model that had been woken
|
||
and was serving; it now says `ready (on demand)`.
|
||
- A batch of embeddings was refused as one over-long prompt when the
|
||
passages together exceeded the context, though each one fit. The longest
|
||
input is what has to fit.
|
||
- The chat's kept conversation sat in the browser under one key for
|
||
everybody, so the next person to sign in on the same browser opened the
|
||
last one's conversation. It is kept per account, and signing out clears
|
||
what the pages stored.
|
||
|
||
## v0.14.1 — 2026-10-09
|
||
|
||
Adapters trained on the nodes from a knowledge base and kept private to
|
||
their owners; the cluster's and the agents' settings edited in one place;
|
||
the tenancy tree and keys over the admin API and in Terraform; knowledge
|
||
bases that hold up with many documents; the firewall setting on every
|
||
machine.
|
||
|
||
**Upgrading.**
|
||
- **Nodes** need this release to follow `agents.*` in the cluster's
|
||
settings; until a node is upgraded its agent.yaml is what it runs on.
|
||
- **A separate mirror machine and store members that run nothing else**
|
||
follow the firewall setting once what reads it is installed: run `sudo
|
||
llmgw mirror init` on the mirror again with the flags it was set up with,
|
||
and `llmgw server init --patroni` again for the store's members. A control
|
||
plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
|
||
- **The store's PostgreSQL memory** applies after each member is restarted,
|
||
one at a time, the leader last; until then it runs as before.
|
||
- The store gains tables and columns; nothing to do. Existing knowledge-base
|
||
collections get their full-text index when a control plane starts, and
|
||
keep `vector` storage; only new collections use `halfvec`, and only on
|
||
pgvector 0.7 or later.
|
||
- Cached knowledge-base passages in vLLM are made again once, under the new
|
||
`cache_salt`.
|
||
|
||
### Added
|
||
|
||
- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
|
||
for administrators. It edits the document `llmgw ctl config edit` opens:
|
||
checked as a control plane would load it, refused when made from a
|
||
version since replaced, with the history to view and roll back. Changes
|
||
are audited by size, never by contents.
|
||
- **The nodes' agent settings in the cluster's settings** (BACKLOG E3):
|
||
`agents.default` and `agents.nodes.<id>` hold everything agent.yaml can
|
||
say but the agent's id, control planes, address and certificate. They are
|
||
sent to the agents as they change; an agent starts again on new settings,
|
||
or on an edited agent.yaml, with its engines serving throughout. Settings
|
||
that would not load are refused on save, and set aside on a node rather
|
||
than keep its agent from starting. Nodes need this release to follow
|
||
them.
|
||
- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
|
||
writes question/answer pairs from a collection with a model of the
|
||
cluster. `llmgw ctl train start` trains an adapter on a node with its
|
||
installed vLLM's torch and a small overlay, on a card with enough memory
|
||
free (beside an engine, if it leaves that much), converting to GGUF for
|
||
llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
|
||
accepts uploads) and makes it the model's adapter.
|
||
- **Adapters with owners.** An adapter's `owner` (org, team, project, user
|
||
or key) is the only part of the tree that sees it; to anyone else it is a
|
||
model that does not exist. An adapter trained on a knowledge base belongs
|
||
to a part of the tree that may read it, and is never made public.
|
||
- The knowledge bases' vLLM `cache_salt` follows the collections searched,
|
||
not the organization, so two teams of one organization no longer share
|
||
cached passages.
|
||
- **Organizations, teams, projects, API keys and memberships in the admin
|
||
API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
|
||
`PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
|
||
issue (the key is in the answer, once), list and revoke; `/api/v1/users`
|
||
and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
|
||
and `users` on a control plane before. The Terraform provider manages them
|
||
as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
|
||
sensitive `secret`) and `llmgw_membership`.
|
||
- **The firewall setting reaches every machine.** A separate mirror and the
|
||
store's members follow `firewall.enabled` and `firewall_networks` as nodes
|
||
and control planes did: the leading control plane tells the mirror over
|
||
its API, and keeps the state in the store for members that run no llmgw,
|
||
which a timer on each reads. The mirror also lets in the network of every
|
||
node the leader knows. `mirror init` and `server init --patroni` install
|
||
what follows it; run them again on a cluster set up before.
|
||
|
||
### Changed
|
||
|
||
- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
|
||
on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
|
||
machine. Now each member gets a share of its own RAM, a smaller one where
|
||
a control plane or agent shares the machine, and the share is read again
|
||
every time `server init --patroni` runs. The new cache is applied by
|
||
restarting one member at a time, the leader last.
|
||
- **Knowledge-base searches hold up with many documents.** New collections
|
||
store their index as `halfvec` where pgvector is 0.7 or later (half the
|
||
memory, and embeddings up to 4000 dimensions instead of 2000); older
|
||
collections, and clusters on pgvector 0.6, keep `vector`. Each collection
|
||
has its own full-text index, as it has its own vector index, made for
|
||
existing collections when the control plane starts. The text arm ranks by
|
||
the cheap measure first and scores only those candidates.
|
||
|
||
### Fixed
|
||
|
||
- A knowledge-base search asking for more than 10 passages
|
||
(`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
|
||
vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
|
||
raised to the number wanted, up to 1000.
|
||
- `llmgw server backup` and `restore` against a replicated store over TLS
|
||
failed looking for `~/.postgresql/root.crt`: pg_dump was given
|
||
`sslmode=verify-full` without the cluster's authority.
|
||
|
||
## v0.14.0 — 2026-10-08
|
||
|
||
Knowledge bases: a company's documents, put in by people from their own
|
||
machines, kept apart per tenant, and answered from.
|
||
|
||
**Upgrading.**
|
||
- **pgvector on a cluster set up before this release.** Knowledge bases
|
||
need pgvector, which `server init` and the high-availability database
|
||
setup now install, but a rerun on a running cluster does not:
|
||
1. Install `postgresql-<version>-pgvector` on every database machine.
|
||
2. As postgres, run `CREATE EXTENSION vector` in llmgw's database.
|
||
3. Restart the control planes.
|
||
Until then everything else works, and `llmgw ctl kb list` says why
|
||
knowledge bases are off. On a SQLite store they stay off.
|
||
- **An embedding model.** A collection needs a catalog model with
|
||
`capabilities.embeddings`, for example bge-m3, whose weights are in the
|
||
mirror.
|
||
- **The client.** People download `llmgw-kb` for their system from this
|
||
release.
|
||
|
||
### Added
|
||
|
||
- **Collections and who may read them.** A collection is owned by an
|
||
organization, team, project or person. Grants add up down the tenancy
|
||
tree and never cross an organization. A collection a key may not read is
|
||
answered as one that does not exist. Chunks are kept in PostgreSQL with
|
||
pgvector, partitioned by organization, with an HNSW index per collection,
|
||
and row-level security on the table as a second line behind every
|
||
query's own filter. A project, team or organization that owns collections
|
||
is not deleted; a person's collections go with their account.
|
||
- **`llmgw-kb`**, its own download for Windows, macOS and Linux, run by
|
||
people with their own key. It reads documents where they are and sends
|
||
text and vectors, never the files, for what changed only; a SQLite ledger
|
||
on the machine remembers what it did.
|
||
- It reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML,
|
||
MHTML, mail with attachments, notebooks, subtitles, ZIP archives, and
|
||
text and source code in any common encoding.
|
||
- Legacy Office files are read with LibreOffice, and scanned pages with
|
||
Tesseract, when those are installed.
|
||
- `--dry-run` says what would leave the machine; `--delete` removes only
|
||
what that machine sent.
|
||
- **Answers from documents.** A chat request names collections in
|
||
`X-Llmgw-Knowledge`, or its part of the tree has a default (`llmgw ctl kb
|
||
default`). The best passages, chosen by meaning plus a little for the
|
||
question's words, are added to the system message, numbered, and
|
||
`x-llmgw-sources` says which were used. This happens before the
|
||
guardrails judge the request, and the answer is never served from the
|
||
answer cache. On vLLM replicas the organization's `cache_salt` keeps
|
||
tenants' prefix caches apart. It works through the Responses and Messages
|
||
APIs too.
|
||
- **Administration**: `llmgw ctl kb` (create, list, grant, revoke, default,
|
||
docs, rm, search, delete), `/api/v1/kb`, the console's Knowledge screen,
|
||
and a Documents picker on the portal's chat page.
|
||
- Metrics `llmgw_kb_retrievals_total` and `llmgw_kb_retrieval_seconds`;
|
||
[docs/knowledge.md](docs/knowledge.md).
|
||
|
||
## v0.13.0 — 2026-10-08
|
||
|
||
The whole system stopped, started and restarted from one command, a
|
||
guardrails judge that no longer refuses ordinary coding requests, and the
|
||
rest of the field report from v0.10.6.
|
||
|
||
**Upgrading.** A cluster keeps the judge its policies name: one installed
|
||
before this release goes on asking Llama Guard 3 1B until `llmgw ctl guard
|
||
judge use qwen3guard-0.6b`, which the leader then fetches and serves by
|
||
itself, and which replaces the policies' Llama Guard category codes.
|
||
|
||
### Added
|
||
|
||
- **A judge on a GPU and one on CPUs together**: a policy's `judge.models`
|
||
is a list asked in turn, the first that answers giving the verdict; the
|
||
next is asked when one has no replica serving, fails, or misses its
|
||
`latency_budget`. `llmgw ctl guard judge use qwen3guard-4b
|
||
qwen3guard-0.6b` sets the order for every policy with a judge, and
|
||
`guard judge status` says which judge stands in for which. A judge on a
|
||
GPU is served as a `standard` model, so it takes a card only from what
|
||
may give one up.
|
||
- **A judge a policy names is fetched if it is missing**: the leader
|
||
fetches the weights of a judge llmgw knows (`qwen3guard-0.6b`, `-4b`,
|
||
`-8b`, `llama-guard-3-1b`, `-8b`) into the mirror and serves it, when a
|
||
policy names it and the catalog does not have it — and again if it is
|
||
removed. `guard judge enable --gpu` serves Qwen3Guard-Gen 4B on the class
|
||
with the most free cards.
|
||
- **The whole system stops, starts and restarts from one command**: `llmgw
|
||
stop` takes every model down and answers every request for one with 503
|
||
(`system_stopped`), keeping the control planes, agents and mirror running
|
||
and no model's definition changed; `llmgw start` brings every model back.
|
||
It is the cluster's setting `serving.stopped`, live on every control plane.
|
||
`llmgw restart` restarts every node's agent and every control plane, the
|
||
leader last, with the engines serving throughout. `llmgw status` says
|
||
whether the system serves. One machine's services are `llmgw service
|
||
status|start|stop|restart`.
|
||
- **A replica says what it computes on**: `ctl replicas` (ON), the console
|
||
and the admin API show `CUDA` with the GPUs its processes hold memory on
|
||
and how much, as nvidia-smi sees them, or `CPU` — for every engine, kept by
|
||
the agent across its own and the control plane's restarts. llama.cpp's
|
||
layer count is added when it says it, which it does only at `-lv 4`.
|
||
- **`ctl engine-check llamacpp` computes a token on the GPU**: a small GGUF
|
||
(`--model`, or the smallest in the weights cache) is loaded with every layer
|
||
on the GPU at `-lv 4` and asked for one token; a build whose CUDA backend
|
||
does not load fails, by llama.cpp's own account of its layers. With no
|
||
`--bin` it checks the build the agent installed.
|
||
- **Engine versions and compute capability on nodes**: `ctl nodes` shows
|
||
`llamacpp b10991` and `6000-24g (7.5)`; the admin API gives
|
||
`engine_versions` and `compute_capabilities`. A profile's `engine_version`
|
||
restricts it to nodes whose engine version starts with it, and
|
||
explain-placement says which version a node has, or which engines.
|
||
- **`ctl restart --max-surge N` and `--max-unavailable N`**: that many of a
|
||
model's replicas restart at once, beside themselves or stopped first.
|
||
- **Multi-node readiness apart from the node's**: doctor gives both, with
|
||
fabric findings in a section of their own; `node add` says the multi-node
|
||
one; `ctl nodes` shows a declared fabric's adapters that are down.
|
||
- **A forgotten password is set through a link**: `/forgot` mails one when
|
||
the cluster's `mail` settings say how (SMTP, sender, the public
|
||
`base_url` it points at), saying the same whether or not the address has an
|
||
account; `/reset` sets the password once and closes every session. Without
|
||
mail, `llmgw users reset-link` makes the link for an operator to hand over.
|
||
|
||
### Fixed
|
||
|
||
- **A replica restarted in place keeps its room**: the replica was stopped
|
||
and the pass went on to other models before placing its replacement, so a
|
||
model waiting for room took the freed cards. On the reference cluster a
|
||
GPU judge waiting for the only card took it from the model being
|
||
restarted there. The replacement is now placed in the same step.
|
||
- **The judge no longer refuses ordinary coding requests**: Llama Guard 3
|
||
1B, the judge `server init` installed, could not tell "continue", "devam
|
||
et" or "how do I kill all child processes in bash" (0.51 to 0.92 unsafe)
|
||
from harmful requests (0.97 to 1.00), at any quantisation. The judge is
|
||
now Qwen3Guard-Gen 0.6B (format `qwen3guard`), which gave the same
|
||
ordinary requests 0.15 at the most and the harmful ones 1.00, on CPUs,
|
||
in 0.8 GB. Its third verdict, controversial, counts as safe unless the
|
||
policy says `controversial: unsafe`. `guard judge use qwen3guard-0.6b`
|
||
moves an installed cluster to it, and replaces Llama Guard's category
|
||
codes, which Qwen3Guard never names, with the baseline's.
|
||
- **A long message is judged rather than refused**: a judge read the whole
|
||
last user message, and one with a file pasted into it overflowed the
|
||
judge's context, so it gave no verdict and `fail: closed` refused the
|
||
request. It now reads the start and the end, about 6000 characters.
|
||
- **A replica no longer stays "starting" while it serves after a restart**:
|
||
an agent that registered before its adopted engines had answered their
|
||
health check never said they had, so the control plane kept them starting
|
||
and routed nothing to them — on the reference cluster, after `llmgw
|
||
restart`, the only replica of a model. The agent now reports each shard
|
||
the registration called not ready once it is.
|
||
- **A failover no longer replaces healthy replicas**: a standby that took
|
||
over placed at once, before the agents had reconnected and said what runs,
|
||
so it started a second copy of replicas it did not know of and then drained
|
||
the healthy ones as surplus — on the reference cluster the guardrails'
|
||
judge and a model's replica, within twenty seconds of a leader restarting.
|
||
Placement now waits after a takeover as it does at start.
|
||
|
||
### Changed
|
||
|
||
- **A GPU placement llama.cpp says it cannot compute on is refused** when it
|
||
warns that it found no usable GPU, even where nvidia-smi cannot be asked;
|
||
where it can, holding no memory on its GPUs refuses it as before.
|
||
|
||
## v0.12.2 — 2026-10-06
|
||
|
||
The guardrails turned off and on as a whole, and a judge that refuses
|
||
ordinary requests explained: the hazard it saw is told to the client and
|
||
kept with the request, and `llmgw ctl guard judge ask` shows what it says of
|
||
any text. Nothing to do to upgrade; the store gains a column.
|
||
|
||
### Added
|
||
|
||
- **The guardrails turn off and on as a whole**: `llmgw ctl guard off` /
|
||
`on`, which is `guard.enabled` in the cluster's settings — in force within
|
||
seconds, the policies and their attachments kept. `ctl status`, `ctl guard
|
||
list` and the guardrails page say when they are off.
|
||
- **`llmgw ctl guard judge ask TEXT`** has the judge read a text and shows
|
||
its answer as given and as read — for finding out why ordinary requests
|
||
are refused. Nothing of the text is kept, not even in the audit.
|
||
|
||
### Changed
|
||
|
||
- **A judge's refusal names the hazard**: the client is told `(judge:
|
||
S6,Specialized Advice)` rather than `(judge)`, and the verdict is kept with
|
||
the request (`guard_rule`): `ctl request`, the server's log and the
|
||
guardrails page show it. Llama Guard 3's codes come with their names,
|
||
S9 Indiscriminate Weapons, without a categories list.
|
||
|
||
## v0.12.1 — 2026-10-06
|
||
|
||
The guardrails' judge, served and watched without server init: a judge
|
||
missing, served as the wrong kind of model, or not answering is said on
|
||
`ctl status` and the console, and `llmgw ctl guard judge enable` puts it
|
||
right. Nothing to do to upgrade.
|
||
|
||
### Added
|
||
|
||
- **`llmgw ctl guard judge status` and `enable`**: each judge model the
|
||
policies ask, whether it is in the catalog, as a judge, with replicas
|
||
ready and answering — and serving the missing one, Llama Guard 3 1B on
|
||
CPUs under the name the policies give it, with `--replace` for a model of
|
||
that name that is not a judge. No more `server init` to put a judge back.
|
||
Also **Do → Serve the guardrails' judge** in the console.
|
||
- **A missing or broken judge is said where it is looked for**: `llmgw ctl
|
||
status` and the console's Overview list each judge with a problem — not in
|
||
the catalog, not a judge, no replica ready, or failing to answer — and
|
||
what that does to requests (refused or unjudged).
|
||
|
||
### Fixed
|
||
|
||
- **A Llama Guard file served from the mirror page is the judge**: it was
|
||
added as a model clients use, under whatever name the form had, so the
|
||
policies never found it and refused everything they judged
|
||
(`guard_unavailable`). It is now added as a judge, under the name the
|
||
policies ask.
|
||
- **`ctl` says so when the API answers with a page**: a command a control
|
||
plane does not know — the leader on an older release, through a standby —
|
||
failed with "invalid character '<'"; it now says which and why.
|
||
|
||
## v0.12.0 — 2026-10-06
|
||
|
||
A minor version, because where the configuration lives changes: everything in
|
||
server.yaml that is not one machine's own is now the cluster's settings, kept
|
||
in the store, the same on every control plane and in force within seconds of
|
||
a change. Nothing needs doing by hand to upgrade. The first control plane to
|
||
start on v0.12.0 brings its server.yaml's cluster keys into the store, with
|
||
the baseline guard policy when the file had none. From then on, change them
|
||
with `llmgw ctl config` or the console's Settings, not in server.yaml, whose
|
||
cluster keys are ignored, with a warning. Upgrade every control plane (`llmgw
|
||
ctl upgrade` does, the leader last): until the leader runs v0.12.0,
|
||
`llmgw ctl config` through a standby is refused. Nodes need v0.12.0 to follow
|
||
the firewall switch.
|
||
|
||
### Changed
|
||
|
||
- **The cluster's settings live in the store, and every change is in force
|
||
within seconds.** Everything in server.yaml that is not one machine's own
|
||
(guardrails, limits and prices, routing, engines, notifications, the
|
||
mirror's policy, declared models, offline, federation, facility budgets) is
|
||
a versioned document in the store, the same on every control plane.
|
||
`llmgw ctl config` shows, sets, edits, lists and rolls it back, and the
|
||
console's Settings screen edits it. A change is checked, kept with who made
|
||
it and why, and applied: guardrails live, anything else by every control
|
||
plane starting again on it, standbys first and the leader last. A change to
|
||
a machine's own keys in its server.yaml is noticed and applied the same way.
|
||
No "restart now?" any more, and no second copy to keep in step. The first
|
||
start against a store brings in the file's cluster keys, with the baseline
|
||
guard policy when the file had none, so an upgraded cluster gets its
|
||
guardrails; after that, those keys in a server.yaml are ignored, with a
|
||
warning.
|
||
|
||
### Added
|
||
|
||
- **llmgw's firewall can be turned off and on for the whole cluster**:
|
||
`llmgw ctl config set firewall.enabled false`. Every node's agent takes its
|
||
table out within seconds, and each control plane's new
|
||
`llmgw-firewall-sync.path` does the same there; `true` puts the rules back
|
||
as they were. `firewall_networks` changes reach the nodes the same way.
|
||
- **`llmgw wipe`** removes llmgw from a machine — services, firewall table,
|
||
configuration, the authority, secrets, state, users — after listing it and
|
||
asking for the machine's name; `--models`, `--store` and `--all` take
|
||
weights, the store's database here, backups and the binary too. Also
|
||
**Do → Wipe this machine** in the console.
|
||
|
||
### Fixed
|
||
|
||
- **The mirror is given an address the nodes reach**: `server init` and
|
||
`mirror init` took the machine's first address for the mirror's `--base`,
|
||
which on a machine with Docker could be 172.17.0.1 — a node then asked its
|
||
own bridge for every model and was refused. They now take the source of
|
||
the default route, else the first address not on a container's or a
|
||
bridge's interface. And `node add` has the node reach the mirror's files
|
||
before it is added: one that cannot fails the checks, naming the address
|
||
and what to change.
|
||
- **A standby that becomes the leader reads the store again**: models added
|
||
or removed, engine builds chosen and domains held through the leader before
|
||
it, rather than placing on what it read when it started.
|
||
- **A node on another network joins**: `node add` set up the node's firewall
|
||
but left the control plane's to the networks it was installed with, so a node
|
||
on another subnet was installed and then never heard from (lost). It now lets
|
||
the node in on the control plane it runs on — its subnet, or its address
|
||
when a router is between them — and says so; a network already let in
|
||
changes nothing.
|
||
|
||
## v0.11.4 — 2026-10-05
|
||
|
||
Two control planes, run in anger on the reference cluster.
|
||
|
||
### Added
|
||
|
||
- **The console sets up a second control plane**: Do → Add a control plane
|
||
asks for the machine and a floating address and runs `server add`.
|
||
- **Who leads, everywhere.** `ctl status` and the console's Overview have a
|
||
`CONTROL` line: which control plane leads, which stand by, one that does not
|
||
answer, and whether the floating address is on the leader.
|
||
- **A standby's console is the leader's.** Its admin API passes `/api/` to the
|
||
leader over the agent port, with the cluster's certificates both ways, so
|
||
`sudo llmgw` and `llmgw ctl` show the same cluster on either machine. Control
|
||
plane certificates are now issued for client use too; `server add` reissues
|
||
older ones.
|
||
- **`ctl upgrade` in waves.** Nodes are upgraded one, then two, four… up to
|
||
`--parallel` (20) at once, so hundreds take minutes; a wave that does not
|
||
come back stops it. The other control plane is upgraded first, over ssh.
|
||
- `server add` says when the mirror's `--api-allow` would refuse the new
|
||
control plane.
|
||
- **Guardrails in the console**: a view of the policies and what each checks,
|
||
whether their judge models are serving, where they are attached (attach
|
||
and detach from it), and what they did in a window. `GET
|
||
/api/v1/guard/events` answers the same, and `GET /api/v1/guard` now says
|
||
what each policy does.
|
||
|
||
- **A standby's portal is the leader's.** Its client port passes everything
|
||
but `/health`, `/ready` and `/leader` to the leader, and its admin port the
|
||
dashboard as well as the API; a proxy left pointing at a standby showed an
|
||
empty cluster. The leader trusts the control planes' `X-Forwarded-For`.
|
||
|
||
- `ctl model serve --tools` says in the catalog that a model makes tool
|
||
calls, for agents and editors.
|
||
|
||
### Fixed
|
||
|
||
- **Continue's agent did nothing.** The configuration the portal writes told
|
||
Continue every model uses tools, and nothing of its context: a model whose
|
||
calls came back as text stalled the agent, and prompts over a CPU
|
||
replica's 4096 tokens were refused. It now says `tool_use` only for a
|
||
model whose catalog says tools, and the context and longest answer the
|
||
gateway allows.
|
||
- `ctl model serve` gave llama.cpp four slots whatever was asked: a
|
||
`--context-length 32768 --ctx-pool 32768` model had slots of 8192 and
|
||
refused what the gateway admitted. Slots are now what the pool holds of
|
||
the context. The longest answer grows with the context (a quarter, from
|
||
1024 to 8192) rather than being 1024 for every model.
|
||
- `ctl upgrade` upgrades the leader last wherever it is run; run on the
|
||
standby, it upgraded the leader first and the leadership moved twice.
|
||
- **Failover took 40 seconds; it takes a few.** A stopping control plane held
|
||
every agent's stream for the thirty seconds of a graceful shutdown; it now
|
||
gives up its lease first and lets the agents go at once.
|
||
- A control plane that lost its lease while running kept its agents, leaving
|
||
the new leader with no nodes. It now lets them go.
|
||
- An agent's reconnect backoff carried over from one outage to the next, so
|
||
each failover made the next one slower; it starts again after a session that
|
||
worked.
|
||
- The `CONTROL` line named no floating address.
|
||
|
||
## v0.11.2 — 2026-10-05
|
||
|
||
### Routing by what each vLLM replica's cache holds
|
||
|
||
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
||
publishing its KV cache events (`--kv-events-config`) to a port on
|
||
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
||
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
||
computed outside Python. The events never leave the node.
|
||
- The agent mirrors each engine's set of cached blocks and sends the net
|
||
changes up its existing stream to the control plane four times a second
|
||
(new `KVEvents` agent message), starting every new stream with a full
|
||
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
||
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
||
remote ones and a hybrid model's second cache group are not indexed.
|
||
- The control plane passes them to the router's index, only for replicas
|
||
the sending node runs, and empties what a stream described when it closes.
|
||
- For a model with `kv_events`, the router asks one of its replicas to
|
||
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
||
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
||
does, and scores replicas by the share of the prompt's leading blocks they
|
||
hold. The imbalance guard still applies. With no answer — no events yet,
|
||
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
||
back to the approximate tree (`router.prefix`) or to load.
|
||
- The agent checks first blocks against llmgw's hashing; an engine that
|
||
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
||
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
||
though it published nothing.
|
||
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
||
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
||
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
||
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
||
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
||
- Experimental: tested against vLLM's event format and hashing as its source
|
||
defines them, and against libzmq and msgspec publishing — not yet against a
|
||
running vLLM.
|
||
|
||
### LoRA adapters on vLLM, loaded on demand
|
||
|
||
- **vLLM models can carry adapters.** A PEFT adapter directory (a mirror
|
||
version ending in `/`, a path on the nodes, or a Hugging Face repository
|
||
id) is registered at start with `--enable-lora --lora-modules`, and a
|
||
request names it as its model, as on llama.cpp. `scale` applies on
|
||
llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
|
||
- **`max_loaded_adapters: N` on a model** has each replica hold at most N of
|
||
its adapters. Replicas start holding none (vLLM with
|
||
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras N`, unless the
|
||
profile's args set their own). The router loads an adapter on a replica
|
||
through `/v1/load_lora_adapter` when a request first wants it there, and
|
||
unloads the least recently used one no request is running on to make room.
|
||
vLLM profiles only; llama.cpp keeps loading every adapter at start, since
|
||
llama-server can rescale an adapter it started with but not load another.
|
||
- **Adapter affinity:** a request for an adapter goes to a replica that has
|
||
it, or is loading it, while that replica is not saturated; otherwise to one
|
||
with room for another adapter; otherwise where load sends it — which is how
|
||
a busy adapter spreads to a second replica. Requests arriving during a load
|
||
wait for it instead of loading the adapter again.
|
||
- **Self-correcting:** each replica's `/v1/models` is read when the router
|
||
first meets it and every 30 seconds, so adapters loaded before a restart or
|
||
by another router are known; a 404 for an adapter the router thought loaded
|
||
loads it again and retries. An adapter vLLM refuses (not found, rank above
|
||
`--max-lora-rank`) fails the request with vLLM's reason, without trying it
|
||
on every replica.
|
||
- **Metrics:** `llmgw_adapter_affinity_total{model,result=hit|joined|loaded}`,
|
||
`llmgw_adapter_loads_total{model,outcome}`,
|
||
`llmgw_adapter_evictions_total{model,outcome}`.
|
||
- The admin API and `ctl model adapters` read the file as GGUF only for a
|
||
model with llama.cpp profiles.
|
||
|
||
### Federating clusters: another llmgw cluster as a provider
|
||
|
||
- `llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags]`
|
||
makes another llmgw gateway a peer. It is checked first (its `/ready`,
|
||
timed, and its `/v1/models` with the key) and kept in the store (migration
|
||
0029, `federation_peers`) with the models it offers.
|
||
- A peer's models join the catalog as `NAME/MODEL`, always prefixed, and
|
||
follow the peer's own list every `federation.interval` (10s). A model the
|
||
peer drops goes, unless a model here still falls back to it: then it is
|
||
kept withdrawn (in the catalog, sent nothing) until the fallback changes.
|
||
The peer's aliases and its own peers' models are not taken.
|
||
- A peer whose `/ready` fails, which refuses the key, or which does not answer
|
||
is taken out of routing until it answers again. `/ready` now counts only a
|
||
gateway's own endpoints, not its peers'.
|
||
- `llmgw ctl model fallbacks MODEL [FALLBACK...]` (and `PUT
|
||
/api/v1/models/{name}/fallbacks`) sets a runtime model's fallbacks, so
|
||
overflow can go to a peer's model.
|
||
- Requests to a peer carry `x-llmgw-hop` with the ids of the clusters they
|
||
have passed through (each cluster's id is kept in the store). A request that
|
||
comes back to a cluster is refused with 508 `federation_loop`, and a peer a
|
||
request came from is not sent it.
|
||
- The peer sees one key, this cluster's, and the request's `X-Request-Id`.
|
||
With `--send-tags`, also `x-llmgw-tags: federated_from=<federation.name>,
|
||
org=,team=,project=` (ids); off by default.
|
||
- In-cluster residency and `no_store` keep requests off peers, by name and as
|
||
fallbacks. Usage records the peer's model as served, and `peer:NAME/MODEL`
|
||
as the replica.
|
||
- `ctl federation list|remove|sync`, a Federation view in the console,
|
||
`llmgw_federation_peer_up`, `_latency_seconds` and `_models` metrics, admin
|
||
API `/api/v1/federation`, and a "Federating clusters" section in
|
||
docs/install.md. New `federation:` block in server.yaml (`name`,
|
||
`interval`).
|
||
- A response header the gateway already set (X-Request-Id,
|
||
x-llmgw-served-model) is no longer repeated from the upstream's.
|
||
|
||
### Facility-aware placement: racks, PDUs, holds and power caps
|
||
|
||
- **Where a node is**, from five labels: `llmgw.io/site`, `room`, `row`,
|
||
`rack` and `pdu` (a list, for a dual-fed rack), in `agent.yaml` or set from
|
||
the control plane with **`llmgw ctl label NODE key=value… [key-…]`**, which
|
||
is kept in the store (migration 0031, `node_labels`), wins over the agent's
|
||
labels, works before the node joins, and refuses a misspelt `llmgw.io/` key.
|
||
A node with no location places as before.
|
||
- **Replicas spread across racks and PDUs.** A model's next replica goes to a
|
||
node sharing no rack or PDU with its others when there is one: an ordering
|
||
above weights already on disk and bin packing, never a filter.
|
||
- **`keep_within: rack`** (or row, room, site) on a profile keeps a
|
||
multi-node replica's nodes in one such domain.
|
||
- **Holds.** **`llmgw ctl facility hold rack:r12-07 --reason … [--drain]`**
|
||
places nothing new in a rack, row, room, site or PDU; `--drain` also drains
|
||
its nodes. **`llmgw ctl facility release`** undoes both. Kept across
|
||
restarts (migration 0031, `facility_holds`).
|
||
- **Power caps.** `facility.budgets` in `server.yaml` gives a domain's
|
||
`cap_watts` and `other_watts`, with a `margin`; a replica's draw is the
|
||
profile's `power_watts`, or its cards at the power limit they report
|
||
(`gpu_watts` otherwise) and its cores at `cpu_core_watts`, with `node_watts`
|
||
per node. Use is the larger of reserved and measured, node by node; a shard
|
||
that would take a domain past its cap less the margin is refused, and a
|
||
multi-node replica's shards are counted together. Unknown draw never blocks.
|
||
- **Seen everywhere it matters.** `ctl explain-placement` names the held
|
||
domain or the kilowatts, and keeps the rejections of a replica that was
|
||
placed, so one in an unexpected rack says why. `ctl nodes` has LOCATION,
|
||
*ready, held* and `--domain`. **`llmgw ctl facility`** lists every domain
|
||
with nodes, use, limit, reserved, measured and holds. The console's Nodes
|
||
view has WHERE; a node's menu has **Location…** and **Hold rack…/Release**.
|
||
Admin API: `GET /api/v1/facility`, `POST /api/v1/facility/holds`,
|
||
`DELETE /api/v1/facility/holds/{domain}`, `POST /api/v1/nodes/{id}/labels`.
|
||
- Docs: `docs/facility-aware-placement.md` says what is built and where it
|
||
departs from the design; `docs/install.md` and `docs/console.md`.
|
||
|
||
### SGLang measured, checked and benchmarked as itself
|
||
|
||
- **Fixed:** the router read SGLang replicas with vLLM's metric names, and
|
||
SGLang was started without `--enable-metrics`, so it served no `/metrics`
|
||
at all: every scrape failed as though the replica were unreachable, quietly,
|
||
and requests were routed to it blind. SGLang is now started with
|
||
`--enable-metrics` and read in its own names — `sglang:num_running_reqs`,
|
||
`sglang:num_queue_reqs`, `sglang:token_usage` (the KV pool's fill), the
|
||
prompt and generation token counters and `sglang:num_requests_total`, as
|
||
0.5.21 publishes them.
|
||
- **Fixed:** `llmgw ctl engine-check sglang` asked the interpreter for
|
||
`--help` and checked SGLang's flags against Python's. It now asks
|
||
`python -m sglang.launch_server --help`, in the environment SGLang was
|
||
installed into (or `--bin`).
|
||
- **Fixed:** SGLang was given `--tp`, which argparse accepts only as an
|
||
abbreviation of `--tp-size`; it is given `--tp-size`.
|
||
- **Fixed:** `llmgw ctl bench --metrics` read an engine's metrics with the
|
||
key given for `--target` — a client's key when the target is the gateway,
|
||
or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the
|
||
engine's key, refused every read, and the run reported a queue depth of
|
||
zero and suggested no thresholds without saying why. The engine key is now
|
||
asked of the admin API for the replica at `--metrics`' address (or given
|
||
with the new `--metrics-key`); metrics that cannot be read — refused, or in
|
||
another engine's names than `--engine` — fail the run before any load,
|
||
saying why; and a step whose metrics went unread shows `-` and the reason
|
||
rather than zeros.
|
||
|
||
### Reusing answers
|
||
|
||
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
||
--project P --ttl 1h` has the same request — same project, endpoint,
|
||
model by its catalog name, and body with `stream`, `stream_options`,
|
||
`user`, `metadata`, `store` and `service_tier` left out and its keys in
|
||
any order — answered from the gateway's memory instead of the model, for
|
||
the TTL. `--off` stops it; without either it says how it stands. The
|
||
console has it as **Cache answers…** on a project.
|
||
- **Only answers that come out the same are reused:** chat and completions
|
||
asked at temperature 0 or with a seed, and embeddings, rerank and score.
|
||
`--any` reuses sampled answers too. Only a whole 200 from the model asked
|
||
for is kept — not an error, a fallback's or a canary's answer, one a
|
||
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
|
||
- **Streams both ways:** a streamed answer is kept put back together into
|
||
one completion (tool calls and reasoning included), and a kept answer is
|
||
streamed as chunks — with the usage chunk if the client asked — to a
|
||
client that streams. `/v1/messages` and `/v1/responses` get it too.
|
||
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
|
||
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
|
||
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
|
||
- **Policies still apply:** limits and the input guardrails run before the
|
||
cache is looked in, the output policy judges a reused answer as it would
|
||
a new one, and a request under `no_store` or continuing a stored
|
||
conversation is never served from it nor kept in it. A project's answers
|
||
are never looked up by another project's keys.
|
||
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
|
||
marked `cache_hit` on the usage record (migration 0027), and shown by
|
||
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
|
||
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
|
||
together in each gateway process, least recently used dropped first;
|
||
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
|
||
`llmgw_cache_bytes` and `llmgw_cache_entries`.
|
||
|
||
### Offline sites and Kubernetes
|
||
|
||
- **Offline bundles.** `llmgw bundle create` (on any machine with internet)
|
||
collects what a cluster would otherwise download — llmgw's release for
|
||
each `--arch` with its signed checksums, every llama.cpp build for those
|
||
architectures (or the `--variant` ones) with the checksums GitHub
|
||
publishes, the starter model and guardrails' judge, and any `--model
|
||
name/version=hf:owner/repo/file` pinned to the registry's commit and its
|
||
checksum — each checked as it arrives, into a directory or one `.tar`
|
||
(`--tar`), with a manifest whose sha256 it prints to carry apart from the
|
||
bundle. Run again into the same directory, it downloads only what is
|
||
missing. `llmgw bundle show --verify` reads one back.
|
||
- **`llmgw bundle import`** checks every file against the manifest and
|
||
llmgw's release against the project's signing key, and only then puts them
|
||
into the mirror — releases and engines under `_llmgw/`, models where the
|
||
mirror keeps models, with their checksums — so a damaged bundle leaves the
|
||
mirror as it was; a different file under a version already there is
|
||
refused. On a control plane it also records the engines' checksums in
|
||
`offline.checksums` (`/etc/llmgw/offline.sha256`); `llmgw bundle trust`
|
||
does that alone, for a mirror on another machine.
|
||
- **`offline.mirror` in server.yaml** makes the cluster reach nothing
|
||
outside: nodes install llama.cpp from the mirror, each file held to the
|
||
checksum the control plane sends them over the agent's authenticated
|
||
connection — never one from the mirror — and `releases` defaults to the
|
||
mirror's copy, so `node add`, `ctl upgrade` and agents upgrading fetch
|
||
from it; no registry is asked about newer weights. A release may now come
|
||
over plain HTTP when the build trusts a signing key, since the signature
|
||
holds whatever carries it. vLLM and SGLang come from PyPI and are said to
|
||
be unsupported on an offline node rather than attempted.
|
||
- **`llmgw server init --bundle PATH --bundle-sha256 …`**, and the same two
|
||
fields on the setup form: installs a control plane offline — the bundle
|
||
into the mirror, the checksums trusted, `offline:` written to server.yaml,
|
||
the starter model and judge served from the bundle or skipped when it does
|
||
not carry them.
|
||
- `docs/data.md` no longer says only weights and `ctl plan` reach outside:
|
||
it lists everything that does, and what an offline cluster changes.
|
||
- **A Helm chart for the control plane** (`deploy/helm/llmgw`): one replica
|
||
in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent
|
||
port's certificate and the admin token from Secrets, the API and metrics
|
||
Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress
|
||
and ServiceMonitor, `config` laid into server.yaml, and the offline
|
||
checksums from a ConfigMap; read-only root, non-root, ready on `/leader`.
|
||
`deploy/container/Dockerfile` builds the image it runs. GPU nodes stay
|
||
outside Kubernetes and join with `node add` (docs/kubernetes.md says why).
|
||
|
||
### The admin API described, and a Terraform provider
|
||
|
||
- **The admin API in OpenAPI 3.1.** Every endpoint under `/api/v1`, its path
|
||
and query parameters, and the schema of each request and answer with the
|
||
meaning of each field, in
|
||
[docs/admin-api.openapi.json](docs/admin-api.openapi.json) and served by
|
||
every control plane at `GET /api/v1/openapi.json` (with the token, like the
|
||
rest). The bearer token, the `X-Llmgw-Actor` header and the error shape are
|
||
in it. Generated by `go generate ./internal/docgen` from a route table in
|
||
`internal/control/adminapi/routes.go` and the Go types the handlers use, with
|
||
the fields' comments as descriptions; a test fails when an endpoint is
|
||
served and not in the table, another when the committed document is stale.
|
||
Checked with openapi-spec-validator and openapi-generator-cli's validate.
|
||
- **[docs/admin-api.md](docs/admin-api.md)**: how to reach the API (loopback,
|
||
an SSH tunnel, `admin_remote`), the token at `/var/lib/llmgw/admin-token`,
|
||
what a request must carry — an address in `Host`, `application/json` on
|
||
every change, even a `DELETE` — and some calls with `curl`.
|
||
- **`GET /api/v1/models/{name}`**: one model's whole definition as it is in
|
||
force, with the defaults the catalog filled in. The listing has only its
|
||
state and its profiles' names.
|
||
- **A Terraform provider**, in `terraform-provider-llmgw/`, a module of its
|
||
own so the gateway does not carry its dependencies. `llmgw_model` (a model
|
||
added at runtime: scaling, class, owner and adapters change in place, the
|
||
rest replaces it), `llmgw_quota`, `llmgw_guard_attachment`, and the
|
||
`llmgw_models` and `llmgw_nodes` data sources. Its client is checked
|
||
against the OpenAPI document, endpoint by endpoint and field by field, so a
|
||
change to the admin API that breaks it fails its tests; CI runs them.
|
||
Built and installed by hand for now: see its
|
||
[README](terraform-provider-llmgw/README.md).
|
||
|
||
### Tracing beyond the router
|
||
|
||
- **The control plane traces what it does to the fleet.** A reconcile pass
|
||
that does something is a `reconcile` trace — passes with nothing to do are
|
||
not traced at all — with what it did beneath it: `scale up` with `place`
|
||
(the profile, engine, GPU class, pool, fabric and nodes chosen, or every
|
||
candidate turned down and why, as `ctl explain-placement` would print it),
|
||
`scale down`, `drain replica` and `stop replica` each with its reason,
|
||
`evict`, `preempt` with its victims, `reap replica` with the error and
|
||
when the model is tried again, `restart replica`, `restart refused`,
|
||
`node unreachable` and `node lost`.
|
||
- **A replica's start is one trace, down to the node.** `replica start`
|
||
stays open from placement until the replica serves, fails or is stopped,
|
||
with each shard's progress as an event and how long it took. The command
|
||
to each node is a `send start` span whose context travels in the command
|
||
(a new `trace` field on the agent protocol's `ControlMessage`), and the
|
||
agent continues the same trace: `start shard`, with `fetch weights`
|
||
(local, cached, downloaded, or downloaded by another shard; a `download`
|
||
with its bytes and where a resume began, the source's credentials and
|
||
query left out), `launch engine` (and `wait for rendezvous` on a worker
|
||
of a multi-node replica), and `wait for engine` with how often the engine
|
||
was asked and its last answer. "Why did qwen take four minutes to start"
|
||
is that one trace.
|
||
- **Stops, drains and adoptions on a node** are in the control plane's
|
||
trace too: `stop shard` and `stop engine`, saying whether the engine had
|
||
to be killed; `drain shard`; and `adopt` when a registering node reports
|
||
engines already running, with how many nobody claimed. An engine install
|
||
is an `install engine` trace on the node.
|
||
- **Model jobs are traces.** `ctl model serve` and `ctl model update` are
|
||
`serve model` and `update model`, with a span per step — registry,
|
||
mirror, check, catalog, placement, smoke test, replicas — and from the
|
||
placement step a link to the replica's own start.
|
||
- **The agent has a `tracing` block** in `agent.yaml`, the same as
|
||
`server.yaml`'s (`endpoint`, `sample_ratio`; `OTEL_EXPORTER_OTLP_ENDPOINT`
|
||
when unset). Point both at one collector. A node follows the control
|
||
plane's sampling for every start it is sent. Traces from a node carry its
|
||
id as `service.instance.id`, and a control plane's its host name.
|
||
- With no endpoint nothing changes: every span is the no-op one, and a
|
||
command carries no trace context.
|
||
|
||
### Quality suites, and new weights held to one
|
||
|
||
- **`llmgw ctl eval MODEL --suite FILE|NAME`** asks a model a quality suite
|
||
and scores its answers: per case, then the cases weighed into one score, 0
|
||
to 1, against the suite's `threshold` (or `--min-score`); it exits non-zero
|
||
below it. `--concurrency`, `--judge`, `--detach`, `--json`.
|
||
- **Suites** are YAML or JSON lines: a prompt or a conversation, a weight,
|
||
and one way to score it — `exact`, `contains`, `regex`, `number` within
|
||
a `tolerance` (the last number in the answer), `json_schema` (the answer
|
||
read as JSON, in a code fence or a sentence or not, held to a schema), or
|
||
`judge`: a rubric another model on the cluster grades 0 to 1. A
|
||
reasoning model's `<think>` is not part of its answer.
|
||
- **Run on the control plane**, through the router as its own requests,
|
||
as a replay is: past admission, quotas, accounting and the guardrails.
|
||
`--suite` is a file sent whole, or a suite's name in `control.eval_dir`
|
||
(`/etc/llmgw/eval`) or built in: `arithmetic` and `json-format`, in
|
||
`deploy/eval/`, sized for small models. `--suites` lists them.
|
||
- **Kept** in the store (migration 0028): every run with its model, the
|
||
repository and revision of the weights that answered, the suite's name
|
||
and hash, its score and each case's answer and reason.
|
||
`llmgw ctl eval --history [MODEL]` lists them, `--run ID` shows one.
|
||
- **Updates and promotions held to a suite.** `llmgw ctl model update` and
|
||
`model promote` take `--eval SUITE --min-score N`, or a model carries one
|
||
(`llmgw ctl model gate MODEL --eval SUITE`, `eval:` in its definition).
|
||
A promotion asks the canary first and leaves it in place below the score;
|
||
an update runs the new weights first as a canary given none of the
|
||
model's requests, asks it, and promotes it or takes it out again, so the
|
||
old weights serve until the new have passed. The job's `eval` step says
|
||
the score; `--skip-eval` moves once without the model's suite.
|
||
- **The console** has **Ask it a quality suite…** (the score counted in as it
|
||
comes), **Its quality runs** and **Hold its updates to a quality suite…**
|
||
on a model's menu.
|
||
- **Fixed:** a change to the admin API with a body over 64 KiB reached its
|
||
handler cut off at 64 KiB, by the audit log's reading of it.
|
||
|
||
### MIG instances as GPUs of their own class
|
||
|
||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||
any other, and placement puts a replica on a free instance of it. Both
|
||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||
instances; `docs/install.md` has a MIG section on making them.
|
||
- **One instance a replica.** A process opens only one MIG device, and
|
||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||
node is refused when the model is added, with that reason; placement
|
||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||
for a model one holds and never several together.
|
||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||
shows the device as `0/mig2`.
|
||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||
and the processes holding it, are its own — read per GPU and compute
|
||
instance from nvidia-smi's report, so a foreign process on one instance
|
||
keeps that instance out of placement and not its neighbours. Temperature,
|
||
power and wear are the card's. No driver reports utilisation per instance,
|
||
so none is published for one: the console says `n/a on MIG`, and
|
||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||
is on.
|
||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||
command for each.
|
||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||
GPU list shows instances under their card with their own memory and the
|
||
card's heat and power, and a node's page lists its classes with counts.
|
||
`ctl status` and the plan count each device under its own class, where a
|
||
node's cards were all put under its first class before.
|
||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||
|
||
### Engines and the mirror
|
||
|
||
- **TensorRT-LLM as an engine** (`engine: trtllm`), on nodes where you
|
||
installed it: name its interpreter in the node's `agent.yaml`
|
||
(`engines.trtllm.python`) and the node offers it while that interpreter
|
||
has the `tensorrt_llm` package. llmgw does not install it. The agent starts
|
||
`trtllm-serve serve` with the replica's host, port, `--tp_size`,
|
||
`--pp_size` and served name, through a few lines of its own that make it
|
||
demand the gateway's key on every path but `/health` (trtllm-serve has no
|
||
key option), turn on the Prometheus endpoint and iteration statistics, and
|
||
keep usage reports to NVIDIA off. The router reads it at
|
||
`/prometheus/metrics` — `trtllm_num_requests_running`,
|
||
`trtllm_num_requests_waiting`, `trtllm_kv_cache_utilization`, tokens and
|
||
successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer
|
||
GPU; one node per replica. Profiles may not give it an options file,
|
||
`--trust_remote_code`, `--grpc`, `--num_serve_frontends` or the flags the
|
||
agent sets; `--set PATH=VALUE` changes a setting instead. `llmgw ctl
|
||
engine-check trtllm --bin <python>` checks an installed one. Written
|
||
against TensorRT-LLM's documented command line, server and metrics and
|
||
tested with stand-ins for them; not yet run on a datacenter GPU.
|
||
- **`llmgw mirror import-ollama [NAME[:TAG]]`** puts a model you already
|
||
have in Ollama into the mirror for llama.cpp: its GGUF, and a vision
|
||
model's projector beside it, each checked against the sha256 Ollama named
|
||
its file by. With no name it lists Ollama's models; it finds Ollama's
|
||
directory itself (`OLLAMA_MODELS`, `~/.ollama/models`, Ollama's Linux
|
||
service) or takes `--ollama-dir`. The version is the tag and the start of
|
||
the digest, since a tag moves in Ollama. It prints a model definition to
|
||
add, with Ollama's `num_ctx` as `--ctx-size`. The console's **Mirror**
|
||
view does the same with **I**, for the mirror beside the control plane.
|
||
- **`llmgw mirror import FILE... --model NAME`** copies files already on the
|
||
mirror machine into it the way a download is kept: whole or not at all, the
|
||
checksum beside it, a version never replaced, a `.gguf` checked to be one,
|
||
and `--sha256` checked when given.
|
||
|
||
### The portal in Turkish
|
||
|
||
- **The web portal speaks English and Turkish** — the landing page, sign-up
|
||
and sign-in, the account page and its editor setup, the claim page, chat,
|
||
the about page, and every administrators' page (cluster, nodes, NVLink and
|
||
fabrics, people, limits, usage, organizations, mirror, guardrails), and
|
||
the docs portal around its documents, which stay English. Flash
|
||
messages and form errors built by the server are translated too.
|
||
- **A switcher at the foot of every page** (English · Türkçe) remembers the
|
||
choice in a cookie; without one the browser's Accept-Language decides, and
|
||
without that the new `landing.language` (`en` or `tr`, empty is English).
|
||
An unknown code stops the server starting. `<html lang>` and
|
||
Content-Language follow the page.
|
||
- **Figures and dates in the page's language**: a Turkish page writes
|
||
1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in
|
||
Turkish. Form values, CSS and SVG coordinates are unchanged.
|
||
- **Ready for more languages**: the catalogue is JSON under
|
||
`internal/web/locales/<code>/`, and a language is a directory. Tests refuse
|
||
a key a page uses that some language lacks, a translation with different
|
||
arguments from English, and any template text left in English — so a new
|
||
page cannot quietly ship English-only.
|
||
- Messages that come from the control plane and other packages (node
|
||
warnings, engine errors, store refusals) are still shown as written, in
|
||
English; the API, CLI and console are English.
|
||
|
||
### Documentation
|
||
|
||
- **A docs portal at /docs.** The manual the binary carries — the pages
|
||
`llmgw docs` prints and the console's Manual shows, the configuration
|
||
reference among them — rendered on the client port in the portal's own
|
||
look: a sidebar of the documents in reading order, the open one's sections
|
||
under it, a link on every heading, and links between documents kept inside
|
||
the portal (links to files that are not in the binary, such as `deploy/`,
|
||
go to the repository). No script on the page, so the Content-Security-Policy
|
||
has nothing to allow. Public by default, like the landing page; set
|
||
`landing.docs_sign_in: true` to send visitors to sign in first. Signed in,
|
||
the page carries the usual menu, which now has a **Docs** link.
|
||
- **A client developer's guide, `docs/clients.md`.** How to use llmgw from
|
||
the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is
|
||
needed — in Python, TypeScript, Go and curl: the base URL and key,
|
||
streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK
|
||
against `/v1/messages`, and what llmgw adds as headers (`x-llmgw-thread`,
|
||
`x-session-id`, metadata and `x-llmgw-tags`, `x-llmgw-variant`,
|
||
`/v1/memory`, `/mcp`), with the statuses a client meets. The portal's index
|
||
leads with it, and the landing page and the account page link to it.
|
||
- The console's Manual and the portal read one list of documents in one
|
||
order (`llmgw.Manual`), so the two cannot drift.
|
||
|
||
### Administration
|
||
|
||
- **An audit log.** Every change made through the admin API (`llmgw ctl`,
|
||
the console) or the pages under /admin is recorded: who, when, through
|
||
what, the fields it was given with passwords, keys and tokens taken out,
|
||
and how it ended — refusals included, in words. So are a person's own new
|
||
keys, revoked keys and password changes. Read it with `llmgw ctl audit`
|
||
(`--since`, `--actor`, `--contains`, `--detail`), `GET /api/v1/audit`, or
|
||
the console's **Audit** view. It moves with the store.
|
||
- **Staff roles** besides administrator: `viewer` looks at every
|
||
administration page and changes nothing, `operator` also runs the cluster
|
||
from them, `billing` sees usage and quotas. Set with `llmgw users role` or
|
||
the console's **Staff role…**.
|
||
- **Single sign-on through OpenID Connect** (`auth.oidc`). A "Sign in
|
||
with …" button sends people to the identity provider and back, with PKCE,
|
||
state and nonce. The ID token is verified, and the account is found by
|
||
the verified email address, or made when `create_accounts` is on. Groups
|
||
give staff roles (`roles`, set at every sign-in) and places in the tree
|
||
(`memberships`). `no_passwords` leaves the provider as the only way in,
|
||
except for administrators.
|
||
- **Provisioning through SCIM 2.0** (`auth.scim.token_file`, `/scim/v2`).
|
||
- **Users:** the identity provider makes, suspends and reinstates
|
||
accounts — the PATCH forms Entra ID and Okta send are both read.
|
||
Deleting a user suspends the account.
|
||
- **Groups:** joining and leaving the groups `auth.oidc` maps gives and
|
||
takes away staff roles and places in the tree.
|
||
- **Key lifecycle.**
|
||
- **Lifetime limits:** `auth.max_key_ttl` limits how long any key may
|
||
live, and `llmgw keys ttl team:Acme/ml 720h` sets shorter limits for an
|
||
organization, team or project. A key is made within the shortest that
|
||
applies, and one asked for with no expiry is given that longest — from
|
||
the CLI, the profile page and the editor connect page alike.
|
||
- **Rotation:** `llmgw keys rotate <id> --overlap 48h` issues a
|
||
replacement where the old key was, and the old one stops after the
|
||
overlap.
|
||
- **Listing:** `keys list --all` shows each key's kind — service (a
|
||
project's), member, personal — with service keys first, and when it
|
||
expires.
|
||
- **Usage export.** `llmgw ctl usage export --from 2026-09-01 --to
|
||
2026-10-01 --by project,model` writes CSV by hour, day or month and any of
|
||
organization, team, project, user, key and model.
|
||
- **Data:** read from the hourly and daily sums, so it reaches back past
|
||
the request records' retention.
|
||
- **Format:** cost is given in micro-units and units, and names that
|
||
look like a spreadsheet formula are quoted.
|
||
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
|
||
and **E** on the console's Usage view.
|
||
|
||
### Forgetting a person, and requests that stay in the cluster
|
||
|
||
- **`llmgw users forget`** erases a person for a request to be forgotten:
|
||
their usage kept under a random pseudonym so totals stay true, their kept
|
||
requests, personal files, batches and limits deleted, their name taken out
|
||
of the audit log, the account removed.
|
||
- **`llmgw keys residency --project P --in-cluster`**: no model served by an
|
||
outside provider for the project's keys, asked for, as a fallback or as a
|
||
canary.
|
||
- Both are on the console's menus.
|
||
|
||
### Secrets in Vault
|
||
|
||
- **Secrets from Vault or the environment.** Every setting naming a secret's
|
||
file also takes `env:VARIABLE` or `vault:mount/path#field` (HashiCorp
|
||
Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN).
|
||
- **The store's own secrets sealed** under `secrets.master_key`: Vault's
|
||
transit engine (`vault-transit:KEY`), or a 32-byte key read from a
|
||
reference. A store, or a backup of it, then does not hold what engine keys
|
||
are derived from. Values from before are sealed as they are next read.
|
||
|
||
### Scaling by the clock
|
||
|
||
- **`scaling.schedule`**: windows of the week with their own `min` and
|
||
`max` — more replicas through working hours, an on-demand model kept
|
||
loaded while people are at work — in `control.timezone`.
|
||
- **`scaling.prewarm`** opens a window early, so its replicas have loaded
|
||
when it starts.
|
||
|
||
### Token-level timing, and admission by SLO
|
||
|
||
- **Per-token metrics:** `llmgw_tpot_seconds`, `llmgw_inter_token_seconds`
|
||
and `llmgw_output_tokens_per_second` by model and pool, and TPOT and
|
||
tokens a second by tenant.
|
||
- **A model's `slo` counts in admission:** past its `ttft_p99` or
|
||
`tpot_p99` (the recent p99), it is treated as saturated, so the lower bands
|
||
wait. `llmgw_model_slo_seconds` and the `llmgw-slo` alerts make the
|
||
promise visible in Prometheus.
|
||
|
||
### Restoring a backup, and copies off the machine
|
||
|
||
- **`llmgw server restore ARCHIVE`** puts a backup back: checks it first
|
||
(whole, opens with `--identity`, not newer than this llmgw, where the store
|
||
goes), says so with `--dry-run`, then moves `/etc/llmgw` aside, restores it
|
||
with modes and owners, restores the store (SQLite, or PostgreSQL with
|
||
`pg_restore --clean`), and starts the server. A drill is in
|
||
docs/install.md.
|
||
- **`backup_copy_to`** copies each archive to a directory or over ssh as it
|
||
is written, keeping as many there as here; refused without
|
||
`backup_recipients`.
|
||
- Archives now record each file's owner by name, so a restore on another
|
||
machine gives the llmgw user its files back.
|
||
|
||
### Anthropic's Messages API
|
||
|
||
- **`/v1/messages`** (and `/v1/messages/count_tokens`) translated to chat
|
||
completions and back, streamed or not: text, images, tools and their
|
||
results, the system prompt, stop sequences. Claude Code and Anthropic's
|
||
SDKs reach the models here; the key may come in `x-api-key`. Errors come
|
||
back in Anthropic's shape.
|
||
|
||
### More of OpenAI's API
|
||
|
||
- **Responses:** `text.format` asks for structured output, `input_image`
|
||
sends images, and `previous_response_id` continues a conversation the
|
||
gateway keeps as a thread (for a project that keeps conversations).
|
||
- **Passed through:** `/v1/rerank` and `/v1/score` for vLLM's rerankers, and
|
||
`/v1/audio/transcriptions` and `/translations` as the multipart form came,
|
||
for an engine serving Whisper.
|
||
- **`/v1/moderations`** is answered by the guardrails' judge model.
|
||
|
||
### Requests kept, and replayed
|
||
|
||
- **A project can keep its requests and answers** (`llmgw keys requests
|
||
--project P --retain-hours N [--sample 0.05]`, or **Keep requests…** in the
|
||
console): what the model was sent, after redaction, and what the client
|
||
got. Off until asked; never under `no_store`; pruned hourly.
|
||
- **`llmgw ctl request ID --body`** shows one; **`--replay MODEL`** sends it
|
||
to another model and shows both answers.
|
||
|
||
### Batches
|
||
|
||
- **OpenAI's Batch and Files APIs.** `POST /v1/files` (purpose batch),
|
||
`POST /v1/batches`, and reading, listing, cancelling and deleting them, so
|
||
OpenAI's clients send batches here unchanged.
|
||
- **Serving:** each line goes as a request of the key that made the batch,
|
||
in the lowest admission band, a few at a time; rate limits are waited
|
||
out. Answers and failures are written to files kept 30 days.
|
||
- **Accounting:** each line is an ordinary usage record tagged
|
||
`batch=<id>`.
|
||
- **Restarts:** answered lines are kept as they are answered, and any
|
||
gateway on the store carries a batch on under a lease.
|
||
- **Limits:** `router.batch` bounds the file, the lines and the
|
||
concurrency. Refused to a project under a `no_store` policy.
|
||
|
||
### New weights tried on a share of the traffic first
|
||
|
||
- **Canaries.** `llmgw ctl model canary MODEL --percent 10` fetches the newer
|
||
weights of a model's repository and runs them beside it as `MODEL-canary`,
|
||
given that share of its requests.
|
||
- **Routing:** a session stays on one side; `x-llmgw-variant` pins a
|
||
request to `canary` or `stable` and says which served it. Clients cannot
|
||
name the canary.
|
||
- **Rolled back by itself** when, over 50 of its requests or more, its
|
||
error rate is five points above the model's or its time to the first
|
||
token half again as long; `canary_rolled_back` is sent to the
|
||
notification webhooks.
|
||
- **Then:** `--percent` changes its share, `llmgw ctl model promote`
|
||
moves the model to it, `--abort` takes it out, and `llmgw ctl model
|
||
canaries` compares the two sides. All of it is also on the console's
|
||
model menu.
|
||
|
||
### Requests can be tagged
|
||
|
||
- **Tags and end users on every usage record.** OpenAI's `metadata` and the
|
||
`x-llmgw-tags` header (`feature=search, env=prod`) tag a request; OpenAI's
|
||
`user` field names the client's own end user. Both are kept with the
|
||
request's usage and shown by `llmgw ctl request`. Tags are not sent to the
|
||
engine. A malformed one is refused with 400.
|
||
- **Counted by tag, for the tags named.** `router.tag_labels` lists the tags
|
||
counted in `llmgw_tagged_requests_total` and `llmgw_tagged_tokens_total`,
|
||
the first 100 values of each apart and the rest as `other`.
|
||
|
||
### A hardened install by default
|
||
|
||
- **`server init` and the setup screens write a hardened configuration.**
|
||
- **Guardrails:** a `baseline` policy on every request. It refuses
|
||
secrets in prompts and answers, masks card numbers and records injection
|
||
attempts.
|
||
- **The judge:** Llama Guard 3 1B, two replicas, fetched and run on CPUs
|
||
(about half a second for a short text). It refuses violence, crime,
|
||
weapons, child abuse, privacy violations, hate, self-harm and sexual
|
||
content. The policy fails closed: a request it cannot read within 5
|
||
seconds is refused (503), and `judge_unavailable` goes to the
|
||
notification webhooks.
|
||
- **Keys:** they live at most 90 days.
|
||
- **Opting out:** `--no-guard-model` (or unticking it on the setup screen)
|
||
leaves the judge out and keeps the pattern checks. A `server.yaml`
|
||
written earlier is never changed.
|
||
- **What is left to you:** the closing text now names the two things
|
||
init cannot do — a certificate for the client port, and the keys
|
||
backups are encrypted to.
|
||
|
||
### Notifications
|
||
|
||
- **Webhooks for what needs a person** (`notify.webhooks`): a model parked,
|
||
a node lost, a budget spent, new weights in a model's repository — sent by
|
||
the gateway itself, with no Prometheus in between.
|
||
- **Formats:** JSON signed with HMAC-SHA256 (`X-Llmgw-Signature`) when
|
||
`secret_file` is set, or a Slack-compatible `{"text": …}`.
|
||
- **Delivery:** queued, never holding up what raised it; tried three
|
||
times; the same event about the same thing at most once an hour, and new
|
||
weights once per revision.
|
||
- **Checking:** `llmgw ctl notify test`.
|
||
- **A judge not answering** (`judge_unavailable`) is one of the events.
|
||
- **Alertmanager receiver templates** for Slack, e-mail, PagerDuty and a
|
||
webhook in `deploy/monitoring/alertmanager.yml`, commented out until
|
||
chosen.
|
||
|
||
### Guardrails
|
||
|
||
- **Hooks for guardrails on the request path** (nothing judges yet; B2 adds
|
||
the policies). A guard, when one is set, judges a request once it is
|
||
authorised, within its limits and has its stored conversation spliced
|
||
in, and before it is dispatched; and filters the answer, buffered or
|
||
streamed, before the client sees it. A blocked request is refused with 400
|
||
and `code: "content_policy_violation"` — the first error with a code — and
|
||
costs nothing; a blocked answer ends with `finish_reason: "content_filter"`,
|
||
and a stopped stream with that chunk and `[DONE]`, the engine cancelled.
|
||
The Responses API reports either as `incomplete` with
|
||
`incomplete_details.reason: "content_filter"`. `x-llmgw-guard` and
|
||
`x-llmgw-guard-policy` say what was done; the usage row records the
|
||
action, policy, category and time taken — never the text.
|
||
- **Guardrail policies** (`guard:` in server.yaml; see docs/guardrails.md).
|
||
- **What is found:** personal data, each kind validated before it counts
|
||
(email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits,
|
||
IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt
|
||
injection seen through disguises (look-alike letters, invisible
|
||
characters, base64/hex/URL encoding); deny patterns.
|
||
- **What is done:** flag, redact as `[EMAIL_1]`, or block. A request can
|
||
be shaped too: a pinned system prompt, the caller's own system messages
|
||
dropped, an output limit, tools stripped or refused.
|
||
- **Answers:** judged buffered, or a sentence at a time when streamed,
|
||
never releasing half a value; tool calls are checked against allow
|
||
lists, a count and the request's own schemas.
|
||
- **Modes:** `shadow` records what it would do; `no_store` refuses
|
||
conversations, notes and models outside the cluster.
|
||
- **Where they apply:** `guard.default` everywhere; others attached with
|
||
`llmgw ctl guard attach` to an organization, team, project, user or
|
||
key, and narrowed like limits.
|
||
- **A judge model on the cluster** (`role: guard` in the catalog, `judge:`
|
||
in a guard policy). It reads what patterns cannot — Llama Guard 3,
|
||
ShieldGemma or a JSON-answering model — and its verdict flags or blocks
|
||
input, output or both.
|
||
- **Calls:** past admission and accounting, bounded by a latency budget
|
||
(250ms by default), after which the policy's `fail` decides.
|
||
- **Caching and streams:** answers are cached for five minutes, and a
|
||
streamed answer is read every 600 characters.
|
||
- **Isolation:** a judge model is hidden from clients and refused to them.
|
||
It is critical by default, and cannot be loaded on demand.
|
||
- **Seeing what the guardrails do.**
|
||
- **Metrics:** `llmgw_guard_checks_total` (stage, policy, action,
|
||
category, mode), latency for checks and for the judge, judge errors,
|
||
fail-open, redactions by kind, stopped streams, and blocks per tenant.
|
||
- **Alerts:** judged traffic served unjudged, a slow or unreachable
|
||
judge, a tenant's blocks spiking, a secret in an answer.
|
||
- **Pages and commands:** `/admin/guard` shows counts by place, policy
|
||
and category and the recent requests by id; `ctl request` shows the
|
||
verdict; the profile page lists the policies in force where the person
|
||
works; the console's organization rows have **Guardrails…** and a
|
||
column.
|
||
- **The chat page** says when a request was refused or an answer stopped
|
||
by policy, instead of showing an error.
|
||
|
||
### Security
|
||
|
||
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
|
||
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
|
||
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
|
||
tests and vet.
|
||
- **Releases are signed, and only signed releases are installed.** The build
|
||
now carries the release signing key (`internal/release/trusted.pub`); the
|
||
control plane upgrading itself, an agent told to upgrade and a node being
|
||
added all refuse a release whose checksums are not signed with it. The CI
|
||
release job stops rather than publish an unsigned release. Adding a node
|
||
no longer has the node fetch the release and compare it with a checksum
|
||
from the same server: the control plane fetches it for the node's
|
||
architecture, checks the signature and the checksum, and hands it over.
|
||
Releases are fetched over https only (loopback excepted), and
|
||
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
|
||
- **The agent port is not opened without TLS by accident.** A control plane
|
||
with no `agent_tls` refused nothing: its agent port was plaintext, on every
|
||
interface, and any machine reaching it could register as any node — be
|
||
given replicas and so be sent prompts. It now refuses to start unless
|
||
`agent_insecure: true` is set, and then listens on loopback unless
|
||
`agent_listen` names an address. `server init` writes `agent_tls`, as
|
||
before.
|
||
- **A model definition cannot run code on its nodes.** A profile's `env` and
|
||
`args` reach a process the agent runs as root. The environment was checked
|
||
against a short denylist that missed NCCL's and CUDA's plugin and
|
||
injection libraries; arguments were not checked at all, so
|
||
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
|
||
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
|
||
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
|
||
name for a path, plugin, library, address or credential, and llama.cpp's
|
||
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
|
||
writes or loads files, loads code, or moves what the agent sets (host,
|
||
port, key, model). Both are refused where a model is defined and again on
|
||
the node before the engine starts.
|
||
- **Engines no longer run as root.** The agent starts them as the system
|
||
user `llmgw-engine` (in the video and render groups for the GPUs), with a
|
||
home and caches of its own under the agent's work directory. `llmgw node
|
||
add` makes the user; a node without it keeps running engines as root and
|
||
the agent says so in its log. `supervisor.run_as` in agent.yaml names
|
||
another user, or `root`.
|
||
- **The admin API wants a token.** It listened on loopback with no
|
||
authentication, so any process or user on the control plane could add a
|
||
model, upgrade the fleet, read the engines' keys or dump the heap through
|
||
pprof. Its `/api` and `/debug` paths now demand the token the server
|
||
writes, 0600, to its state directory (`/var/lib/llmgw/admin-token`) on
|
||
first start; `llmgw ctl`, the console and `node add` read it, so they run
|
||
with sudo, and `LLMGW_ADMIN_TOKEN` or `ctl --token-file` give it from
|
||
elsewhere. pprof is off unless `debug_pprof: true`. `admin_listen` off
|
||
loopback is refused unless `admin_remote: true`. The read-only dashboard
|
||
pages, `/metrics` and `/health` are unchanged.
|
||
- **The replicated store asks for a password and speaks TLS.** Patroni's
|
||
REST API took changes from anyone who reached port 8008 — rewriting the
|
||
access rules to `trust`, or moving the leader — and PostgreSQL traffic
|
||
between the machines was plaintext (`sslmode=disable`). Patroni now asks
|
||
for a password before a change (kept with the cluster's others in
|
||
`/root/llmgw-pg`, and on the control plane in
|
||
`/etc/llmgw/secrets/patroni-rest` for `server add`); each member has a
|
||
certificate from the cluster's authority, PostgreSQL has `ssl` on, every
|
||
remote access rule is `hostssl`, and the control planes connect with
|
||
`sslmode=verify-full` against the authority's certificate. A new cluster is
|
||
set up this way; an existing one is changed in place, by reload, the
|
||
cluster serving throughout, by running **Replicate the store** in the
|
||
console again — or `sudo --preserve-env=SSH_AUTH_SOCK llmgw server init
|
||
--patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node
|
||
--no-model` — on the control plane that holds the authority, then on each
|
||
other control plane so it reconnects by TLS. etcd is not yet given TLS:
|
||
keep ports 2379 and 2380 to the cluster's machines.
|
||
- **A capacity fallback respects the key's allowed models, and is charged
|
||
at its own price.** When the model asked for had no capacity, the next in
|
||
its `fallbacks` served the request even if the key was not allowed it, and
|
||
the request was priced as the model asked for. Fallbacks a key may not use
|
||
are now skipped, and a request is charged what the model that answered
|
||
costs.
|
||
- **Suspending an account stops its keys.** Suspension closed the person's
|
||
sessions and left every API key they had made working. A key whose owner's
|
||
account is not active — suspended, pending, or gone — is now refused as a
|
||
revoked key is, within `auth.cache_ttl`, and works again when the account
|
||
is reinstated.
|
||
- **A join code is a credential only for its hour, and only on its
|
||
address.** The setup checked a code's expiry on the joining side alone, so
|
||
a code stayed good for as long as the setup ran; it now refuses an expired
|
||
one itself. Its listener binds the address the codes name rather than
|
||
every interface, and a joined machine is known by the address it connected
|
||
from, not the one it claimed.
|
||
- **The setup's sudo grant cannot outlive it.** Given a sudo password, the
|
||
setup writes a temporary `NOPASSWD` grant and removes it when it
|
||
finishes; a setup that crashed or was interrupted left it for good. The
|
||
grant now schedules its own removal within the hour as it is written, and
|
||
the password is dropped once the grant stands in for it. An address given
|
||
to the system ssh can no longer be read as one of its options.
|
||
- **Made-up keys cannot fill memory or the database.** Every well-formed key
|
||
presented, real or not, was cached for good and cost a database read the
|
||
first time. The cache is now bounded, and keys not in it are looked up at
|
||
most 20 a second per client address (50 at once); beyond that the request
|
||
is answered 429 without a read.
|
||
- **Slow or idle connections cannot pile up.** The servers bounded only how
|
||
long headers took. Every listener now also bounds how long a connection
|
||
may sit idle and how large its headers may be, and reading a request
|
||
(5 minutes on the client port, a minute or less elsewhere — never on the
|
||
agents' long-lived streams); the client port holds at most
|
||
`max_connections` (10,000) at once, and a message from an agent at most
|
||
16 MB.
|
||
- **The sign-in and signup forms resist scripts.** Ten wrong passwords from
|
||
anywhere locked an address's owner out for the window; attempts are now
|
||
counted per address and client, so somebody guessing elsewhere cannot lock
|
||
the owner out, with a looser count per address alone so guesses spread
|
||
over many clients are still bounded. The sign-in form carries a token its
|
||
cookie matches, so another site cannot sign a visitor into an account of
|
||
its choosing. Posting either form is rationed per client address (sign-in
|
||
30 a minute, signup 5), and a password hash that cannot get a turn within
|
||
ten seconds answers "busy" instead of holding the request. `server init`
|
||
now writes `signup_open: false`.
|
||
- **Forwarding headers are believed only from the proxies named.**
|
||
`X-Forwarded-Host` and `-Proto` were taken from anyone, and decide the
|
||
gateway address written into the `/setup` scripts clients pipe to a
|
||
shell; behind a caching proxy one request could poison them for others.
|
||
They — and `X-Forwarded-For`, now used for the per-address limits — are
|
||
believed only from `router.trusted_proxies` (addresses or CIDR prefixes).
|
||
**Behind a reverse proxy, set it**, or the pages and scripts name the
|
||
address the proxy reached and every client counts as the proxy. `/setup`
|
||
scripts are served `no-store`.
|
||
- **Removing a node revokes its certificate.** `ctl node remove` only
|
||
dropped the node from the inventory; its certificate, valid for a year,
|
||
still connected and was given work. Now the removal is recorded in the
|
||
store and the agent port refuses any certificate issued for that node
|
||
before it — onboarding the node again issues one that works. A mirror
|
||
serving its management API with `--tls-ca` now requires `--api-client`
|
||
(the control plane's name), since the same authority signs every node.
|
||
- **Engines listen on the management address only**, the one the control
|
||
plane reaches them on, instead of every address the node has — a public
|
||
one included. (When that address is a name, or not one of the node's,
|
||
they still listen on all.) **SGLang's key is off its command line**,
|
||
where every user on the node could read it: it reaches the engine in its
|
||
environment, and a short launcher hands it to SGLang inside the process.
|
||
Both take effect as engines restart under an upgraded agent.
|
||
- **Backups can be encrypted.** `backup_recipients` in server.yaml (age
|
||
recipients or SSH public keys) has `llmgw server backup` encrypt each
|
||
archive to them with age, as `.tar.gz.age`: an archive holds the CA's key
|
||
and every secret in the store, and is meant to be copied off the machine.
|
||
Open one with `age -d -i <key>`.
|
||
- **PyTorch's index gives torch and nothing else.** vLLM's CUDA 12.9 build
|
||
was resolved with PyTorch's index beside PyPI and `unsafe-best-match`, so
|
||
either index could supply any package — how torchtriton was hijacked in
|
||
2022. Now that index is explicit, named only for torch, torchvision and
|
||
torchaudio; every other package comes from PyPI alone, and the lock is
|
||
PEP 751's (`pylock.llmgw.toml`), pinning each file's address as well as
|
||
its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
|
||
- **The control plane's own fetches stay off link-local addresses.**
|
||
Checking a model's source and reading a GGUF's header went wherever the
|
||
URL said, cloud instance metadata (169.254.169.254) included. They now
|
||
refuse link-local, multicast and unspecified addresses, while still
|
||
reaching the cluster's network, where the mirror is. The mirror's fetches,
|
||
which refuse private addresses too, now also see through IPv6 addresses
|
||
carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy:
|
||
the target's addresses are checked before the request is handed to it.
|
||
- **A CPU-only node reporting a problem no longer panics the agent
|
||
handler**, and **thread ids are bounded** to 128 letters, digits and
|
||
`- _ . :`. A thread is readable by every key of its project that knows
|
||
the id; the docs now say to use unguessable ids.
|
||
- **`/v1/models` lists only what the key may use**, rather than every
|
||
model the gateway serves to anyone.
|
||
- **Portal sessions end after a day unused** (still at most 14 days in
|
||
all), and expired or idle sessions and used or expired key claims are
|
||
deleted hourly — they were refused, but never removed.
|
||
- **Every page carries a Content-Security-Policy**: scripts only from the
|
||
gateway itself and the page's own inline ones, each named by its hash —
|
||
no inline handlers, no eval, nothing a model's reply or a model card
|
||
could slip into a page. The few `onchange`/`onsubmit` attributes moved
|
||
to `/_static/pages.js`, htmx runs with `allowEval: false`, and the live
|
||
pages' "only while visible" refresh no longer needs an evaluated filter.
|
||
Pages now carry their headers even when answered with an error status.
|
||
- **KaTeX 0.16.47** (from 0.16.11), past the advisories fixed in 0.16.21.
|
||
- **Smaller hardening.** The mirror no longer serves its bookkeeping files
|
||
(a file's `.source` URL and `.sha256`) on `:8081`. A GGUF header nesting
|
||
arrays more than 4 deep is refused rather than recursed into. Serve and
|
||
update jobs keep the last 100 finished, and the mirror forgets finished
|
||
downloads after a week, instead of remembering every one until restart.
|
||
- **A firewall for llmgw's internal ports.** `server init` (and the store
|
||
members it sets up with `--patroni`), `node add`, `server add` and
|
||
`mirror init` now keep the machine's internal ports — the agents' stream,
|
||
engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the
|
||
cluster's own networks, in an nftables table of llmgw's own loaded by
|
||
`llmgw-firewall.service`. Only those ports are touched: the table accepts
|
||
everything else, so other services on the machine and any firewall it
|
||
already has are left as they were. Extra networks go in
|
||
`firewall_networks`; `--no-firewall` skips it; `llmgw doctor` says when a
|
||
node lacks it.
|
||
- **etcd and Patroni's API speak TLS.** etcd's peers and clients use
|
||
certificates from the cluster's authority, and a client must present one;
|
||
Patroni reaches it with its member's certificate and serves its API over
|
||
TLS. `server init --patroni`, run again, moves an existing cluster: with
|
||
Patroni in maintenance mode, so PostgreSQL serves on and nothing fails
|
||
over, etcd is rebuilt on TLS and Patroni's keys carried into it. The
|
||
control plane asks the API over HTTPS, verified, and over HTTP only where a
|
||
member has not moved yet.
|
||
- **A cluster keeping its access rules per member got no `hostssl`.** The
|
||
TLS step looked only at the shared configuration, and on a cluster set up
|
||
before the rules moved there said the rules were `hostssl` while
|
||
plaintext was still accepted. Each member's own rules are now made
|
||
`hostssl` too.
|
||
|
||
|
||
### Fixed
|
||
|
||
- **An engine that died after an agent restart went unnoticed.** The watcher
|
||
that reports a ready engine dying was tied to the stream it was started
|
||
on, so after a reconnect nothing reported it, and an engine adopted from
|
||
the previous agent had no watcher at all: its replica stayed "ready" and
|
||
was routed to. Every connection now watches every ready shard, started or
|
||
adopted. Found on the PoC: a killed engine on node06 stayed ready for
|
||
minutes.
|
||
- `server init` rewriting the store's address no longer leaves
|
||
`/etc/llmgw/secrets` unreadable to the service when it was made 0700 for
|
||
the service user, which kept the gateway from starting.
|
||
### Everything that can be changed on a running cluster, from the console
|
||
|
||
`sudo llmgw` only watched the cluster and edited the configuration files;
|
||
changing a model's replicas still meant `llmgw ctl`. Now Enter on a row opens
|
||
what can be done to it, as forms and questions:
|
||
|
||
- **Models**: replicas (min, max, idle timeout), class, owner, resources, LoRA
|
||
adapters, restart, retry when parked, why it is placed where it is, remove;
|
||
A adds one from a definition file.
|
||
- **Nodes**: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its
|
||
agent, delete model files nothing uses, remove.
|
||
- **Replicas**: restart, replaced or in place.
|
||
- **Quotas** (new): capacity quotas, set and removed.
|
||
- **Mirror** (new): what it holds and which models use it; download into it,
|
||
remove from it.
|
||
- **People** (new): accounts — approve, suspend, reinstate, the cluster
|
||
dashboard, memberships, limits, a new password, delete, create; API keys —
|
||
issue (shown once), limits, revoke; organizations, teams and projects — add,
|
||
rename, move, remove, limits for the level and for each member, capacity
|
||
quotas, how long conversations and agent notes are kept.
|
||
- **Serve a model** (new view): GGUF models on Hugging Face — trending, most
|
||
downloaded, most liked or newest, or searched with / — then one of the
|
||
chosen repository's files, a name and a replica count, and it is fetched,
|
||
checked, added and started. The registry is mirror.registry's when set.
|
||
- **Do**: upgrade the cluster, restart every replica, back up now.
|
||
|
||
The people side runs the same commands as `llmgw users`, `keys` and `org`,
|
||
against the same store, so their checks and messages are the ones shown. A
|
||
sidebar taller than the terminal now scrolls with the current entry.
|
||
|
||
### New weights in a model's repository, found and moved to
|
||
|
||
A repository on Hugging Face can publish new weights under the same file
|
||
names — a fixed chat template, a better quantisation — and nothing said so. A
|
||
model served from the registry now records its origin: the repository, the
|
||
commit, and each file's checksum there; a model served before is worked out
|
||
from the mirror's record of where it fetched each file. Every
|
||
`mirror.update_check` (six hours by default; 0 turns it off) the control
|
||
plane asks whether those files changed. A new commit that leaves them alone
|
||
is not an update.
|
||
|
||
- `llmgw ctl model updates [--check]` lists what each repository has.
|
||
- `llmgw ctl model update MODEL` fetches the new files into the mirror beside
|
||
the old ones, checks them against the registry's checksums and the nodes'
|
||
llama.cpp, points the catalog at them, replaces the replicas one at a
|
||
time and asks one to say hello.
|
||
- `llmgw ctl model rollback MODEL` goes back to the files the update
|
||
replaced, which stay on the mirror.
|
||
- The console shows UPDATE on Models and offers all three on a model's row.
|
||
|
||
A model from `server.yaml` is reported but not changed: its source is edited
|
||
there.
|
||
|
||
### The monitor: the cluster drawn like btop
|
||
|
||
The console has a **Monitor** under Watch: CPU, memory, GPU busy and memory,
|
||
requests, tokens, time to first token, in flight and queued, network and
|
||
disk, as Braille charts filled and coloured by height, over 5 minutes, an
|
||
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
|
||
each card's busy, memory, heat and power, and each model's requests, as
|
||
sparklines. A node's and a model's own pages carry their charts too.
|
||
|
||
The history is read from the Prometheus in `monitoring.prometheus`, through
|
||
the admin API (`POST /api/v1/metrics/range`); without one, the charts draw
|
||
what the console has seen since it opened.
|
||
|
||
The control plane now lists the cluster's machines for Prometheus' service
|
||
discovery, at `/prometheus/targets` on its metrics port — nodes, itself and
|
||
the store's members, each labelled with its llmgw id — and `llmgw node add`
|
||
installs `prometheus-node-exporter` on the machine (`--no-node-exporter`
|
||
leaves it off). `deploy/monitoring/prometheus.yml` scrapes them as the
|
||
`nodes` job.
|
||
|
||
### Serving and updating run in the background
|
||
|
||
Serving a model or moving one to new weights from the console starts a job
|
||
on the control plane and leaves the console free at once, instead of
|
||
closing it to follow a download that can take an hour. A new **Jobs** view
|
||
shows each job's progress and steps, and the sidebar how many are running.
|
||
`llmgw ctl model serve --detach` and `llmgw ctl model update --detach` do the
|
||
same; `llmgw ctl jobs` lists the jobs and `llmgw ctl jobs ID` follows one.
|
||
|
||
### The file that suits the hardware, and how to spread it
|
||
|
||
`llmgw ctl model plan REPO` now plans the files the repository actually has
|
||
against the fleet as it is, before anything is downloaded: the model's shape
|
||
comes from one GGUF file's header, read from the registry, or from its
|
||
config.json. Every file is listed with the best way it runs here — one card;
|
||
several of a node's cards (llama.cpp splits by layer, vLLM by tensor
|
||
parallelism, dividing the KV heads); several nodes' cards over one fabric,
|
||
pipeline-parallel between them (vLLM); or a CPU node's memory — what each
|
||
card or node would hold and whether that hardware is free now. Best first:
|
||
on a GPU, the fewest cards, then the better quantisation up to eight bits;
|
||
whether the cards are free now only breaks a tie. Busy is not unfit: a model
|
||
for cards that are in use is added and starts when there is room, the job
|
||
saying so rather than failing. `--precision` keeps the old plan in the abstract.
|
||
|
||
`llmgw ctl model serve` without `--quant` takes the plan's file, and without
|
||
`--cpu` or `--gpu-class`/`--gpus`/`--nodes` the plan's way of running it —
|
||
a GGUF model on GPUs is now given a GPU profile rather than always the CPU.
|
||
`--fallbacks` writes the next best ways too, lower in priority, so placement
|
||
uses one of them when the first is busy. The console's Serve a model shows
|
||
the plan for each file, marks the best, and asks for the way and the
|
||
fallback.
|
||
|
||
### Models of several files: vision projectors, original weights
|
||
|
||
- A vision model's projector (`mmproj`) is fetched beside its GGUF weights
|
||
and loaded with them: `llmgw ctl model serve REPO --projector FILE`, or the
|
||
choice offered by the console. The model is marked as taking images. Nodes
|
||
load it once their agent is this version or later.
|
||
- `--format safetensors` serves a repository's original weights, with their
|
||
config.json, tokenizer and shard index, on vLLM across `--tp` GPUs of
|
||
`--gpu-class` (the fleet's one class when there is one). In the console,
|
||
**F** on Serve a model switches to them.
|
||
- The mirror keeps where it fetched each file beside it, and reports that
|
||
and the checksum after it restarts. The console's Mirror view shows one
|
||
row per version of a model, with its files on Enter.
|
||
- An update watches every file of a model, the projector and the tokenizer
|
||
included; one whose files the mirror cannot trace is found on the
|
||
registry by name, commit and size, or given with `llmgw ctl model origin`.
|
||
- `llmgw_model_update_available` is published, and `ModelUpdateAvailable`
|
||
alerts on it.
|
||
|
||
### The manual, in the binary
|
||
|
||
`llmgw docs` reads every document — installation, operation, the
|
||
configuration reference, the README, the changes — and a page for every
|
||
command, built from the commands' own help, like `info`: topics on the left,
|
||
T for a page's contents, / to find, n for the next match. Piped, it prints
|
||
the page. In the console it is **Help › Manual**, and **?** opens the new
|
||
page about the console itself from any screen.
|
||
|
||
## v0.11.1 — 2026-10-03
|
||
|
||
### The console's overview no longer spills over the screen
|
||
|
||
On a cluster with a GPU, `sudo llmgw` printed the overview's GPU table onto the
|
||
terminal under the console every three seconds, scrolling across the full
|
||
screen. The summary's tables were written to standard output whatever they
|
||
were given; they are now written where the summary is, which for the console
|
||
is its panel. The test cluster the console was tried on had no GPU, which is
|
||
why it was not seen there.
|
||
|
||
## v0.11.0 — 2026-10-03
|
||
|
||
A minor version: the store can be replicated and the control plane doubled,
|
||
both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
|
||
usage is read from sums kept as it is written. Released without candidates.
|
||
|
||
**Upgrading.** Migrations 0015 (usage sums, filled from the request table
|
||
already recorded — once, at the first start) and 0016 (control planes) run by
|
||
themselves. An agent older than this ignores the list of control planes and
|
||
keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
|
||
meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
|
||
|
||
The GPU and fabric work below began from the field report in issue #2, after
|
||
a deployment on GPU nodes.
|
||
|
||
### GPU metrics that move, and charts of them
|
||
|
||
A GPU's utilisation and memory in use were read once, when the agent
|
||
started, and published from then on: a card busy since looked as idle as it
|
||
was at registration, and one idle since as busy. They are now sampled with
|
||
the temperatures and power every heartbeat (ten seconds at most), with the
|
||
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
|
||
An agent too old to send them still has its registration figures published,
|
||
and a figure the driver does not give is left out rather than shown as zero.
|
||
|
||
- **New series**: `llmgw_gpu_memory_utilization_percent`,
|
||
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
|
||
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
|
||
sample.
|
||
- **The node page** shows each card's utilisation, memory, clock and ECC
|
||
errors in its table, and charts of utilisation, memory, temperature and
|
||
power, one line per card.
|
||
- **The cluster page** charts the fleet's GPUs by class: average utilisation
|
||
and memory in use.
|
||
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
|
||
utilisation, memory, memory bandwidth, temperature, power against the
|
||
limit, clock, and what holds cards back.
|
||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||
with what holds it and its figures under the pointer; free cards dashed.
|
||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||
5% busy, with when they were last asked — the first place to find room.
|
||
- **What each model gets from its cards**, on the cluster page and in
|
||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||
draw and average utilisation, over the last quarter hour.
|
||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||
log (counted from when the agent started), memory rows remapped or pages
|
||
retired and whether either waits for a reset, a card out of spare rows,
|
||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||
`GPUNVLinkErrorsClimbing`.
|
||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||
held over the window, charted and summed into GPU-hours, whether or not the
|
||
models were asked anything — the time a GPU quota is about. From the
|
||
metrics store, as far back as it keeps history.
|
||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||
each card and whose quota it counts against, for joining the GPU series to
|
||
models and tenants.
|
||
|
||
### Usage is read from sums kept as it is written
|
||
|
||
Every chart and total — the usage tab, quotas, the dashboard, a person's own
|
||
page, and budgets restored at start — summed the request table again on each
|
||
view: a row per request, scanned whole. Usage is now also summed as it is
|
||
written, in the same transaction, into five-minute slices, hours and days per
|
||
key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
|
||
0015 fills them from what is already recorded).
|
||
|
||
- **Charts** read the coarsest of the three their slices are made of: a day in
|
||
20-minute slices from the five-minute table, a week or a month from hours.
|
||
- **Totals and budgets** read the rows up to the next five-minute slice, then
|
||
slices, hours and days — exact, not rounded to a slice. On three million
|
||
requests over thirty days, a month's total per person went from 705 ms to
|
||
18 ms on PostgreSQL.
|
||
- **Latency and time to first token** are kept as histograms in the sums, so a
|
||
percentile per team or model can be read after the rows are gone.
|
||
- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
|
||
that, hourly. Totals, charts and budgets are unchanged by it — they read the
|
||
sums — but `ctl request` finds nothing older. Unset keeps every record, as
|
||
before.
|
||
|
||
### Alerts per tenant and per budget
|
||
|
||
- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
|
||
and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
|
||
tokens and time to first token per tenant. The tenant is the level
|
||
`router.tenant_label` names — `team` by default, or `org`, `project`, or
|
||
`none` — named by its path, "acme/platform". One level rather than every
|
||
project, so a cluster of thousands of projects does not become a Prometheus
|
||
of millions of series.
|
||
- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
|
||
over the budget for every organization, team, project and person given one,
|
||
refreshed every minute.
|
||
- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
|
||
`BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
|
||
`TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
|
||
`TenantSlowFirstToken`.
|
||
- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
|
||
(uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
|
||
refused by a limit, first-token p95, every budget's share spent, and who used
|
||
most in the last day — with a tenant chooser at the top.
|
||
|
||
### How each team's requests fared, not only what they cost
|
||
|
||
- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
|
||
per organization, team, project, person or model, read from the histograms
|
||
kept with the usage sums: on the usage tab's table, and in a branch's or
|
||
person's panel on the people page, beside the error rate and requests
|
||
refused by a limit.
|
||
- **Refused requests are counted against their owner.** A request a rate,
|
||
token or budget limit refused was counted only in Prometheus, by model; a
|
||
team running into its limit looked like a team that had gone quiet. It is
|
||
now recorded as a 429 against its key, person and project — no tokens, no
|
||
cost — and shown as "Limited".
|
||
- **Usage by model** is a grouping of the usage tab, by the model that
|
||
answered.
|
||
### The console watches the cluster and holds every setting
|
||
|
||
`sudo llmgw` on a control plane is a console in three groups down the left:
|
||
|
||
- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
|
||
memory, GPUs free, disk and agent version, and a node's detail — load,
|
||
engines, each card's busy, memory, temperature and power; every GPU in the
|
||
fleet and which model holds it; models, with a model's profiles, warnings
|
||
and replicas; replicas and where they run; the last day's use by team,
|
||
project or organization; and a node's agent log.
|
||
- **Settings**: every key of `server.yaml`, block by block, with what applies
|
||
when the file sets nothing and what each key is for, from its comment in
|
||
the code. Enter changes a key or opens a block; lists and maps of blocks
|
||
(endpoints, models, prices) take entries added and removed. A save is read
|
||
by the same loader the service uses first — nothing is written that would
|
||
not load — keeps the file's comments, leaves the old one beside it, and
|
||
offers to restart the service.
|
||
- **Do**: add a node, replicate the store, leave.
|
||
|
||
On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
|
||
|
||
The keys come from the configuration structs themselves, through the
|
||
generator that writes `docs/reference.md`, so a setting added to the code is on
|
||
the screen without anyone listing it, and a test fails if the two drift.
|
||
That generator told blocks apart by name alone, so `tlsconf.Config`,
|
||
`engines.Config` and `supervisor.Config` were each taken for the agent's
|
||
`Config` and dropped: the reference had no TLS, engine or supervisor settings
|
||
at all. It now has 37 blocks, six more than before.
|
||
|
||
### A second control plane with `llmgw server add`, and the screens as windows
|
||
|
||
- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
|
||
control plane, makes another machine a standby: this llmgw copied over, a
|
||
certificate from the cluster's authority, the same configuration and store
|
||
address, the machine let into the store (for a store under Patroni, one
|
||
change through Patroni's API), and the service started. Every node is told
|
||
of it within half a minute and fails over to it untouched. `--vip` keeps a
|
||
floating address on whichever control plane leads, with keepalived on both.
|
||
Tried on new machines: switching the leader off, the standby led after 14
|
||
seconds and the floating address answered from it after 23.
|
||
- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
|
||
message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
|
||
ones in their configuration.
|
||
- **A node notices a dead control plane within a quarter of a minute.** A
|
||
leader whose machine stopped dead left its nodes' streams open until the
|
||
kernel gave up retransmitting, about fifteen minutes; the agent now pings
|
||
the stream after ten quiet seconds, and bounds dialling too.
|
||
- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
|
||
open like `/ready` — what keepalived follows.
|
||
- **The store's access rules live in Patroni's shared configuration** for a
|
||
cluster set up by `server init --patroni`, so another control plane is let
|
||
in without touching each member.
|
||
- **The screens are windows**: full screen, a title bar, the steps or actions
|
||
down the left, the content in a bordered panel, the keys along the bottom,
|
||
questions and errors as dialogs over it — the setup, the console, adding a
|
||
node and one-machine init alike. Keyboard only.
|
||
|
||
### Machines nothing may log into: `llmgw join`
|
||
|
||
For sites that allow no ssh into a machine, the setup screens offer a fourth
|
||
way to reach one: a join code. The setup listens on port 7444 while it waits
|
||
and shows a code for each such machine, with the commands that fetch llmgw
|
||
onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
|
||
checks by the certificate fingerprint in the code that it reached the setup
|
||
that made it, admits itself with the code's secret, and runs what the setup
|
||
sends, printing each step, until the setup says it is done. A code is good
|
||
for one machine and an hour, and is under 90 characters so it can be copied
|
||
whole from one screen into another.
|
||
|
||
To the setup a joined machine is one more way of running a script there:
|
||
the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
|
||
machine joining by code and another logged into with a password, in one
|
||
setup.
|
||
|
||
### Setting up from nothing in screens: `sudo llmgw`
|
||
|
||
Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
|
||
setup on a machine with nothing installed, and a console on a control plane.
|
||
In a pipe or a script it prints its help, as before.
|
||
|
||
- **Setup** asks one machine or three. For three — this one as the control
|
||
plane and etcd's third vote, two more for the store under Patroni — it asks
|
||
how to reach the other two: the ssh agent, a key file, or a password, and a
|
||
sudo password where sudo asks for one. Passwords stay in memory for the run;
|
||
a sudo password grants the user passwordless sudo until the install
|
||
finishes, and the grant is removed after, failed or not.
|
||
- **Every machine is checked before anything is installed**: login, system
|
||
(Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
|
||
role, clock against this one, NTP, the ports its role needs, internet, and
|
||
whether it reaches the other machines. Failures stop it, with what to do;
|
||
warnings let it go on.
|
||
- **Then the plan, and the command line that does the same** from a script,
|
||
then the install with its progress, and the administrator's password.
|
||
- **The console** shows `ctl status`, refreshed every five seconds, and offers
|
||
to add a node or, for a store on one machine, to replicate it.
|
||
- Machines are reached with Go's ssh client rather than the system's, so a
|
||
password can be used; a host key is recorded on first sight and refused if it
|
||
changes.
|
||
|
||
Tried end to end on three new machines whose user logs in and sudoes only with
|
||
a password.
|
||
|
||
### A store that survives losing its machine
|
||
|
||
`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
|
||
under Patroni on two machines, its leader lock in a three-member etcd, and
|
||
llmgw given every member with `target_session_attrs=read-write`. Measured: a
|
||
planned switchover cost one failed lease renewal; switching off the primary's
|
||
machine promoted the standby in 24 seconds with the gateway serving
|
||
throughout. Example configurations are beside it.
|
||
|
||
**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
|
||
PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
|
||
third vote — the role and database, and the store already in use moved in.
|
||
Every step looks first; run against a cluster that is up, it changes nothing.
|
||
Tried from scratch on three new machines (two minutes, then a failover by
|
||
switching the primary off) and against the reference cluster (nothing
|
||
touched).
|
||
|
||
**Watching it**: for a store of several hosts the control plane publishes
|
||
`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
|
||
`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
|
||
showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
|
||
line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
|
||
`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
|
||
|
||
**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
|
||
failed: the flag was sent as a bool to an integer column, which SQLite takes
|
||
and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
|
||
by being copied from SQLite, which is why nobody had met it.
|
||
|
||
`llmgw server backup` reads a store address of several hosts. It took the
|
||
list for one host name and reached the database only by falling through to
|
||
the second; it now hands libpq the list, and dumps from a standby when there
|
||
is one.
|
||
|
||
### People, limits and usage are one page
|
||
|
||
Managing a team meant its members on the people page, its quota on the quotas
|
||
page and its spend on the usage page, each linking to the others. They are now
|
||
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
|
||
each at the address it had before, so links and bookmarks still land. The menu
|
||
has one entry for them, People.
|
||
|
||
Choosing an organization, team, project or person in the tree, or a person in
|
||
the account list, opens a panel beside it with everything about that one
|
||
thing: its own limits and what it is held to from above, spend against its
|
||
budget, its per-member limit, its capacity quota, and its requests, tokens
|
||
and spend over the last week, drawn. Its limits can be set or changed there,
|
||
and the answer comes back to the same panel.
|
||
|
||
The administrator's menu no longer has an Organizations entry beside it: `/org`
|
||
and `/org/<id>` send an administrator to that organization's panel here. The
|
||
page stays as it was for an organization's managers, who cannot see this one.
|
||
|
||
### The chat page says whether a GPU or a CPU answered
|
||
|
||
A model can have replicas on a card and on CPUs at once, and the two answer
|
||
at very different speeds; the chat page gave no way to tell which one a reply
|
||
came from. The picker now says where each model runs — `GPU`, `CPU` or
|
||
`GPU + CPU` — and the line under each reply names what answered it: `GPU
|
||
6000-24g` or `CPU`, beside its tokens a second.
|
||
|
||
Every answer through the gateway carries the same as `x-llmgw-hardware`:
|
||
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
|
||
none.
|
||
|
||
### A card shared with something else is judged every heartbeat
|
||
|
||
A card with another process on it when its agent started was kept out of
|
||
placement, and stayed out — "5014 MiB already allocated by another process" —
|
||
after that process let go of it, until the agent was restarted. Found on a
|
||
node shared with ComfyUI. The other way round, a tenant arriving after
|
||
registration was never seen, and a card the agent's own engine held when the
|
||
agent restarted read as someone else's.
|
||
|
||
The agent now reads every card's memory per process on each heartbeat,
|
||
leaves out what its own engines hold, and sends the rest; the control plane
|
||
keeps a card out of placement on that reading, and lets it back in when it
|
||
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
|
||
agent too old to send it keeps the reading from registration.
|
||
|
||
### A GPU placement no longer runs on the CPU unnoticed
|
||
|
||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||
|
||
- **Installing** the CUDA build asks it which devices it sees
|
||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||
own `ldd` could not say it: the backend is opened at run time.
|
||
- **After the upgrade**, engines the agent finds already running are held to
|
||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||
is stopped, and its replacement starts through the check. The installed
|
||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||
sees no GPU.
|
||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||
included). One that does not is stopped and reported failed: "placed on GPU
|
||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||
|
||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||
node with two kinds of GPU, a shard could get a different card from the one it
|
||
was given.
|
||
|
||
### A model of many files, mirrored whole for vLLM
|
||
|
||
vLLM loads a directory — shards, config.json, the tokenizer — and does not
|
||
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
|
||
shards alone, a node could be handed a version still downloading or missing a
|
||
shard, and a repository's page given as a model's source was fetched as one
|
||
file of HTML.
|
||
|
||
- **The explorer fetches a safetensors set with what the engine reads beside
|
||
it**: config.json, generation_config.json, the tokenizer files, the shard
|
||
index, the quantisation config. Never code: a `.py` would be the engine
|
||
running what the repository ships, which is to be chosen on purpose.
|
||
- **A version is offered to nodes only whole.** The mirror refuses its
|
||
manifest (409, with what is missing) while a file is still downloading, or
|
||
while a shard its `model.safetensors.index.json` names, or a part of a
|
||
split (`-00002-of-00004`), is absent. A node reports "not complete on the
|
||
mirror yet" and is tried again.
|
||
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
|
||
it and the manifest carries it; a node checks every file of a set against
|
||
it before using the set, and fetches one that differs again.
|
||
- **`llmgw mirror add` takes several URLs** into one version.
|
||
- **A Hugging Face repository page given as a source** is read as the
|
||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||
itself (the node then needs to reach Hugging Face), and refused for one
|
||
llama.cpp runs, which needs a file.
|
||
|
||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||
such a model, as before.
|
||
|
||
### Signing up with an address that has an account
|
||
|
||
On PostgreSQL, signing up with an address already registered showed
|
||
"something went wrong; try again": the store recognised a duplicate by the
|
||
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
|
||
code 23505. It now knows both, and the address is answered as on SQLite —
|
||
with the same page as a new sign-up, so the page cannot be used to find out
|
||
who has an account. That page now also says what to do if the address had
|
||
one: sign in, or ask an operator to reset the password.
|
||
|
||
An error the page does not explain says the account was not created and
|
||
gives a reference (`err-…`), logged beside the error, for an operator to find
|
||
it; the database's own words stay in the log.
|
||
|
||
### An adapter no fabric uses does not keep a node out
|
||
|
||
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
|
||
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
|
||
added for single-node inference, with no fabric given, that refused it for
|
||
RDMA nobody had asked for.
|
||
|
||
- Those are failures only for ports a declared fabric uses, and only when a
|
||
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
|
||
- Which fabric an undeclared InfiniBand port is on, and whether it has a
|
||
subnet manager LID, are notes rather than warnings; `node add` still warns
|
||
once that adapters are up with no fabric given.
|
||
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
|
||
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
|
||
written afterwards.
|
||
|
||
### A restart with no room beside it
|
||
|
||
`ctl restart` places each replacement beside the replica it replaces. With
|
||
no room for it — one GPU, every card that fits held by the replica itself —
|
||
the old replica was stopped only after the drain grace period, five minutes
|
||
of "restart replacing" with nothing saying why, which read as a restart that
|
||
never happened; scaling to zero and back was the way out. And a replacement
|
||
that fit nowhere even with the old replica gone was stopped for all the same,
|
||
leaving the model with nothing. Now:
|
||
|
||
- **The replica is the only thing in the way**, found by placing the
|
||
replacement on a copy of the inventory without it: stopped at once, the
|
||
replacement taking its place.
|
||
- **The replacement fits nowhere even then**: the restart is given up and the
|
||
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
|
||
- **Room being made** — a lower class preempted, an idle on-demand replica
|
||
stopping — is waited for, as before.
|
||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||
|
||
### Every GPU node gets engine builds it can run, by itself
|
||
|
||
Which build of an engine a node installs follows its hardware — the CPU
|
||
architecture, the driver, and the oldest GPU on it — from what each project
|
||
publishes and how it was compiled. Each engine has a list of builds, best
|
||
first; the node takes the first its hardware meets, and the first of those
|
||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||
about costs its download, and the next one is tried.
|
||
|
||
| | llama.cpp | vLLM |
|
||
|---|---|---|
|
||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||
| No NVIDIA GPU | the CPU build | none |
|
||
|
||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||
starts at 7.5.
|
||
|
||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||
column shows the version installed, and the installation's detail says why
|
||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||
vLLM is not placed on a V100 by accident.
|
||
- **The install computes on the GPU before it counts**: a small tensor
|
||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||
at once.
|
||
- **Whether an engine computes on its GPUs is read from every process
|
||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||
`CUDA_VISIBLE_DEVICES`.
|
||
|
||
### Engines per node, and nodes that do not run them all
|
||
|
||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||
and `--ignore-checks` passed over every other problem with it.
|
||
|
||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||
node joins without it and serves what it does run. `node add` checks the
|
||
vLLM the control plane installs, not the binary's own default.
|
||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||
advertised there, and not checked by doctor.
|
||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||
or labels a different version of an engine, or none — for exceptions; the
|
||
build each GPU generation needs is chosen without it (above).
|
||
- **The requirements follow the version**, as the table above sets out; the
|
||
threshold had been CUDA 13's for every version.
|
||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||
its own beside it, which was then used first.
|
||
- The node page shows each GPU's compute capability.
|
||
|
||
### Serving from the mirror says where a model will run
|
||
|
||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||
set. Now:
|
||
|
||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||
cards is chosen.
|
||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||
how many a llama.cpp replica serves at once.
|
||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||
typed.
|
||
- A line under the form says what the model will run as — engine, CPU or how
|
||
many of which GPU, memory, context pool and how it is divided — and turns
|
||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||
it.
|
||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||
anyway** is ticked.
|
||
|
||
### llama.cpp on several GPUs of one node
|
||
|
||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||
`nodes` above one, is refused.
|
||
|
||
## v0.10.6 — 2026-10-01
|
||
|
||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
Tried on the PoC cluster before release: LoRA adapters asked for by name
|
||
through the gateway, two on one model, each answering as trained and the model
|
||
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
|
||
requests on the preferred replica, 4 on the other); and a bronze request
|
||
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
|
||
live: a model served from an external provider, which needs a provider key.
|
||
|
||
### Capacity from the admin pages
|
||
|
||
What v0.10.5 made settable from the command line is now on the pages too:
|
||
|
||
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
|
||
cores and memory beside what it holds, with a form to set, change or remove
|
||
one. It goes through the same checks as `llmgw ctl quota`.
|
||
- **A model's card** on the mirror page sets its class, its owner (when there
|
||
are quotas to count against), and the memory and cores a replica reserves,
|
||
beside its replica counts. Only what changed is applied, and each change is
|
||
said.
|
||
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
|
||
the headroom API count the replicas a model could get by stopping those its
|
||
class outranks, separately from free room, and name whose they would be. A
|
||
critical model's card no longer refuses a replica that preemption would make
|
||
room for.
|
||
|
||
### LoRA adapters
|
||
|
||
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
|
||
nodes fetch them with the model's weights and llama.cpp loads them all at start,
|
||
applied to nothing (`--lora-scaled FILE:0`); a request that names an
|
||
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
|
||
so one replica serves the model and all its adapters. Adapter names share the
|
||
namespace of models, are listed in `/v1/models`, and follow key allowlists on
|
||
their own — a key allowed an adapter does not reach the model under it. Adapter
|
||
files are kept from weight cleanup like the model's own. Not yet for vLLM or
|
||
SGLang profiles, whose adapters are loaded differently.
|
||
|
||
### Finding and attaching adapters from the mirror page
|
||
|
||
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
|
||
held model's card, *find adapters on Hugging Face* — those made for the model
|
||
its repository was made from, since a GGUF repository is a quantisation and
|
||
adapters are made for the original.
|
||
- **Opening a repository** reads the first bytes of each GGUF file and marks it
|
||
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
|
||
the adapter merged in about as often as an adapter, and only the file says.
|
||
- **Held adapter files** are marked in the list, and a model's card lists its
|
||
adapters, offers the held adapter files of its architecture to attach under a
|
||
name and scale, and removes them. A file that is not an adapter, or is for
|
||
another architecture, is refused before it reaches the catalog.
|
||
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
|
||
lists and changes them from the command line, through
|
||
`POST /api/v1/models/{name}/adapters`, with the same checks.
|
||
|
||
### Models served outside the cluster
|
||
|
||
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
|
||
model, api_key_file}` and no profiles. Nothing is placed for it; the router
|
||
sends its requests to the provider, rewritten to the provider's name for the
|
||
model and with its key, read from the file on every publish so a rotated key is
|
||
picked up. Named in another model's `fallbacks`, it is used only when the
|
||
cluster has no capacity left. It is never scraped for metrics, and never reads
|
||
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
|
||
model — and since its prompts leave the cluster, the allowlist is how to decide
|
||
who may send them (docs/install.md). `ctl models` and the cluster page mark it
|
||
external.
|
||
|
||
### Pool spillover
|
||
|
||
A model served from pools of different priority — H100 before A100, say — was
|
||
routed to the preferred pool however full it was. Now a pool whose replicas are
|
||
saturated, by the same measure admission uses (queue past its healthy depth, or
|
||
KV cache past its ceiling), passes requests to the next pool that is not. When
|
||
every pool is saturated the preferred one is kept: a pool that is just as full
|
||
gains nothing.
|
||
|
||
### Displacement: a higher band is not refused while a lower one waits
|
||
|
||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||
gateway turned an interactive request away while batch requests sat queued
|
||
behind it. `admission.queue_capacity_total` now bounds everything queued —
|
||
by default half of every band's capacity together. Reached, a request is not
|
||
refused while a lower band has any queued: the newest request of the lowest
|
||
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
|
||
and returns. The newest, because the oldest have waited longest and are
|
||
nearest their TTL. Counted as `displaced` in the admission rejection metric.
|
||
|
||
### Releases can be signed
|
||
|
||
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
|
||
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
|
||
public key in `internal/release/trusted.pub`, every binary built after refuses
|
||
a release not signed with it, wherever it installs one. Off until then, and
|
||
docs/install.md says in which order to turn it on.
|
||
|
||
### Restarts borrow a lower class's room
|
||
|
||
A restart places the replacement beside the replica it replaces, and where
|
||
there is no free room it now makes some the way any placement does: by
|
||
stopping a replica its class outranks, which gets its room back once the old
|
||
replica has gone. The in-place restart — the model unserved while its
|
||
replacement loads — is left for when nothing of a lower class is running. On a
|
||
cluster where every model is standard, that is every full one: mark the models
|
||
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
|
||
|
||
## v0.10.5 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
After upgrading:
|
||
|
||
- **Declare what CPU models need.** A CPU model added from now on must say its
|
||
memory; one stored before this still loads and is flagged in `ctl models`
|
||
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
|
||
are then reserved per replica and become the engine's thread count, for
|
||
replicas started after the change: `llmgw ctl restart --model MODEL`.
|
||
- **Upgrade the mirror on its own host**, now with one command:
|
||
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
|
||
mirror older than this). `llmgw ctl upgrade` says so at the end.
|
||
|
||
### CPU cores are reserved, and memory has to be declared
|
||
|
||
The first step of resource allocation for larger clusters: what a replica
|
||
holds of a node is reserved, not assumed.
|
||
|
||
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
|
||
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
|
||
node holds as many CPU replicas as it has cores for, and two on one node no
|
||
longer spin against each other. Agents report their CPU count; a node that
|
||
does not, and a profile that declares no cores, constrain nothing.
|
||
- **Memory is required of new CPU models.** A model with no GPU is all host
|
||
memory, and one declaring none looked to placement as if it fitted
|
||
everywhere, any number of times. Adding one without `min_memory_gb` is now
|
||
refused. Models stored before this still load, and say what is missing.
|
||
- **Declared against measured.** Each model shows what one replica reserves
|
||
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
|
||
cluster page — with a warning when nothing is declared or when it uses more
|
||
than a tenth over what it reserves.
|
||
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
|
||
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
|
||
`ctl model serve` and the mirror's serve form fill in cores — half the
|
||
smallest node the model fits on — beside the memory they already estimate.
|
||
Running replicas keep their old figures until restarted.
|
||
|
||
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
|
||
rc.7 the first was briefly `llmgw ctl resources`.
|
||
|
||
### Capacity classes, and preemption
|
||
|
||
A model has a class — `critical`, `standard` or `opportunistic` — saying what
|
||
its capacity is worth against another's when there is not enough for both.
|
||
Unset, an on-demand model is opportunistic and every other model standard,
|
||
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
|
||
sets it.
|
||
|
||
When a model cannot be placed, replicas are stopped to make room for it, by
|
||
these rules:
|
||
|
||
- A lower class gives way to a higher one.
|
||
- Within a class, only replicas above a model's minimum — room it grew into
|
||
under load — and only for a model still below its own minimum.
|
||
- A critical model's minimum is never taken, nor a replica that has only just
|
||
loaded, nor one spanning several nodes.
|
||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||
the same within the class, then lower classes' minimums; the longest unused
|
||
first within each, and of equally idle ones the smallest.
|
||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||
replica whose stopping turns out not to be needed is left running, so an
|
||
idle replica tried first on the wrong node is not stopped with the one that
|
||
made the room.
|
||
|
||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||
is serving. Its model is held back for up to five minutes, or until the model
|
||
it gave way to has its replicas, so it does not take the room straight back.
|
||
`ctl models` shows each model's class and "gave room to X" while held; the
|
||
cluster page shows the class and how many replicas were lost; the metric is
|
||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||
|
||
### Capacity quotas for the tenancy tree
|
||
|
||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||
memory, and limited to at most some amount of each. What it holds is what the
|
||
replicas of the models it owns hold — and those of the teams and projects
|
||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||
one with no owner is shared and counts against nobody.
|
||
|
||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||
class elsewhere could take would not be one.
|
||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||
whatever their class — except a critical model's minimum.
|
||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||
up the tree gets no further replica, and says which limit; headroom and the
|
||
mirror card stop there too.
|
||
|
||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||
two is refused. Changes take effect within half a minute. Metrics:
|
||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||
by scope and resource; alert: CapacityAtLimit.
|
||
|
||
### Sized by what a replica was measured to serve
|
||
|
||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||
replica served while meeting its SLO, and its throughput there — now drives
|
||
decisions rather than only being printed:
|
||
|
||
- **Scaling up** when requests per replica pass what was measured, not only
|
||
when the engines' queues fill: an engine takes requests into its slots long
|
||
after latency has gone.
|
||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||
they were measured to hold, so a replica taken back does not put the rest
|
||
straight over.
|
||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||
operator still wins.
|
||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||
more replica would serve.
|
||
|
||
A model never measured behaves exactly as before. Measurements are read from
|
||
the store every half minute.
|
||
|
||
### Upgrading a machine the control plane does not reach
|
||
|
||
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
|
||
the mirror's host above all, whose service runs unprivileged and cannot replace
|
||
its own binary as an agent does. The release's checksums are checked, the old
|
||
binary kept, the unit restarted and watched until it stays up.
|
||
|
||
### Nodes report their own heat and power
|
||
|
||
The first step of facility-aware placement (docs/facility-aware-placement.md),
|
||
observe-only: nothing is decided on these yet. Every agent reads, at most every
|
||
ten seconds, and sends with its heartbeat:
|
||
|
||
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
|
||
it, power draw and the enforced limit, and what is holding the clocks down —
|
||
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
|
||
at registration; it is now current.
|
||
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
|
||
the package temperature, and the board's inlet air sensor where it has one.
|
||
- **CPU power** from RAPL's energy counters.
|
||
|
||
A sensor a machine does not have is not reported, never shown as zero. New
|
||
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
|
||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
|
||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
|
||
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
|
||
table of temperature, power against its limit, and what is throttling it.
|
||
|
||
### The mirror page in tabs
|
||
|
||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||
explorer and the fetch form, one under another, with every visit waiting on
|
||
Hugging Face for a list it might not show. It is now three tabs — Held (with
|
||
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
|
||
URL — each with its own address. Only the tab shown does its work, and only
|
||
Held reloads while something downloads. A download started from the explorer
|
||
links to Held, where its progress is; a fetch the mirror refused comes back on
|
||
the form it was sent from.
|
||
|
||
### A held model opens to where it runs, and how many
|
||
|
||
Clicking a model under Held opens a card beneath it: each replica with its
|
||
node, state, age and the CPU and memory its engine uses, and how many more
|
||
replicas the cluster has room for now, on which nodes, and why not more. The
|
||
replicas it always runs and the most it grows to under load are changed there.
|
||
Room is found by trying the scheduler itself on a copy of the inventory — the
|
||
same decisions the next passes would make, not an estimate beside them — and a
|
||
number of always-running replicas it cannot place is refused with the reason,
|
||
rather than accepted and left short. A ceiling above the room is taken, and
|
||
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
|
||
API.
|
||
|
||
Each row also says how its model runs without being opened: replicas ready of
|
||
those it should have, how many are still starting, and the nodes they are on —
|
||
in amber when short, in red when parked or when it does not fit. It is one line
|
||
per model beside its mode, and nodes drop the part of the name every node
|
||
shares (node05 for llmgw-node05, the full name on hover).
|
||
|
||
### One line per record, in every table
|
||
|
||
The same rule across the admin pages. Headers do not wrap; a record is one
|
||
line; free text is cut to its column with the whole of it on hover rather than
|
||
given a row of its own; a row across the whole table is only a note or an open
|
||
form. On the cluster page that means a node is one line (its address and logs
|
||
beside its name, not under it), a model's description is a column rather than a
|
||
second row, node names drop the prefix they all share, and the GPUs, classes,
|
||
fabrics and fallbacks columns appear only when some row has something in them —
|
||
on a CPU-only cluster they were three columns of dashes pushing the table past
|
||
the page. On People, a person in many teams shows two and a count, the rest on
|
||
hover.
|
||
|
||
## v0.10.4 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.4-rc.1 to rc.9.
|
||
|
||
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
|
||
thread count below only when they start.
|
||
|
||
### CPU replicas no longer fight over the CPUs
|
||
|
||
llama.cpp computes on every CPU of the machine unless told otherwise, and the
|
||
agent did not tell it. Two busy CPU replicas on one node then spin against each
|
||
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
|
||
0.8 each with both busy, in two rounds of the same test. The agent now starts a
|
||
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
|
||
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
|
||
node — all of them suits a node that only ever runs one CPU replica — and a
|
||
profile's own `--threads` still wins. Measured on VMs whose host is
|
||
oversubscribed; the proportions, not the numbers, are the point.
|
||
|
||
### Tracing
|
||
|
||
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
|
||
request is traced over OTLP/HTTP: a span for the request, carrying its
|
||
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
|
||
it. A caller's own `traceparent` is continued, and the engine is sent one for
|
||
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
|
||
without a decision. Off, it costs nothing.
|
||
|
||
### Restarting replicas
|
||
|
||
`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
|
||
ones. Each replacement is started beside the replica it replaces, which keeps
|
||
serving until the new one is ready; replicas of one model go one at a time.
|
||
Where no second copy fits anywhere, the old replica is stopped after the drain
|
||
grace period and the new one starts in its place. This is how running engines
|
||
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
|
||
version.
|
||
|
||
### A model that keeps failing is parked, and everybody is told
|
||
|
||
A model whose replicas failed at every start was retried forever: kev-0.5b,
|
||
whose architecture the nodes' llama.cpp did not know, was started 155 times in
|
||
eight hours. After five failures in a row a model is now parked — not placed
|
||
again until its definition changes or `llmgw ctl retry MODEL` is run — and
|
||
`ctl status`, `ctl models` and the admin API say so with the last error.
|
||
|
||
A request for a model that will not come up — parked, or fitting on no node —
|
||
is answered at once with the reason, instead of "the model is loading; retry
|
||
shortly", which a client would retry forever. Each request still makes the
|
||
control plane try placing it afresh, so a model that fits once room is made is
|
||
not refused on an old answer. "no placement for model" is logged when its
|
||
reason changes or every ten minutes, not on every pass: it was 250 identical
|
||
lines in two days for one model.
|
||
|
||
### Metrics and alerts for what goes wrong
|
||
|
||
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
|
||
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
|
||
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
|
||
`llmgw_model_replicas_ready` is now published at 0 for every model that should
|
||
be running; it was published only while a replica was ready, so
|
||
ModelHasNothingServing could never fire. New alerts: ModelParked,
|
||
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
|
||
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
|
||
|
||
`ctl status` counts a model against the replicas it should have now rather
|
||
than its minimum, so an on-demand model that was asked for and fitted nowhere
|
||
no longer reads as satisfied.
|
||
|
||
### Backups
|
||
|
||
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
|
||
secrets, and the certificate authority's key — to one archive readable by root
|
||
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
|
||
nightly. The control plane publishes the newest archive's time as
|
||
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
|
||
more than 26 hours old or there has never been one. docs/install.md now says how
|
||
to restore, including why running engines survive it: the engine-key secret
|
||
comes back with the store.
|
||
|
||
### Replicas follow load between min and max
|
||
|
||
A model's `max` was a ceiling nothing reached: every model ran exactly its
|
||
minimum however long its queue. Now, when requests are waiting at the engines —
|
||
every slot of every replica busy — at one or more per replica for a minute, one
|
||
replica is added, up to `max`; ten minutes with nothing waiting takes one of
|
||
those away, down to `min`. One step at a time, and not while the previous one
|
||
is still starting, so a spike that is over before a replica loads does not
|
||
leave several behind. Each decision is logged with the load that made it.
|
||
`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
|
||
|
||
### A model from the registry in one command
|
||
|
||
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
|
||
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
|
||
mirror fetch it and checks what arrived against the registry's published
|
||
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
|
||
header, adds it to the catalog, waits for a replica — stopping early with the
|
||
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
|
||
is shown as it happens. It runs in the control plane as a job, so a download of
|
||
hours does not depend on a terminal staying open. `--on-demand` adds it without
|
||
starting a replica.
|
||
|
||
### Upgrading from one command
|
||
|
||
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
|
||
machine and then every node in turn. Each agent is told the version over its
|
||
existing stream, fetches the release for its own platform, checks it against
|
||
the release's checksums, keeps the old binary beside the new and restarts,
|
||
while its engines keep serving. A node that does not come back on the new
|
||
version, ready, with its replicas, stops the upgrade before the next one.
|
||
Running it again resumes. Agents from before this have to be moved by hand
|
||
once.
|
||
|
||
### Following a request
|
||
|
||
Every response carries `X-Request-Id` — the client's own when it sends a plain
|
||
one, otherwise one the gateway makes — and the engine receives it too. One
|
||
access log line per request records it with the model asked for and served,
|
||
the replica, status, time queued, time to the first token, total time,
|
||
tokens, and the key, project and user. Usage records keep the id, replica,
|
||
queue time and time to first token, and `llmgw ctl request ID` reads one back:
|
||
the answer to "why was this slow?".
|
||
|
||
### Rate-limit headers
|
||
|
||
Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
|
||
`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
|
||
tenancy tree will refuse first, so SDKs and editors can slow down before they
|
||
are refused.
|
||
|
||
### Memory sized from the model
|
||
|
||
The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
|
||
weights, plus the KV cache for the context pool from its layers, KV heads and
|
||
head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
|
||
to 5.25 GB across the models on the live mirror, which the old "weights plus
|
||
half again plus two" could not see.
|
||
|
||
### Security
|
||
|
||
- **Alertmanager answered anyone on the network**, including requests to
|
||
silence alerts. The monitoring README now binds it to loopback, where
|
||
Prometheus beside it is its only client.
|
||
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
|
||
64 MiB of argon2, with no bound on how many ran at once, and the per-address
|
||
throttle grew by one entry per address tried. Hashes now run a few at a time
|
||
and the rest wait; the throttle expires old entries, and when it is still
|
||
full it refuses new addresses rather than dropping live ones — which would
|
||
have let a flood of made-up addresses reset the throttle on a real one.
|
||
- **Any key holder could grow Prometheus without bound** by asking for made-up
|
||
model names, each of which became a metric label. Requests are counted under
|
||
the catalog's name for their model, or "unknown".
|
||
- **A SQLite backup left a full copy of the store readable** by local users
|
||
while it ran. The snapshot is now made in a private directory, and
|
||
llmgw-backup.service sets `UMask=0077`.
|
||
- **An engine archive could write outside its directory through chained
|
||
links.** Archives are unpacked through `os.Root`, and a link may only point
|
||
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
|
||
follows that rule.
|
||
- **Onboarding made every node trust the cluster's root for everything.** It
|
||
was added to the system store, so a certificate the control plane signed
|
||
would pass for any site with every TLS client on the node. The agent trusts
|
||
it through its own configuration only; onboarding no longer adds it, and
|
||
takes it out of a node that an older version onboarded.
|
||
- **Any node could manage a mirror running mutual TLS**, since one root signs
|
||
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
|
||
the certificate with that common name, and the mirror warns when mutual TLS
|
||
runs without one.
|
||
- **The administrator's password stayed in `init.log`.** It is shown once, on
|
||
the screen, and the log keeps a placeholder.
|
||
- **Auth off on a public address was silent.** The server now warns at start
|
||
when the API listens beyond loopback without keys.
|
||
- **The actor recorded with a change was whatever the caller sent**, line
|
||
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
|
||
records the person who ran sudo rather than root.
|
||
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
|
||
reaches it. It is off. The head itself still has no authentication, and
|
||
SGLang still takes its key as a flag; docs/install.md says what to do about
|
||
both. The monitoring README says how to keep Prometheus's queries to the
|
||
control plane.
|
||
|
||
### Fixes
|
||
|
||
- Tests that slept a fixed time and then asserted now wait for their
|
||
condition. They are the likeliest cause of CI's intermittent failures on a
|
||
loaded runner.
|
||
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
|
||
`ctl restart` did on its first real use.
|
||
- The status each failure is answered with — 499, 502, 503 with and without
|
||
Retry-After, 504, 429 — is now tested case by case.
|
||
|
||
### Releases
|
||
|
||
Small fixes are published as release candidates of the next version —
|
||
v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
|
||
notes come from this section, and they are marked as pre-releases. Onboarding
|
||
treats a release candidate as a release and downloads it.
|
||
|
||
## v0.10.3 — 2026-09-30
|
||
|
||
Security fixes from a review of v0.10.2. Three of them were confirmed on the
|
||
live cluster before being fixed.
|
||
|
||
### Engines demand a key only the router holds
|
||
|
||
Engines listened on every address of their node with no key. A request from a
|
||
neighbouring node reached the model directly, past the gateway's
|
||
authentication, quotas and accounting. Each replica's engine is now started
|
||
with a key derived from the replica id and a secret in the store. The router,
|
||
its metrics scrape, the agent's readiness check and `ctl bench` send it.
|
||
llama.cpp and vLLM receive it through their environment, not their command
|
||
line. Engines started before this keep running without a key until their
|
||
replica is next started.
|
||
|
||
### The mirror's management API answers only who it was told to
|
||
|
||
Without `--api-allow`, the management API answered anyone who could reach it,
|
||
including every node. It now answers this machine only; the control plane is
|
||
let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
|
||
add `--api-allow <control plane address>` to its unit.** Downloads started
|
||
through the API go to http and https only, and never to loopback, private or
|
||
link-local addresses (`--allow-private-sources` to permit that). Model,
|
||
version and file names can no longer be `..` or start with a dot.
|
||
|
||
### The admin listener refuses what a browser page could send
|
||
|
||
The admin API has no login and trusts whoever reaches it. A page open in an
|
||
operator's browser, reaching it over an ssh tunnel, could post a model
|
||
definition. The listener now requires Host to be an address or localhost,
|
||
refuses cross-site requests, and takes changes only as `application/json`.
|
||
A profile's `env` may no longer set variables that make an engine run other
|
||
code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
|
||
They are refused where a model is defined and dropped again before an engine
|
||
starts.
|
||
|
||
### Fixes
|
||
|
||
- The agent installs only engine versions that look like one, whatever the
|
||
control plane sent; a version of `../x` could have installed outside
|
||
`/opt/llmgw/engines`.
|
||
|
||
## v0.10.2 — 2026-09-29
|
||
|
||
- After `server init`'s screen closes, only the administrator's password and
|
||
the next steps are printed, not the whole run again under the screen that
|
||
just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
|
||
only, and printed in full when init fails.
|
||
- The screens' spinner and text field are llmgw's own rather than from
|
||
bubbles, one dependency fewer. Every llmgw command still asks the terminal
|
||
for its colours as it starts, which bubbletea v1 does on purpose; a terminal
|
||
that never answers waits for its timeout.
|
||
|
||
## v0.10.1 — 2026-09-29
|
||
|
||
### `llmgw ctl fabric-test`: two nodes, rail by rail
|
||
|
||
`llmgw doctor` checks a node on its own, and every check can pass while two
|
||
nodes still cannot reach each other: a cable into the wrong switch, a
|
||
partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
|
||
`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
|
||
one rail at a time, through the agents — no SSH between nodes, and perftest is
|
||
installed the first time — and grades each against its link's rate: ok at 80%,
|
||
slow at 50%, fail below or when nothing got through.
|
||
|
||
Agents now say which requests they answer beyond the usual (`features`), so
|
||
one too old for the test is refused at once instead of waited for.
|
||
|
||
Written against perftest's real output and tested with it, but not yet run
|
||
over real InfiniBand or RoCE: this cluster has neither.
|
||
|
||
### The mirror on its own machine, in two commands
|
||
|
||
`llmgw mirror init --control-plane <address>` sets a machine up as the model
|
||
mirror: the user, `/srv/models`, and a service serving files to every node on
|
||
:8081 and the management API on :8082. `server init --mirror
|
||
http://<mirror>:8082` uses it instead of running one beside the control plane.
|
||
It took a unit written by hand and an edit to `server.yaml` before.
|
||
|
||
The management API can fetch and delete models; `mirror serve --api-allow`
|
||
makes it answer only the addresses given and the machine itself, and `mirror
|
||
init` gives it the control plane's. With `--no-node`, the starter model is
|
||
fetched and catalogued anyway, unpinned, and starts on the first node added.
|
||
|
||
Tried on three empty VMs: mirror, control plane and a node, and a request
|
||
answered by the starter model on the node.
|
||
|
||
### `server init` asks, and `node add` takes several nodes
|
||
|
||
- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
|
||
administrator's email, the other names nodes will reach the machine by
|
||
(showing the addresses it already found), SQLite or PostgreSQL, and whether
|
||
to run the mirror, a node and the starter model here. Then its steps fill
|
||
in as a list, downloads updating in place, and the administrator's password
|
||
stays in a box until the end — and is printed again after, with the whole
|
||
run, so it is in the terminal's history. The last screen says how to add a
|
||
node and to back up the authority's key. `--plain`, or no terminal, prints
|
||
lines as before.
|
||
- **`llmgw node add a b c`** adds each in turn and ends with which were added.
|
||
A fabric named for one node's adapters is recognised on the next by what the
|
||
adapters report, and not asked again.
|
||
|
||
Both tried on empty VMs: init from the form to a starter model answering, and
|
||
two nodes added to it in one command in 40 seconds.
|
||
|
||
## v0.10.0 — 2026-09-29
|
||
|
||
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
|
||
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
|
||
installed by the nodes themselves; people in many places at once; and a chat
|
||
page, editor setup and a model explorer for the people using it.
|
||
|
||
### Independent InfiniBand fabrics, told apart and drawn
|
||
|
||
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
|
||
with several independent fabrics, two given one name would have had a replica
|
||
spread across both — started, then hung on its first collective. Now:
|
||
|
||
- **Identity.** Each active InfiniBand port reports the master subnet manager
|
||
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
|
||
asked once a minute through one port per fabric. Ports whose managers are
|
||
listed together are one physical fabric, so a failover does not split one in
|
||
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
|
||
installs `infiniband-diags` where there are adapters and no `sminfo`.
|
||
- **Placement** divides a declared fabric by that identity and forms a replica
|
||
on one part only. A node whose adapters for one name are cabled to two
|
||
fabrics is refused; a node that cannot say is kept apart from those that can.
|
||
- **Subnet managers.** The cluster page lists each fabric's master, its
|
||
priority, its standbys and its master changes over the last day, and warns
|
||
when a fabric has no standby, when its ports disagree on the master, when a
|
||
port is up with no subnet manager (INIT), and when one name covers two
|
||
fabrics or two names one fabric.
|
||
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
|
||
each heartbeat: state, physical state, rate, error and data counters. A port
|
||
going down takes its fabric off the node within seconds, and back when it
|
||
returns — it used to wait for the agent to restart.
|
||
- **Maps.** Every five minutes one node on each fabric
|
||
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
|
||
the fabric: switches in their tiers, adapters under their leaves, cables
|
||
coloured by rate, degraded ones — trained below the rest of their kind —
|
||
marked in red, rail alignment, and the cluster's own adapters live, with
|
||
their traffic running along the cables. It zooms, pans, highlights what a
|
||
switch or adapter touches, and opens detail on a click. Placement prefers
|
||
the nodes of one leaf and puts nodes with a degraded cable last.
|
||
|
||
### NVLink, on every build and across nodes
|
||
|
||
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
|
||
which was the only source of NVLink peers: on every NVLink machine the
|
||
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
|
||
peers, each card's NUMA node, and how near each network adapter is.
|
||
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
|
||
within it, then — for a multi-node shard — nearness to the fabric's adapters,
|
||
then NUMA nodes spanned. It used to grow one card's peers, which took cards
|
||
joined to that one but not to each other.
|
||
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
|
||
found from each GPU's fabric state, ClusterUUID and CliqueId, with
|
||
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
|
||
domain first, on nodes whose every card is joined and whose IMEX is up, and
|
||
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
|
||
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
|
||
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
|
||
|
||
### Multi-node launches fixed before they meet hardware
|
||
|
||
- **vLLM launches multi-node its own way by default.** An empty
|
||
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
|
||
it now means `mp`, every node running `vllm serve` with `--nnodes`,
|
||
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
|
||
documents. The `mp` path used to give workers no rank and the leader's
|
||
address as their own. Set `launch_mode: ray` to keep Ray.
|
||
- **NCCL bootstraps over the node's own interface.** The agent's default of
|
||
`eth0` meant it never looked for the interface holding its management
|
||
address, and the control plane sent one name for every node. A node on
|
||
`ens3` or `bond0` had NCCL bind nothing.
|
||
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
|
||
address; NCCL's default is a v1 link-local one that does not route.
|
||
- A Ray worker now waits for its head before joining.
|
||
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
|
||
|
||
### `llmgw doctor`
|
||
|
||
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
|
||
wrong and how to fix it, most serious first: NVSwitch without the fabric
|
||
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
|
||
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
|
||
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
|
||
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
|
||
shard on each fabric would be given. `--json` writes every fact it read, to
|
||
send back; it exits non-zero when something fails.
|
||
|
||
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
|
||
model, where llama.cpp runs several times slower and nothing else looks wrong —
|
||
and whether vLLM can be installed on the node, and why not.
|
||
|
||
### vLLM installs itself on GPU nodes
|
||
|
||
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
|
||
that can run it, with nothing done by hand. A Python 3.12 environment is built
|
||
by uv — itself a GitHub release checked against its checksum — from PyPI,
|
||
resolved as PyPI stood a day after that vLLM was published and installed with
|
||
`--require-hashes`, so every node gets the same files whenever it is built. The
|
||
resolved list stays in the environment as `llmgw-requirements.txt`.
|
||
|
||
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
|
||
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
|
||
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
|
||
25 GB free while it installs. A node that lacks one says which on the
|
||
cluster page; a node with no GPU skips vLLM without a word.
|
||
- gcc is installed with it, since Triton compiles its kernel launcher on the
|
||
first model load.
|
||
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
|
||
version.
|
||
|
||
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
|
||
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
|
||
itself was not started there.
|
||
|
||
A node is asked only for engines its agent says it can install. An agent older
|
||
than this release says nothing, and is asked for llama.cpp alone, so a cluster
|
||
half way through its upgrade does not show every node as failing vLLM.
|
||
|
||
### `llmgw node add`: a node in one command
|
||
|
||
Adding a node was a bash script from the repository, not installed anywhere,
|
||
that needed the node's name, the control plane's address and a dozen flags.
|
||
It is now part of the binary:
|
||
|
||
```
|
||
sudo llmgw node add ubuntu@10.0.0.21
|
||
```
|
||
|
||
run on the control plane. The node id is the machine's hostname and the
|
||
address its agent dials is this machine's on the route to it, checked against
|
||
the server certificate. It installs llmgw on the node — the release for its
|
||
architecture, fetched and checked there, which works on RHEL too — runs
|
||
`llmgw doctor` there and stops on a problem, issues and installs the
|
||
certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
|
||
the node to join and install llama.cpp.
|
||
|
||
In a terminal it is a checklist that fills in, and asks what a script cannot:
|
||
which fabric each group of active adapters is on — grouped by the subnet
|
||
manager or RoCE network they report — and whether to go on past a problem.
|
||
Elsewhere it prints one line per step. Running it again on a node is safe.
|
||
|
||
`llmgw node remove` forgets a node that is gone for good, which otherwise
|
||
stays listed as lost until the control plane restarts; only a lost node, or an
|
||
unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
|
||
longer lists vLLM as skipped on every CPU node.
|
||
|
||
Tried on an empty VM, from the terminal screen and plain: joined with
|
||
llama.cpp ready in under a minute, and 17 seconds to add it again.
|
||
|
||
### People belong to many places, or to none
|
||
|
||
A person was in exactly one project, chosen at approval, and could not use the
|
||
gateway before that. Now:
|
||
|
||
- **Memberships.** A person can be a member of any number of organizations,
|
||
teams and projects, or of none. A member of a team or organization may use
|
||
every project in it. `llmgw users join/leave`, and the people page adds and
|
||
removes them from either side. Approving no longer requires a project.
|
||
- **Personal keys.** A key can belong to a person with no project. On
|
||
`/profile` a person chooses what a new key spends against: one of their
|
||
projects, or Personal. `llmgw keys create --for <person>` does the same.
|
||
`auth.personal` sets default limits for personal use, or turns it off.
|
||
- **Limits for people.** `--scope user` limits a person across every key,
|
||
project and the chat page. `--per-member` on an organization, team or
|
||
project gives each member their own share there. Both are on the quotas
|
||
page, and are enforced alongside the tree, each in its own pool.
|
||
- **The chat page** asks what to spend against when a person has more than one
|
||
choice, and checks it on every message.
|
||
- **Spend is recorded with who spent it**, so a person's budget survives a
|
||
restart, personal use and the chat page included.
|
||
|
||
Leaving a project revokes the keys the person made there, unless another
|
||
membership still reaches it. A personal key keeps no conversations or notes:
|
||
those are kept per project.
|
||
|
||
### An organization's own page
|
||
|
||
A member of an organization can be made its **manager** (`users join --org …
|
||
--role manager`, or the star on the people page). A manager gets `/org`: the
|
||
organization's usage over time by team, project or person, its teams and
|
||
projects with limits and spend, and its members — read-only, and only what was
|
||
spent in its projects. Administrators can open every organization's page.
|
||
|
||
### The chat page
|
||
|
||
- A model that is not running shows a progress bar while it loads — waiting
|
||
for a node, downloading, starting — and the message is sent once it is up,
|
||
rather than failing after 45 seconds with "the model is loading".
|
||
- A finished reply is rendered as markdown, by the server under the model
|
||
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
||
- Stop, while a reply streams; its tokens, speed and time to first token under
|
||
it; a system prompt and a temperature; and the conversation kept in the
|
||
browser across a reload — the gateway still keeps none.
|
||
|
||
### Connect an editor, and the Responses API
|
||
|
||
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
||
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
||
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
||
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
||
no secret — the key comes from the environment — keeps a copy of whatever it
|
||
replaces, and leaves the rest of the file alone.
|
||
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
||
chat completions and back — instructions, messages, function and custom
|
||
tools with their calls and outputs, usage, streamed or not — down the same
|
||
path as any request.
|
||
|
||
### Your own use
|
||
|
||
The account page (was "Your keys") shows the last thirty days of a person's own
|
||
use and, for each place they spend, a bar for every budget that applies — their
|
||
own, or a project's shared with others, said as such.
|
||
|
||
### The mirror page
|
||
|
||
- **Explore models** opens on what is trending on Hugging Face, with views for
|
||
new and most downloaded, filters for task and size, and each repository's
|
||
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
||
base model and license. "Only what these nodes can run" is on by default.
|
||
- A repository opens beneath its row with its model card, and a download
|
||
started there starts in place, its progress followed in the table above.
|
||
- What the mirror holds is grouped by model family.
|
||
- An importance matrix published beside a model's quantisations is no longer
|
||
offered as a model.
|
||
|
||
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
|
||
demand, or stopped — and it can be changed there, without the CLI. The serve
|
||
form asks the same: on demand loads the model when somebody asks for it and
|
||
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
||
mode and says to change it there, because the file is read on every start.
|
||
|
||
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
||
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
||
The architecture was in the first kilobytes of the file all along. Now:
|
||
|
||
- The mirror page's registry listing shows each file's architecture and, for
|
||
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
||
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
||
- A fetch reads the file's header first — a range request, not a download —
|
||
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
||
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
||
|
||
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
||
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
||
not refused, only not checked.
|
||
|
||
### Each node's CPU and memory, and a page per node
|
||
|
||
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
||
used and available, and each engine's cores and resident memory. The cluster
|
||
page's node table shows CPU and RAM as bars; each node has a page with its
|
||
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
||
from Prometheus.
|
||
|
||
### The admin pages
|
||
|
||
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
||
it refreshes in place instead of reloading, and its usage table names people
|
||
and projects instead of ids.
|
||
- The usage page's legend gives each line's total, not its last hour.
|
||
- The people page keeps an account on one line and its rarer actions in a menu.
|
||
- Every page fits a phone: a wide table scrolls in its own box.
|
||
- Static files are named by their content, so a deploy's new styles and
|
||
scripts reach browsers at once instead of a day later.
|
||
|
||
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
|
||
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
||
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
||
left for an administrator, Chat, Account and Sign out on the right for
|
||
everyone — with the current page marked. The chat and key pages name the
|
||
person's project by its path rather than its id.
|
||
|
||
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
|
||
its organization held it to sixty requests a minute. Each limit now shows the
|
||
value it is held to from above, muted and marked ↑, with where it comes from on
|
||
hover; a larger value set below a stricter one says what caps it, for rate and
|
||
token limits as well as budgets. Spend and budget share a column with a meter
|
||
that turns red past 90%. The tree is indented like the people page, opening a
|
||
form or saving one no longer reloads the page, and the explanations moved under
|
||
"How limits combine". Projects are placed under their team by id rather than by
|
||
the start of their path, which misplaced them when two organizations shared a
|
||
name.
|
||
|
||
### The project's website, in every gateway
|
||
|
||
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
|
||
the reverse proxy configurations, and the release's files to download. It is
|
||
served without a key like `/`, whose footer links to it, and says nothing about
|
||
the installation. Its fonts are in the binary, so it makes no request elsewhere.
|
||
The published website is rendered from the same template.
|
||
|
||
### Metrics
|
||
|
||
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
||
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
||
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
||
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
||
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
||
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
||
|
||
### Fixes
|
||
|
||
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
||
rewriting read it as the start of a string. Comments are now skipped.
|
||
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
||
- Usage by team, project or organization, and the spend on the quotas page,
|
||
left out every request made on the chat page: they were joined through the
|
||
key, and the chat page has none. They now count where a request was spent.
|
||
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
||
a test now drives every session page through the authentication middleware.
|
||
|
||
### Upgrading
|
||
|
||
Upgrade the control plane first, then the agents: an older agent is served as
|
||
before and simply reports none of the new measurements. Existing agent units
|
||
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
||
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
||
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
||
start with `mp` unless `launch_mode: ray` is set.
|
||
|
||
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
|
||
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
||
the keys table; it was run against a copy of a production PostgreSQL store
|
||
before release. `llmgw users move` is gone: use `join` and `leave`.
|
||
|
||
vLLM is installed on every GPU node that can run it, once its agent is
|
||
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
|
||
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
|
||
replaced by `llmgw node add`; it still works, for a machine holding the
|
||
authority but not this llmgw.
|
||
|
||
## v0.9.2 — 2026-09-28
|
||
|
||
### A new control plane serves a model by itself
|
||
|
||
`llmgw server init` now also runs a model mirror on the same machine, makes the
|
||
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
|
||
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
|
||
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
|
||
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
|
||
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
|
||
and every step is skipped on a rerun when it is already done.
|
||
|
||
Not yet run end to end on a real machine.
|
||
|
||
## v0.9.1 — 2026-09-28
|
||
|
||
### The mirror page is one line per model
|
||
|
||
What the mirror holds used to take two table rows per file — seven columns and
|
||
the address under them — so a sharded model filled the screen. Now each version
|
||
is one line: model and version, the file (or "4 files" for a set), total size,
|
||
and which catalog models use it. The address moved into a "copy address" button;
|
||
the checksum is in the file's tooltip. A version already in use is not offered
|
||
for serving again, and removing one warns that the catalog uses it. The header
|
||
counts versions rather than files.
|
||
|
||
## v0.9.0 — 2026-09-28
|
||
|
||
### Managing people and the tenancy tree
|
||
|
||
Organizations, teams and projects can now be renamed, moved and removed, and an
|
||
approved person can be moved to another project or removed — none of which
|
||
could be done at all before, from the web or the CLI.
|
||
|
||
- **`/admin/users`** shows the whole tree, each project opening onto its people
|
||
and keys, with approve, move, suspend, reinstate and remove for people,
|
||
revoke for keys, and add, rename, move and remove at every level. A search
|
||
over every account sits below it. Changes update the page in place (htmx,
|
||
embedded in the binary) and it still works with scripts off.
|
||
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
|
||
- **`llmgw users move`** and **`llmgw users delete`**.
|
||
- **`llmgw keys list --all`** lists every key with its project and owner.
|
||
- Every command that takes a project, team or organization accepts an id, a
|
||
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
|
||
be omitted when there is only one. Users are named by id or email.
|
||
|
||
Moving a person takes the keys they made with them; keys an operator issued stay
|
||
with the project. Only empty things are removed, and a project that ever had a
|
||
key is never removed, because usage history refers to it.
|
||
|
||
### Changes
|
||
|
||
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
|
||
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
|
||
built from the web has no project of that name. With one project it is
|
||
chosen; with several the command lists them.
|
||
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
|
||
rather than micro-units. `--budget 50000000` now means fifty million units;
|
||
divide existing scripts' figures by 1 000 000.
|
||
|
||
### Fixes
|
||
|
||
- `keys init` could silently move an existing team or project to another
|
||
organization or team by redeclaring it. It now refuses.
|
||
- An account rejected while pending and then reinstated became active with no
|
||
project, and failed at its first key. Reinstating one is refused; approving
|
||
gives it a project. Rejecting a signup now removes it.
|
||
|
||
## v0.8.8 — 2026-09-28
|
||
|
||
### Fixes
|
||
|
||
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
|
||
executable file not found". The PostgreSQL project's own packages (PGDG)
|
||
provide `postgresql-server` too, so dnf can answer that name with
|
||
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
|
||
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
|
||
reads the data directory from it, and runs `initdb` itself when the directory
|
||
is empty. A PostgreSQL that was already installed is used as it is.
|
||
|
||
### Known gaps
|
||
|
||
Not yet run on a real machine. Three arrangements are not handled: a
|
||
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
|
||
and says so), one listening on a port other than 5432, and several PGDG
|
||
versions where the highest is not the one on 5432.
|
||
|
||
## v0.8.7 — 2026-09-28
|
||
|
||
### PostgreSQL in one command
|
||
|
||
```
|
||
sudo llmgw server init --postgres
|
||
```
|
||
|
||
Installs PostgreSQL with the machine's own package manager when there is none,
|
||
creates the `llmgw` role and database with a generated password kept in
|
||
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
|
||
already running on SQLite it stops the gateway, copies the store across so
|
||
every account and key survives, points `auth.store` at PostgreSQL and starts
|
||
the gateway again; if the move fails, the gateway is started again on SQLite.
|
||
Given on the first run, there is never a SQLite store.
|
||
|
||
`server.yaml` is edited in place with its comments intact, and the previous
|
||
file is kept beside it. A PostgreSQL that init did not install is left sized as
|
||
it was, since something else may depend on it.
|
||
|
||
The PostgreSQL installation itself has not yet run on a real machine; the move,
|
||
the configuration edit and the rerun behaviour are tested.
|
||
|
||
### Fixes
|
||
|
||
- **A new control plane with no models did not start.** A configuration with
|
||
no static endpoints and no models was refused as having nothing to serve, so
|
||
under systemd the unit restarted every two seconds and nothing listened on
|
||
8080. It now runs as a control plane with an empty catalog.
|
||
|
||
## v0.8.6 — 2026-09-28
|
||
|
||
### Reverse proxy configurations
|
||
|
||
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
|
||
port, and both are attached to every release beside the binaries. They do not
|
||
buffer streamed replies, their read timeout outlasts a model's wait before its
|
||
first token, their body limit matches the router's, and they set
|
||
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
|
||
cookies `Secure` from the first and shows the second as the base URL on its
|
||
landing page.
|
||
|
||
## v0.8.5 — 2026-09-28
|
||
|
||
### A control plane in one command
|
||
|
||
```
|
||
sudo llmgw server init --admin you@example.com
|
||
```
|
||
|
||
Creates the llmgw user and its directories, the certificate authority, the
|
||
control plane's certificate for the machine's hostname and every address it
|
||
has, `server.yaml`, the default project, and an administrator whose password is
|
||
generated and printed once — then installs the systemd unit, starts it and
|
||
waits until it answers. What used to be two steps of install.md is now this.
|
||
|
||
Each step is skipped when its result exists, so running it again finishes an
|
||
install that stopped and changes nothing on one that did not. `--host` adds an
|
||
address the machine cannot see of itself (a DNS alias, a NAT or floating
|
||
address) and reissues the certificate keeping the old ones. It never overwrites
|
||
`server.yaml` or an authority, and certificates issued by an authority kept on
|
||
another machine are left as they are.
|
||
|
||
### Fixes
|
||
|
||
- **The configuration install.md showed could not start.** An empty catalog
|
||
without `control.enabled: true` is refused as having nothing to serve. The
|
||
documentation and the file `init` writes both set it.
|
||
|
||
## v0.8.4 — 2026-09-28
|
||
|
||
### Node logs on the admin page
|
||
|
||
Each node on the cluster page links to its logs — the agent's own, and the
|
||
output of every engine running there — with an option to follow them. `llmgw
|
||
ctl logs NODE` reads the same. They come over the agent's existing stream, so
|
||
no ssh, jump host or root is needed to see what a node did. The agent keeps its
|
||
last 2000 lines in memory; each engine its last few hundred.
|
||
|
||
## v0.8.3 — 2026-09-28
|
||
|
||
Found by building a node from nothing with v0.8.2.
|
||
|
||
### Fixes
|
||
|
||
- **A new node's first engine install failed on apt's lock.** cloud-init and
|
||
unattended-upgrades run apt in a machine's first minutes, and
|
||
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
|
||
package command held up by a lock now waits for up to ten minutes, and says
|
||
so; a failed install is retried after a minute rather than ten.
|
||
- **A new node reported no disk** until its first download created the weights
|
||
directory. It is created when the agent starts.
|
||
- **Onboarding took one control plane address.** It takes several, comma
|
||
separated, so a new node fails over like the rest.
|
||
|
||
## v0.8.2 — 2026-09-28
|
||
|
||
A node no longer runs whatever engine somebody once put on it. It installs its
|
||
own.
|
||
|
||
### Engines are installed by the node
|
||
|
||
`engines:` in server.yaml names the engines every node has and their versions,
|
||
defaulting to the llama.cpp build this release was checked against (b10991).
|
||
Each agent converges on it: it picks the build for its hardware — CUDA with its
|
||
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
|
||
packages that build needs with the machine's own package manager, downloads the
|
||
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
|
||
anything that would land outside its directory, and runs it. `ldd` names a
|
||
missing library or a system too old for the build ("libstdc++.so.6 is too old
|
||
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
|
||
again every ten minutes.
|
||
|
||
An engine is reported only once installed and seen to run, or when the agent's
|
||
configuration names it on purpose. A node that has installed nothing yet is not
|
||
a candidate for any model, where before it was a candidate for all of them.
|
||
`ctl nodes`, the cluster page and placement's refusals say where each
|
||
installation stands, and warn about a CPU without AVX2.
|
||
|
||
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
|
||
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
|
||
installed this way and must still be named in the agent's configuration.
|
||
|
||
### Onboarding
|
||
|
||
Onboarding installs llmgw from the CI release for the node's architecture,
|
||
checked against the release's checksums, and writes an agent configuration of
|
||
only the control plane's addresses and the TLS files.
|
||
|
||
### Fixes
|
||
|
||
- A race on the weights fingerprint, which two heartbeat loops touched during a
|
||
failover.
|
||
|
||
## v0.8.1 — 2026-09-28
|
||
|
||
Nothing here changes a configuration format: every file that worked with
|
||
v0.8.0 still works. What changes is how much of it needs writing, and how much
|
||
of running the cluster can be done from a page rather than a shell.
|
||
|
||
### A node describes itself
|
||
|
||
An agent file used to name its node, its address, the path to llama-server and
|
||
its weight directories — different on every node and kept true by hand. The
|
||
agent now finds these out: its id is the hostname, its address is the one on
|
||
the route to the control plane, llama-server is looked for on PATH and in
|
||
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
|
||
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
|
||
Memory is re-read on every heartbeat and installed engines once a minute, so a
|
||
VM given more RAM or a node given an engine becomes a candidate without an
|
||
agent restart; the control plane logs both. The agent file shrinks to the
|
||
control plane's addresses and the TLS files, the same on every node.
|
||
|
||
### Models from the mirror, without writing YAML
|
||
|
||
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
|
||
and serve a mirrored version from its row — name, source, engine and an
|
||
estimated memory requirement filled in. A model can say what it is for, and
|
||
`/v1/models` shows it. Placement no longer hands the same memory to two models
|
||
or puts an engine on a node that lacks it, and says when a replica runs an
|
||
edited profile.
|
||
|
||
### Tenancy, quotas and usage on pages
|
||
|
||
The tenancy tree, quotas and budgets are managed from the admin pages; an
|
||
organization's budget bounds its teams, a group's quota counts against the
|
||
group, and a budget that cannot be spent against says so. `/admin/usage`
|
||
charts requests, tokens and spend over time by organization, team, project or
|
||
person, and a request is priced by the model that served it.
|
||
|
||
### Watching the cluster
|
||
|
||
The cluster page has its own charts, backed by a year of Prometheus history,
|
||
with alerts that were made to fire before being trusted — including for the
|
||
disk the metrics share with the models.
|
||
|
||
### Chat
|
||
|
||
The picker offers each model once rather than once per alias, never asks for
|
||
more tokens than the model allows, and offers a model that loads on demand —
|
||
sending it a message is what loads it.
|
||
|
||
## v0.8.0 — 2026-09-27
|
||
|
||
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
||
an existing installation onto it without reissuing every key, and the two
|
||
things a second control plane needs so that running one is safe rather than
|
||
clever.
|
||
|
||
### A second control plane, and a way to fail over to it
|
||
|
||
Leadership is a lease in the store, so two control planes against one database
|
||
cannot both place work — which they otherwise would, both seeing a model
|
||
wanting replicas and none running, leaving twice the replicas each holding GPUs
|
||
the other believes are free.
|
||
|
||
A standby also refuses agent streams. Commands travel down the stream the agent
|
||
opened, so a standby holding one is a node the leader cannot command, and the
|
||
leader's placements would fail with "no agent connected" for a node that is
|
||
plainly connected. Agents therefore take a list of addresses rather than one,
|
||
trying each in turn: a standby refusing, a stopped control plane refusing the
|
||
connection and a crashed one never answering are the same thing from a node.
|
||
|
||
Measured on the cluster. Leadership moved three seconds after the leader was
|
||
stopped — it released the lease on the way out rather than letting it lapse —
|
||
and all five agents had registered with the standby six seconds later. The
|
||
standby adopted every replica rather than restarting any: six replicas, all
|
||
ready, engines that never stopped serving.
|
||
|
||
Everything is unset-means-leader, so a single control plane and every
|
||
deployment that existed before this behave exactly as they did.
|
||
|
||
What this does not do is move client traffic. The standby served inference
|
||
correctly on its own port while the public address returned 502, because
|
||
nothing in front of them moved — that needs a load balancer or a shared
|
||
address, and it is the remaining piece.
|
||
|
||
### PostgreSQL
|
||
|
||
The store runs on PostgreSQL, chosen by what the address looks like: a path is
|
||
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
|
||
address out of a file so a DSN carrying a password stays out of a configuration
|
||
that gets copied and pasted into issues. Every existing installation keeps
|
||
working untouched.
|
||
|
||
One set of queries rather than one per database, because the SQL here is almost
|
||
entirely portable and two copies would drift one query at a time. The rewrite
|
||
of placeholders is a wrapper around the handle rather than a change at sixty
|
||
call sites — which found six places reaching past it, including all three
|
||
manual transactions and two prepared statements, every one of which worked on
|
||
SQLite and would have failed on PostgreSQL.
|
||
|
||
Exactly one query is not the same sentence in both: the note search, MATCH
|
||
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
|
||
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
|
||
to keep in step and no trigger to forget. It uses the 'simple' configuration
|
||
rather than English — these are an agent's own notes, in whatever language it
|
||
chose, and stemming English rules over Turkish would quietly stop finding
|
||
things.
|
||
|
||
`llmgw keys move-store` copies an existing store rather than starting a fresh
|
||
one, because keys are stored hashed: a migration that drops them cannot
|
||
recreate them, and every person and every script would need a new one.
|
||
|
||
Testing it against a real database found three things nothing else would have.
|
||
`schema_migrations` is created in Go rather than in a migration file, so it
|
||
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
|
||
and a millisecond timestamp does not fit. The copy order followed the order the
|
||
migrations create tables in, and `users` is created after `api_keys` even
|
||
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
|
||
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
|
||
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
|
||
difference pgx cannot see, so it arrived as NULL.
|
||
|
||
The driver is pgx, which is pure Go: the whole binary is built with
|
||
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
|
||
and a cgo driver would take that away from every deployment, including the ones
|
||
staying on SQLite.
|
||
|
||
## v0.7.0 — 2026-09-25
|
||
|
||
What a cluster needs to be lived with rather than demonstrated: somewhere to
|
||
keep a conversation, somewhere to keep a note, and an honest answer about the
|
||
disk underneath both.
|
||
|
||
### Memory notes for agents, and threads without API keys
|
||
|
||
Notes an agent writes for itself and reads back in later conversations, offered
|
||
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
|
||
`forget`). A note is a key, text and tags, searched by words or tag, private to
|
||
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
|
||
tools into chat requests: an agent that is not given the server never sees
|
||
memory.
|
||
|
||
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
|
||
and the retention is required, as it is for threads.
|
||
|
||
On a gateway with no authentication, a `threads:` block (`store`,
|
||
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
|
||
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
|
||
nothing refuses it with an explanation instead. The shared pool is for one
|
||
trusted machine: anyone who can reach the port and knows an id can read it.
|
||
|
||
### Conversations that can move between models
|
||
|
||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||
llmgw replays the stored history to whichever model is serving, which is what
|
||
lets a conversation begun on a large model continue on a small one — including
|
||
when a spent budget moved it there and the caller never chose.
|
||
|
||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||
stores no message content anywhere, as it never has, and that default is worth
|
||
more than the feature. The retention is required rather than optional, because
|
||
a conversation store with no expiry is a transcript of everything a team ever
|
||
typed kept forever by omission, and it is measured from the last turn so a
|
||
conversation in progress does not expire underneath it.
|
||
|
||
History that does not fit the serving model is trimmed oldest-first and the
|
||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||
are never dropped — losing one silently changes what the model is for while it
|
||
carries on answering — and neither is the turn the caller just sent.
|
||
|
||
Threads are scoped to a project, and reading, writing or deleting one that
|
||
belongs to another answers exactly as one that never existed: two answers would
|
||
let a caller map which ids are real in projects they cannot read.
|
||
|
||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||
accumulates nothing on the hottest path in the system, and one with a thread
|
||
stops at a cap rather than growing to it.
|
||
|
||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||
operator: the person who said the thing should not have to ask someone to read
|
||
it before it can go.
|
||
|
||
### Disk, and taking back only what llmgw put there
|
||
|
||
Disk was the one resource nothing reported. `statfs` was already being called
|
||
on every model directory, but only to ask whether the filesystem was a network
|
||
one — the free space in the same struct was discarded, so the only way to find
|
||
out why a node was filling up was to ssh to it.
|
||
|
||
Nodes now report it, per filesystem rather than per configured directory: free
|
||
space belongs to the disk, and two model directories on one disk are one pool
|
||
of bytes. Shared filesystems are kept out of a node's own figure, because space
|
||
on a filer belongs to everyone mounting it and twenty nodes would otherwise
|
||
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
|
||
Prometheus, and a node that reported nothing reads as unknown everywhere — an
|
||
older agent must not look like a full disk.
|
||
|
||
A profile may declare `min_disk_gb`, and a node without the room is no longer a
|
||
candidate, with a reason `explain-placement` prints. Skipped on a node that
|
||
already holds the weights: it needs no room for them, and without that the node
|
||
best placed to run a model is the one most likely to be rejected.
|
||
|
||
`ctl weights` lists model files llmgw downloaded for models no longer in the
|
||
catalog, and `--collect` deletes them. The design is mostly about what it will
|
||
not do. Only files llmgw fetched are candidates, and that is a calculation
|
||
rather than a heuristic — the cache name is a hash of the source, so the wanted
|
||
set is computable exactly. Anything an operator put on a node is left alone
|
||
however unreferenced it looks. Each node checks every path again before
|
||
unlinking it, refusing anything outside its cache directory or without the
|
||
cache name, so a control plane that is wrong cannot talk a node into losing a
|
||
model that took a week to copy.
|
||
|
||
### Fixes
|
||
|
||
- **A capability was lost at an indirection.** The controller is handed a
|
||
forwarding dispatcher, and reaches past its interface by type assertion to
|
||
find what lies beyond scheduling. Deleting weights was implemented on the
|
||
agent service and not on the forwarder, so it was simply absent at runtime —
|
||
with the whole suite green, because the dispatchers the tests use do not
|
||
implement it either and its absence there is the expected case.
|
||
- **`ctl tool-check` blamed the engine for what the model did.** It had one
|
||
verdict for a failed attempt and told operators not to try another model.
|
||
Measured: the engine renders the tool prompt correctly, the coder models do
|
||
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
|
||
the same engine returns a populated `tool_calls` array. A failed attempt is
|
||
now two verdicts, told apart by whether the reply used the wrapper.
|
||
|
||
## v0.6.0 — 2026-09-18
|
||
|
||
Phase 4: a replica can be bigger than one machine, and the fabric it spans
|
||
decides where it is allowed to go. Around that, everything a cluster needs
|
||
before anyone other than its operator is let near it — keys, budgets, a way in
|
||
through a browser, and enough visibility to answer "why is nothing scheduling".
|
||
|
||
### Multi-node replicas
|
||
|
||
A replica is now a set of shards with ranks, a leader and a bootstrap address,
|
||
rather than one process. Two invariants are enforced rather than documented: a
|
||
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
|
||
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
|
||
several times slower, and nothing says which.
|
||
|
||
A replica missing a shard is not ready. It was previously derived from the
|
||
shards that had reported, so a two-node replica with one node up looked
|
||
servable and took traffic into a model that could not answer.
|
||
|
||
Adoption after a restart happens one agent at a time, because that is how
|
||
agents reconnect. Merging them as they arrive is what keeps a multi-node
|
||
replica whole instead of being torn into fragments that are then killed as
|
||
strays — and an adopted engine is health-checked before it is called ready,
|
||
rather than being assumed alive because its process exists.
|
||
|
||
### Fabrics are distribution domains too
|
||
|
||
An isolated fabric is not only a compute boundary: weights present in one are
|
||
not present in the other. Node weight inventories are reported and summarised
|
||
per fabric, so "the model is on the cluster" is never mistaken for "the model
|
||
is reachable from the nodes that would run it". A fabric name that matches
|
||
nothing a node actually has is refused at registration rather than becoming an
|
||
unschedulable replica hours later.
|
||
|
||
### Who may use it
|
||
|
||
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
|
||
still no cgo). Rate limits and budgets are enforced on the request path; usage
|
||
is written off it, in batches, so accounting never adds latency to inference.
|
||
Budgets survive a restart — counters are rebuilt from the store, because a
|
||
limit anyone can clear by bouncing the process is not a limit.
|
||
|
||
A spent budget is now served a cheaper model where the catalog offers one,
|
||
rather than refused. For a team trying models out, being dropped to something
|
||
smaller is a far better answer than a wall, and the response says which model
|
||
answered.
|
||
|
||
People can sign up, be approved, and issue and revoke their own keys. A key is
|
||
shown once; only its hash is kept.
|
||
|
||
### The browser surface
|
||
|
||
A landing page that says how to connect, a read-only cluster dashboard, and an
|
||
in-browser chat that never puts an API key in the browser. The dashboard shows
|
||
who is using the cluster and what they are running — per person and per model,
|
||
with requests, tokens, errors, spend and latency.
|
||
|
||
All of it is served from the client-facing port behind a session, deliberately
|
||
not from the admin port, which also serves pprof — and pprof can dump the heap
|
||
that holds every API key in plaintext.
|
||
|
||
### Models without a restart
|
||
|
||
The catalog is live. Models can be added and removed from the command line or
|
||
the API, take effect immediately, and outlive a restart; configuration still
|
||
wins over the store, and the whole catalog is validated before anything is
|
||
written.
|
||
|
||
A model is only listed once it can actually serve. It was listed as soon as it
|
||
was known, so the first thing a new user did was ask for a model that returned
|
||
503.
|
||
|
||
### Loading on demand, and letting go
|
||
|
||
A model whose scaling allows it is started when someone asks for it, and
|
||
stopped when nobody has for a while. An idle replica is evicted to make room
|
||
for one that was asked for, with guards that exist because each is a way to
|
||
lose work quietly: only on-demand models, only single-node replicas, never one
|
||
that has never been used, and never one younger than its minimum lifetime.
|
||
|
||
Unknown is not idle. A replica with no recorded use is left alone rather than
|
||
treated as idle since the epoch.
|
||
|
||
### Knowing whether it will run before it runs
|
||
|
||
- `ctl plan` works out whether a model fits the cluster and how it would be
|
||
split, in the precision the chosen engine can actually serve — recommending
|
||
a quantisation the engine cannot load would cause the failure the command
|
||
exists to prevent.
|
||
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
|
||
installed. Flags drift between releases, and the alternative is finding out
|
||
during a model load.
|
||
- Placement now knows a model's context length and memory footprint, and node
|
||
RAM, so a CPU profile is not placed on a machine that cannot hold it.
|
||
- Placement waits for the fleet to settle after a restart rather than deciding
|
||
against a half-reported cluster.
|
||
|
||
### Weights
|
||
|
||
Nodes report the model files they can see, without llmgw owning the storage:
|
||
a shared filer, an image exported over the fabric, rsync or a copy made by hand
|
||
all look the same to placement, which only cares whether the files are there.
|
||
Reporting nothing means "not asked to look", not "has nothing".
|
||
|
||
Where the simple arrangement is not enough, a node can fetch a model by URL —
|
||
single-flight, resumable, written to a temporary name and renamed atomically,
|
||
so a killed download never leaves something that looks like weights.
|
||
|
||
### Seeing the fleet
|
||
|
||
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
|
||
scrape time rather than from a background goroutine, and served on their own
|
||
endpoint rather than through the one that also exposes pprof. The same view is
|
||
in the dashboard, so nothing else needs installing to see it.
|
||
|
||
### Onboarding a node
|
||
|
||
Adding a node is one command. There are systemd units, a private CA and a
|
||
from-nothing install guide that starts with a machine that has no CUDA and no
|
||
binary.
|
||
|
||
### llama.cpp, and CPU profiles
|
||
|
||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||
can all be exercised on ordinary machines.
|
||
|
||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||
allocation to stop two of them sharing a machine's cores.
|
||
|
||
### Metric mappings are per engine
|
||
|
||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||
and a single global mapping left half of it unmeasurable.
|
||
|
||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||
violation. `Running` and `Waiting` can never be omitted.
|
||
|
||
### Mutual TLS on the agent stream
|
||
|
||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||
node id does not match its certificate's common name. Public authorities cannot
|
||
supply these: they stopped issuing the `clientAuth` usage.
|
||
|
||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||
agent turn it on.
|
||
|
||
### Fixes
|
||
|
||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||
context and then asked that context whether it was shutting down, so every
|
||
stream error read as a clean close: no message, and the backoff reset.
|
||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||
level, so a cluster could look healthy while every routing decision was made
|
||
blind. It is now a warning, once, with a matching line when it clears.
|
||
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
|
||
over about 2 KB was undercounted, so tokens-per-minute limits let through far
|
||
more than they were set to.
|
||
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
|
||
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
|
||
bfloat16 does not fit two 80 GB cards. It does.
|
||
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
|
||
already written the response header, which is the one point at which headers
|
||
stop being changeable. A client had no way to tell a fallback from the model
|
||
it asked for.
|
||
- **Fleet metrics published nothing.** They were registered after the registry
|
||
had been built, so the collector existed and was never scraped.
|
||
- **GPU discovery broke on older drivers.** A field added for utilisation was
|
||
required rather than optional, so a driver that does not report it produced no
|
||
inventory at all instead of an inventory without that number.
|
||
- **Replica ids and ports were reused** while the processes holding them were
|
||
still alive.
|
||
|
||
## v0.5.0 — 2026-09-16
|
||
|
||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||
control plane places replicas, and `ctl` makes all of it visible.
|
||
|
||
This release covers both phases because their exit criteria depended on the same
|
||
piece — the agent stream — and neither was demonstrable without it.
|
||
|
||
### ctl
|
||
|
||
```
|
||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||
```
|
||
|
||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||
the answer when a node looks idle but nothing schedules onto it.
|
||
- `explain-placement` prints every candidate considered and the reason it was
|
||
rejected. "Unschedulable" on its own says nothing about which of six
|
||
constraints bit.
|
||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||
is placed elsewhere, and the original is stopped only once the replacement is
|
||
serving. Bounded by a grace period, because draining the only node that can
|
||
host a model would otherwise never finish and leave it undrainable.
|
||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||
|
||
The management API is on the admin listener, not the client-facing one. The two
|
||
have different audiences: one serves inference to whoever holds an API key, the
|
||
other lets a caller cordon nodes.
|
||
|
||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||
not have. The failure path already ships an engine log tail with every status
|
||
report, so the case that matters most is covered; reading a healthy shard's
|
||
output on demand is not.
|
||
|
||
### Phases 2 and 3, in detail
|
||
|
||
Written as the work landed, which is why it reads as a log rather than as
|
||
notes. Everything here shipped in v0.5.0.
|
||
|
||
### Added
|
||
|
||
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
|
||
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
|
||
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
|
||
without IB is not an error — it is a single-node island.
|
||
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
|
||
would force the whole binary, server included, to be built with a C
|
||
toolchain. Health flags anything that should keep a card out of placement:
|
||
uncorrected ECC errors, and memory already held by another process.
|
||
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
|
||
always match the marketing name: an L40S is sold as 48GB and reports 44.99
|
||
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
|
||
box, because the number decides whether a model fits.
|
||
- **Engine adapters** for vLLM and SGLang: command and environment
|
||
construction, kept free of process handling so it is testable without GPUs.
|
||
vLLM supports both Ray and multiprocessing launch modes, because which one
|
||
works depends on the version.
|
||
- **NCCL environment generation**, the part plan §5.3 marks critical.
|
||
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
|
||
NCCL pick an adapter on another fabric where the peer is unreachable, and the
|
||
job hangs rather than failing. Bootstrap is pinned to the management
|
||
interface for the same reason.
|
||
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
|
||
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
|
||
compute capability 8.9 rather than silently falling back to a slower path.
|
||
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
|
||
process group and are stopped as a group, because engines fork workers that
|
||
hold GPU memory and outlive a signal sent only to the leader. Bounded log
|
||
tail, a state machine that separates Starting from Loading (a 70B model takes
|
||
minutes, and an operator needs to know that is normal), and orphan adoption
|
||
through pid files fingerprinted with the command line — pids get reused, and
|
||
adopting an unrelated process is worse than leaking the original.
|
||
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
|
||
says the weights loaded. Routing on `/health` alone sends traffic into a model
|
||
that is still loading.
|
||
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
|
||
as a table or JSON. Run it before adding a node: it is the fastest way to find
|
||
a card another process is using, or a port with no subnet manager.
|
||
|
||
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
|
||
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
|
||
a tensor-parallel group off the slow path inside a node. `Discover` falls back
|
||
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
|
||
topology. CI compiles the tagged build so it cannot rot unnoticed.
|
||
|
||
### Not verified
|
||
|
||
The NVML path compiles but has never run against a driver — there is no GPU
|
||
machine to try it on. Treat the NVLink domains it reports as unverified until
|
||
someone runs `llmgw agent inventory` on a real node.
|
||
|
||
- **Control plane wiring** (`internal/control`): Ready replicas become router
|
||
endpoints, with no static configuration involved. The router keeps live state
|
||
for endpoints that survive an update — inflight counts, breaker state, drain
|
||
rate — so a replica that was already serving does not start from cold because
|
||
another one moved. The sink is an interface, so when the router and control
|
||
plane split into separate processes this becomes a watch stream and nothing
|
||
above it changes.
|
||
|
||
### Phase 3 exit criterion, measured in `test/e2e`
|
||
|
||
A node hosting a live replica is killed while four clients stream continuously.
|
||
The control plane marks it unreachable, then lost, rebuilds the replica on a
|
||
survivor, and republishes. Across three runs: **~370 requests completed, zero
|
||
failures**, and no replica remained on the dead node.
|
||
|
||
This proves the control logic, not the transport: the agent fleet is simulated,
|
||
and the real path still needs the stream.
|
||
|
||
### A real bug the race detector caught
|
||
|
||
`startReplica` read the replica's fields after publishing it to the map, and
|
||
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
|
||
status reports, which arrive from another goroutine and mutate exactly those
|
||
fields. Under the simulated fleet the reports came back fast enough to hit the
|
||
window every time. Everything is snapshotted under the lock now and the
|
||
functions work from values.
|
||
|
||
### The agent stream
|
||
|
||
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
|
||
carrying registration, heartbeats and shard status upward, and start, stop and
|
||
drain commands downward. The agent dials out, because GPU nodes sit behind
|
||
firewalls far more often than a control plane does and an outbound connection
|
||
needs no inbound rule on every node in the fleet.
|
||
- **Registration carries the whole inventory**, so there is no separate
|
||
discovery call and a reconnecting agent refreshes what the scheduler knows.
|
||
The fabric labels travel with it — they cannot be inferred from the wire, so
|
||
losing them in transit would silently disable multi-node placement.
|
||
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
|
||
the reconciler's dispatcher: it holds the open streams, so it is the only
|
||
thing that can deliver a command to a node.
|
||
- **Agent** (`internal/agent`) validates a profile before launching anything —
|
||
an engine that refuses a profile does so after minutes of loading — then
|
||
supervises the process and reports every transition, including the engine's
|
||
output tail on failure.
|
||
- **Reconnects forever with jittered backoff.** A control plane restart must not
|
||
require touching every GPU node, and a fleet must not reconnect in lockstep
|
||
and knock the control plane over again. Engines keep serving throughout: the
|
||
router talks to them directly.
|
||
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
|
||
any model declares a profile. The static endpoint list remains for
|
||
router-only runs against sim-engines.
|
||
|
||
### Verified with real binaries
|
||
|
||
Server and agent as separate processes over a real stream: the agent registers,
|
||
a replica is placed, an engine is launched through the vLLM adapter's actual
|
||
argument construction, streaming traffic is served through the router, and
|
||
killing the agent moves the node through unreachable to lost with placement
|
||
explaining exactly why it can no longer run anything.
|
||
|
||
### Surviving a control plane restart
|
||
|
||
Not persisting replica state left one hole that had to be closed before it
|
||
mattered: restarting the control plane orphaned live engines. They kept holding
|
||
GPUs the inventory believed were free, so the next placement failed to allocate
|
||
with an error naming the new process rather than the old one.
|
||
|
||
The fix is on the agent, not in a database, and it cannot disagree with reality
|
||
the way a stale row can:
|
||
|
||
- The supervisor now writes the full shard spec, not just a pid and command
|
||
line. A pid identifies a process; the spec is what lets it be described back
|
||
to the control plane and matched to a replica.
|
||
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
|
||
waited on — it is not this process's child — so it is watched by polling, and
|
||
stopped by signalling its group directly. Its earlier output is gone, and the
|
||
log says so rather than presenting emptiness as silence.
|
||
- Every `Register`, not only the first, reports what the agent is running. A
|
||
reconnect may be a control plane restart and has to recover the same way.
|
||
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
|
||
Anything whose model or profile left the catalog is stopped: nothing will
|
||
route to it and it is holding cards.
|
||
- An engine the agent cannot identify — a corrupt state file — is killed. A
|
||
process holding GPUs that nothing can describe is worse than no process,
|
||
because placement will hand those GPUs out again.
|
||
|
||
One mechanism covers agent restarts and control plane restarts.
|
||
|
||
### A test that was passing for no reason
|
||
|
||
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
|
||
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
|
||
tracking hijacked connections. The agent never reconnected and the assertion —
|
||
on the connection map, which still held a stale entry — passed anyway. Tests now
|
||
sever connections through their own listener and count opened streams.
|
||
|
||
### Multi-node launch
|
||
|
||
Placement and environment generation had landed early, which made multi-node
|
||
look further along than it was: a multi-node replica could not come up at all.
|
||
Three things were missing, and all three came down to two assumptions that do
|
||
not hold once a replica spans nodes.
|
||
|
||
**Not every rank serves HTTP.** The agent health-checked every shard at its own
|
||
port, but only the leader answers. Workers timed out, reported Failed, and took
|
||
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
|
||
worker is judged by staying alive instead. That is weaker evidence on purpose —
|
||
the replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined.
|
||
|
||
**A shard is not always one process.** `Engine.Commands(spec)` returns a
|
||
sequence: every command but the last is a sidecar kept alive for the shard's
|
||
lifetime and torn down with it, and the last one's exit is the shard's exit.
|
||
vLLM under Ray is why — the leader runs a Ray head and then the server against
|
||
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
|
||
had been written and tested but never called, so `launch_mode: ray` would have
|
||
run `vllm serve` against a cluster that did not exist.
|
||
|
||
A command can also require the rendezvous point to be listening before the next
|
||
one starts, bounded by a timeout so a head that never appears fails the shard
|
||
rather than hanging it. Sidecars are killed with the shard: a Ray head left
|
||
behind holds the port the next replica needs.
|
||
|
||
**Start order was backwards.** Workers were started first, on the theory that
|
||
the head should wait for them. Both engines put the rendezvous point on rank 0,
|
||
so a worker started first has nothing to connect to and spends its startup
|
||
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
|
||
needs, so the leader can start immediately and waits on its own.
|
||
|
||
None of this is verified against real engines. It is tested with stub processes;
|
||
the Ray sequencing and which launch mode a given vLLM version wants can only be
|
||
confirmed on hardware.
|
||
|
||
### Still missing for phase 2
|
||
|
||
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
|
||
network.
|
||
|
||
### Phase 3: placement
|
||
|
||
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
|
||
their fabric attachments, and what is allocated where. A re-registering agent
|
||
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
|
||
without doing so either. A GPU with an ECC fault or another process on it is
|
||
not counted as free. Replicas are released whole, because they fail whole.
|
||
- **Placement** (`internal/control/placement`): picks the highest-priority
|
||
profile that fits and explains every candidate it rejected. Two constraints
|
||
are enforced structurally rather than checked afterwards — there is no code
|
||
path that can combine nodes from two fabrics, and only GPUs of the profile's
|
||
class are ever considered.
|
||
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
|
||
large models that cannot be split. Multi-node replicas get the opposite
|
||
treatment: they are the least movable thing in the cluster, so they take the
|
||
emptiest machines.
|
||
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
|
||
domain where the topology is known. Without the NVML build it falls back to
|
||
the lowest free indices, which is usually right but only by accident of how
|
||
vendors number slots.
|
||
- **Property-based invariant tests** (plan §13): 400 randomly generated
|
||
clusters, checking on every placement that a multi-node replica lives on one
|
||
fabric, that each shard receives only that fabric's adapters, and that no
|
||
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
|
||
The table-driven tests cover the cases someone thought of; this covers the
|
||
ones nobody did.
|
||
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
|
||
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
|
||
rather than at request time — a typo in a fallback should stop a deployment,
|
||
not surface during the outage when the fallback is finally needed. The
|
||
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
|
||
load instead of minutes into model loading.
|
||
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
|
||
loop. It places one replica per pass, because placement depends on what is
|
||
already allocated and starting several from one snapshot would hand them the
|
||
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
|
||
vLLM under Ray needs its workers attached before the head serves.
|
||
- **Replica failure is whole-replica failure.** One dead shard makes the others
|
||
useless — they wait on collectives that will never complete while holding
|
||
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
|
||
exponential backoff carried on the model rather than the replica it just
|
||
destroyed.
|
||
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
|
||
blip must not tear down replicas that are still serving, and rebuilding a
|
||
multi-node replica costs tens of minutes of model loading.
|
||
- **Scale-down drains before stopping**, so in-flight requests finish instead of
|
||
being cut off mid-stream.
|
||
|
||
## v0.3.0 — 2026-09-16
|
||
|
||
Flow control. A batch workload no longer starves interactive traffic.
|
||
|
||
### Added
|
||
|
||
- **Admission control** (`internal/router/admission`), inserted between quota
|
||
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||
still answers *where* it goes. Keeping them apart is the point: an engine
|
||
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||
no amount of endpoint selection rescues the request behind it.
|
||
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||
nearly full cache, or the reverse.
|
||
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||
"flow up to the healthy limit"; above 1 means never wait.
|
||
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||
one caller sending a thousand requests occupies one slot in the rotation
|
||
rather than a thousand.
|
||
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||
(504), because sending the GPU work the client has already abandoned helps
|
||
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||
`x-llmgw-queue-ms` when a request waited.
|
||
|
||
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||
|
||
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||
With admission switched off the same load pushes it to **55–60x**. Both
|
||
directions are asserted, so the test fails if admission regresses *and* if the
|
||
load generator ever stops being a real load.
|
||
|
||
### Two lags, two different fixes
|
||
|
||
Getting this right took longer than the queueing itself:
|
||
|
||
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||
count does not — but it includes *running* requests, and treating those as
|
||
queue depth makes a healthy engine look overloaded every interval. What is
|
||
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||
the metric contract, so it is learned as the high-water mark of observed
|
||
`running`.
|
||
- Between admission granting a request and the router dispatching it, the
|
||
request is invisible to every signal. A simultaneous burst therefore passed
|
||
the check en masse — exactly the batch arrival pattern this layer exists to
|
||
survive. The controller now counts its own outstanding leases against
|
||
dispatched inflight.
|
||
|
||
### Known limitations
|
||
|
||
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||
Anyone who can reach the port can promote themselves to platinum.
|
||
- The tenant key is the session id, for the same reason.
|
||
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||
|
||
## v0.2.0 — 2026-09-16
|
||
|
||
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||
engine replicas with an OpenAI-compatible API.
|
||
|
||
### Added
|
||
|
||
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||
propagates cancellation: when a client disconnects, the upstream request is
|
||
aborted so the GPU stops working. Separate deadlines for dial, response
|
||
headers, first token, inter-token idle and total duration — a wedged engine is
|
||
indistinguishable from a slow one without the idle watchdog.
|
||
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||
reached the client; once bytes are out, splicing two responses together would
|
||
be worse than the short stream. Models can name fallback models, and the
|
||
response says which one served it via `x-llmgw-served-model`.
|
||
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||
Only routing-relevant fields are parsed; the rest of the body passes through
|
||
untouched.
|
||
- **Usage accounting that does not depend on the client.** The router can ask
|
||
engines for usage the client never requested, then drop that chunk on the way
|
||
back, so token counts are recorded either way.
|
||
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||
duplicate endpoint ids and aliases that would shadow another model — both
|
||
would otherwise misroute silently at request time.
|
||
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||
without seeing what the scorers actually produced. Served on a separate admin
|
||
listener alongside health and pprof.
|
||
|
||
### Phase 1 exit criterion, measured in `test/e2e`
|
||
|
||
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||
- Every client disconnecting frees the KV blocks on every engine.
|
||
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||
|
||
### Known limitations
|
||
|
||
- No admission control, so a batch workload can still starve interactive
|
||
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||
|
||
## v0.1.0 — 2026-09-16
|
||
|
||
First tagged version. It is useful for exactly one thing: developing and testing
|
||
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
||
returns "not implemented".
|
||
|
||
### Added
|
||
|
||
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
||
that models the behaviours a router has to cope with:
|
||
|
||
- continuous batching with a bounded concurrent sequence count
|
||
- KV cache block accounting, queueing when blocks run out, including
|
||
head-of-line blocking
|
||
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
||
effect on time to first token
|
||
- per-token latency that grows with batch size
|
||
- dropping a request and freeing its blocks when the client disconnects
|
||
- `/metrics` under vLLM's real metric names
|
||
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
||
|
||
**Routing primitives** — libraries, not yet wired into a server:
|
||
|
||
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
||
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
||
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
||
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
||
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
||
detection and drain rate derived from engine counters.
|
||
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
||
endpoints missing any of the mandatory three. The metric name mapping is
|
||
overridable, because names drift between engine versions.
|
||
- `router/breaker`: cooldown and circuit breaking in one component, with
|
||
exponential backoff and half-open probes.
|
||
|
||
### Known limitations
|
||
|
||
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
||
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
||
- No admission control, so a batch workload can still starve interactive traffic
|
||
once the router exists. That is what v0.3.0 is for.
|
||
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
||
|
||
### Notes
|
||
|
||
- The project language is English throughout: code, comments, tests, commits and
|
||
the design document.
|
||
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
||
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|