The form offered no placement and named every GGUF profile cpu. It now lists the fleet's GPU classes (the freest chosen) and the CPU, a GPU count and parallel requests, names the profile from the placement, shows what the model will run as, and refuses the CPU while GPUs are free unless that is ticked. Refs #2 (item 1). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2086 lines
109 KiB
Markdown
2086 lines
109 KiB
Markdown
# Changelog
|
||
|
||
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
||
is only released once its **exit criterion** is met, not when its code lands.
|
||
|
||
| Version | Phase | Scope | State |
|
||
|---|---|---|---|
|
||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
|
||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
|
||
| v0.7.0 | — | Conversations, agent memory, disk management | released |
|
||
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
|
||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
|
||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||
|
||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||
APIs change between minors without a deprecation period.
|
||
|
||
## Unreleased
|
||
|
||
From the field report in issue #2, after a deployment on GPU nodes.
|
||
|
||
### A GPU placement no longer runs on the CPU unnoticed
|
||
|
||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||
|
||
- **Installing** the CUDA build asks it which devices it sees
|
||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||
own `ldd` could not say it: the backend is opened at run time.
|
||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||
included). One that does not is stopped and reported failed: "placed on GPU
|
||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||
|
||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||
node with two kinds of GPU, a shard could get a different card from the one it
|
||
was given.
|
||
|
||
### Serving from the mirror says where a model will run
|
||
|
||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||
set. Now:
|
||
|
||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||
cards is chosen.
|
||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||
how many a llama.cpp replica serves at once.
|
||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||
typed.
|
||
- A line under the form says what the model will run as — engine, CPU or how
|
||
many of which GPU, memory, context pool and how it is divided — and turns
|
||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||
it.
|
||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||
anyway** is ticked.
|
||
|
||
### llama.cpp on several GPUs of one node
|
||
|
||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||
`nodes` above one.
|
||
|
||
## v0.10.6 — 2026-10-01
|
||
|
||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
Tried on the PoC cluster before release: LoRA adapters asked for by name
|
||
through the gateway, two on one model, each answering as trained and the model
|
||
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
|
||
requests on the preferred replica, 4 on the other); and a bronze request
|
||
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
|
||
live: a model served from an external provider, which needs a provider key.
|
||
|
||
### Capacity from the admin pages
|
||
|
||
What v0.10.5 made settable from the command line is now on the pages too:
|
||
|
||
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
|
||
cores and memory beside what it holds, with a form to set, change or remove
|
||
one. It goes through the same checks as `llmgw ctl quota`.
|
||
- **A model's card** on the mirror page sets its class, its owner (when there
|
||
are quotas to count against), and the memory and cores a replica reserves,
|
||
beside its replica counts. Only what changed is applied, and each change is
|
||
said.
|
||
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
|
||
the headroom API count the replicas a model could get by stopping those its
|
||
class outranks, separately from free room, and name whose they would be. A
|
||
critical model's card no longer refuses a replica that preemption would make
|
||
room for.
|
||
|
||
### LoRA adapters
|
||
|
||
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
|
||
nodes fetch them with the model's weights and llama.cpp loads them all at start,
|
||
applied to nothing (`--lora-scaled FILE:0`); a request that names an
|
||
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
|
||
so one replica serves the model and all its adapters. Adapter names share the
|
||
namespace of models, are listed in `/v1/models`, and follow key allowlists on
|
||
their own — a key allowed an adapter does not reach the model under it. Adapter
|
||
files are kept from weight cleanup like the model's own. Not yet for vLLM or
|
||
SGLang profiles, whose adapters are loaded differently.
|
||
|
||
### Finding and attaching adapters from the mirror page
|
||
|
||
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
|
||
held model's card, *find adapters on Hugging Face* — those made for the model
|
||
its repository was made from, since a GGUF repository is a quantisation and
|
||
adapters are made for the original.
|
||
- **Opening a repository** reads the first bytes of each GGUF file and marks it
|
||
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
|
||
the adapter merged in about as often as an adapter, and only the file says.
|
||
- **Held adapter files** are marked in the list, and a model's card lists its
|
||
adapters, offers the held adapter files of its architecture to attach under a
|
||
name and scale, and removes them. A file that is not an adapter, or is for
|
||
another architecture, is refused before it reaches the catalog.
|
||
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
|
||
lists and changes them from the command line, through
|
||
`POST /api/v1/models/{name}/adapters`, with the same checks.
|
||
|
||
### Models served outside the cluster
|
||
|
||
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
|
||
model, api_key_file}` and no profiles. Nothing is placed for it; the router
|
||
sends its requests to the provider, rewritten to the provider's name for the
|
||
model and with its key, read from the file on every publish so a rotated key is
|
||
picked up. Named in another model's `fallbacks`, it is used only when the
|
||
cluster has no capacity left. It is never scraped for metrics, and never reads
|
||
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
|
||
model — and since its prompts leave the cluster, the allowlist is how to decide
|
||
who may send them (docs/install.md). `ctl models` and the cluster page mark it
|
||
external.
|
||
|
||
### Pool spillover
|
||
|
||
A model served from pools of different priority — H100 before A100, say — was
|
||
routed to the preferred pool however full it was. Now a pool whose replicas are
|
||
saturated, by the same measure admission uses (queue past its healthy depth, or
|
||
KV cache past its ceiling), passes requests to the next pool that is not. When
|
||
every pool is saturated the preferred one is kept: a pool that is just as full
|
||
gains nothing.
|
||
|
||
### Displacement: a higher band is not refused while a lower one waits
|
||
|
||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||
gateway turned an interactive request away while batch requests sat queued
|
||
behind it. `admission.queue_capacity_total` now bounds everything queued —
|
||
by default half of every band's capacity together. Reached, a request is not
|
||
refused while a lower band has any queued: the newest request of the lowest
|
||
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
|
||
and returns. The newest, because the oldest have waited longest and are
|
||
nearest their TTL. Counted as `displaced` in the admission rejection metric.
|
||
|
||
### Releases can be signed
|
||
|
||
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
|
||
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
|
||
public key in `internal/release/trusted.pub`, every binary built after refuses
|
||
a release not signed with it, wherever it installs one. Off until then, and
|
||
docs/install.md says in which order to turn it on.
|
||
|
||
### Restarts borrow a lower class's room
|
||
|
||
A restart places the replacement beside the replica it replaces, and where
|
||
there is no free room it now makes some the way any placement does: by
|
||
stopping a replica its class outranks, which gets its room back once the old
|
||
replica has gone. The in-place restart — the model unserved while its
|
||
replacement loads — is left for when nothing of a lower class is running. On a
|
||
cluster where every model is standard, that is every full one: mark the models
|
||
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
|
||
|
||
## v0.10.5 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
After upgrading:
|
||
|
||
- **Declare what CPU models need.** A CPU model added from now on must say its
|
||
memory; one stored before this still loads and is flagged in `ctl models`
|
||
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
|
||
are then reserved per replica and become the engine's thread count, for
|
||
replicas started after the change: `llmgw ctl restart --model MODEL`.
|
||
- **Upgrade the mirror on its own host**, now with one command:
|
||
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
|
||
mirror older than this). `llmgw ctl upgrade` says so at the end.
|
||
|
||
### CPU cores are reserved, and memory has to be declared
|
||
|
||
The first step of resource allocation for larger clusters: what a replica
|
||
holds of a node is reserved, not assumed.
|
||
|
||
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
|
||
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
|
||
node holds as many CPU replicas as it has cores for, and two on one node no
|
||
longer spin against each other. Agents report their CPU count; a node that
|
||
does not, and a profile that declares no cores, constrain nothing.
|
||
- **Memory is required of new CPU models.** A model with no GPU is all host
|
||
memory, and one declaring none looked to placement as if it fitted
|
||
everywhere, any number of times. Adding one without `min_memory_gb` is now
|
||
refused. Models stored before this still load, and say what is missing.
|
||
- **Declared against measured.** Each model shows what one replica reserves
|
||
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
|
||
cluster page — with a warning when nothing is declared or when it uses more
|
||
than a tenth over what it reserves.
|
||
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
|
||
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
|
||
`ctl model serve` and the mirror's serve form fill in cores — half the
|
||
smallest node the model fits on — beside the memory they already estimate.
|
||
Running replicas keep their old figures until restarted.
|
||
|
||
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
|
||
rc.7 the first was briefly `llmgw ctl resources`.
|
||
|
||
### Capacity classes, and preemption
|
||
|
||
A model has a class — `critical`, `standard` or `opportunistic` — saying what
|
||
its capacity is worth against another's when there is not enough for both.
|
||
Unset, an on-demand model is opportunistic and every other model standard,
|
||
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
|
||
sets it.
|
||
|
||
When a model cannot be placed, replicas are stopped to make room for it, by
|
||
these rules:
|
||
|
||
- A lower class gives way to a higher one.
|
||
- Within a class, only replicas above a model's minimum — room it grew into
|
||
under load — and only for a model still below its own minimum.
|
||
- A critical model's minimum is never taken, nor a replica that has only just
|
||
loaded, nor one spanning several nodes.
|
||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||
the same within the class, then lower classes' minimums; the longest unused
|
||
first within each, and of equally idle ones the smallest.
|
||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||
replica whose stopping turns out not to be needed is left running, so an
|
||
idle replica tried first on the wrong node is not stopped with the one that
|
||
made the room.
|
||
|
||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||
is serving. Its model is held back for up to five minutes, or until the model
|
||
it gave way to has its replicas, so it does not take the room straight back.
|
||
`ctl models` shows each model's class and "gave room to X" while held; the
|
||
cluster page shows the class and how many replicas were lost; the metric is
|
||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||
|
||
### Capacity quotas for the tenancy tree
|
||
|
||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||
memory, and limited to at most some amount of each. What it holds is what the
|
||
replicas of the models it owns hold — and those of the teams and projects
|
||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||
one with no owner is shared and counts against nobody.
|
||
|
||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||
class elsewhere could take would not be one.
|
||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||
whatever their class — except a critical model's minimum.
|
||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||
up the tree gets no further replica, and says which limit; headroom and the
|
||
mirror card stop there too.
|
||
|
||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||
two is refused. Changes take effect within half a minute. Metrics:
|
||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||
by scope and resource; alert: CapacityAtLimit.
|
||
|
||
### Sized by what a replica was measured to serve
|
||
|
||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||
replica served while meeting its SLO, and its throughput there — now drives
|
||
decisions rather than only being printed:
|
||
|
||
- **Scaling up** when requests per replica pass what was measured, not only
|
||
when the engines' queues fill: an engine takes requests into its slots long
|
||
after latency has gone.
|
||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||
they were measured to hold, so a replica taken back does not put the rest
|
||
straight over.
|
||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||
operator still wins.
|
||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||
more replica would serve.
|
||
|
||
A model never measured behaves exactly as before. Measurements are read from
|
||
the store every half minute.
|
||
|
||
### Upgrading a machine the control plane does not reach
|
||
|
||
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
|
||
the mirror's host above all, whose service runs unprivileged and cannot replace
|
||
its own binary as an agent does. The release's checksums are checked, the old
|
||
binary kept, the unit restarted and watched until it stays up.
|
||
|
||
### Nodes report their own heat and power
|
||
|
||
The first step of facility-aware placement (docs/facility-aware-placement.md),
|
||
observe-only: nothing is decided on these yet. Every agent reads, at most every
|
||
ten seconds, and sends with its heartbeat:
|
||
|
||
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
|
||
it, power draw and the enforced limit, and what is holding the clocks down —
|
||
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
|
||
at registration; it is now current.
|
||
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
|
||
the package temperature, and the board's inlet air sensor where it has one.
|
||
- **CPU power** from RAPL's energy counters.
|
||
|
||
A sensor a machine does not have is not reported, never shown as zero. New
|
||
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
|
||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
|
||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
|
||
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
|
||
table of temperature, power against its limit, and what is throttling it.
|
||
|
||
### The mirror page in tabs
|
||
|
||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||
explorer and the fetch form, one under another, with every visit waiting on
|
||
Hugging Face for a list it might not show. It is now three tabs — Held (with
|
||
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
|
||
URL — each with its own address. Only the tab shown does its work, and only
|
||
Held reloads while something downloads. A download started from the explorer
|
||
links to Held, where its progress is; a fetch the mirror refused comes back on
|
||
the form it was sent from.
|
||
|
||
### A held model opens to where it runs, and how many
|
||
|
||
Clicking a model under Held opens a card beneath it: each replica with its
|
||
node, state, age and the CPU and memory its engine uses, and how many more
|
||
replicas the cluster has room for now, on which nodes, and why not more. The
|
||
replicas it always runs and the most it grows to under load are changed there.
|
||
Room is found by trying the scheduler itself on a copy of the inventory — the
|
||
same decisions the next passes would make, not an estimate beside them — and a
|
||
number of always-running replicas it cannot place is refused with the reason,
|
||
rather than accepted and left short. A ceiling above the room is taken, and
|
||
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
|
||
API.
|
||
|
||
Each row also says how its model runs without being opened: replicas ready of
|
||
those it should have, how many are still starting, and the nodes they are on —
|
||
in amber when short, in red when parked or when it does not fit. It is one line
|
||
per model beside its mode, and nodes drop the part of the name every node
|
||
shares (node05 for llmgw-node05, the full name on hover).
|
||
|
||
### One line per record, in every table
|
||
|
||
The same rule across the admin pages. Headers do not wrap; a record is one
|
||
line; free text is cut to its column with the whole of it on hover rather than
|
||
given a row of its own; a row across the whole table is only a note or an open
|
||
form. On the cluster page that means a node is one line (its address and logs
|
||
beside its name, not under it), a model's description is a column rather than a
|
||
second row, node names drop the prefix they all share, and the GPUs, classes,
|
||
fabrics and fallbacks columns appear only when some row has something in them —
|
||
on a CPU-only cluster they were three columns of dashes pushing the table past
|
||
the page. On People, a person in many teams shows two and a count, the rest on
|
||
hover.
|
||
|
||
## v0.10.4 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.4-rc.1 to rc.9.
|
||
|
||
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
|
||
thread count below only when they start.
|
||
|
||
### CPU replicas no longer fight over the CPUs
|
||
|
||
llama.cpp computes on every CPU of the machine unless told otherwise, and the
|
||
agent did not tell it. Two busy CPU replicas on one node then spin against each
|
||
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
|
||
0.8 each with both busy, in two rounds of the same test. The agent now starts a
|
||
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
|
||
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
|
||
node — all of them suits a node that only ever runs one CPU replica — and a
|
||
profile's own `--threads` still wins. Measured on VMs whose host is
|
||
oversubscribed; the proportions, not the numbers, are the point.
|
||
|
||
### Tracing
|
||
|
||
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
|
||
request is traced over OTLP/HTTP: a span for the request, carrying its
|
||
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
|
||
it. A caller's own `traceparent` is continued, and the engine is sent one for
|
||
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
|
||
without a decision. Off, it costs nothing.
|
||
|
||
### Restarting replicas
|
||
|
||
`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
|
||
ones. Each replacement is started beside the replica it replaces, which keeps
|
||
serving until the new one is ready; replicas of one model go one at a time.
|
||
Where no second copy fits anywhere, the old replica is stopped after the drain
|
||
grace period and the new one starts in its place. This is how running engines
|
||
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
|
||
version.
|
||
|
||
### A model that keeps failing is parked, and everybody is told
|
||
|
||
A model whose replicas failed at every start was retried forever: kev-0.5b,
|
||
whose architecture the nodes' llama.cpp did not know, was started 155 times in
|
||
eight hours. After five failures in a row a model is now parked — not placed
|
||
again until its definition changes or `llmgw ctl retry MODEL` is run — and
|
||
`ctl status`, `ctl models` and the admin API say so with the last error.
|
||
|
||
A request for a model that will not come up — parked, or fitting on no node —
|
||
is answered at once with the reason, instead of "the model is loading; retry
|
||
shortly", which a client would retry forever. Each request still makes the
|
||
control plane try placing it afresh, so a model that fits once room is made is
|
||
not refused on an old answer. "no placement for model" is logged when its
|
||
reason changes or every ten minutes, not on every pass: it was 250 identical
|
||
lines in two days for one model.
|
||
|
||
### Metrics and alerts for what goes wrong
|
||
|
||
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
|
||
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
|
||
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
|
||
`llmgw_model_replicas_ready` is now published at 0 for every model that should
|
||
be running; it was published only while a replica was ready, so
|
||
ModelHasNothingServing could never fire. New alerts: ModelParked,
|
||
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
|
||
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
|
||
|
||
`ctl status` counts a model against the replicas it should have now rather
|
||
than its minimum, so an on-demand model that was asked for and fitted nowhere
|
||
no longer reads as satisfied.
|
||
|
||
### Backups
|
||
|
||
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
|
||
secrets, and the certificate authority's key — to one archive readable by root
|
||
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
|
||
nightly. The control plane publishes the newest archive's time as
|
||
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
|
||
more than 26 hours old or there has never been one. docs/install.md now says how
|
||
to restore, including why running engines survive it: the engine-key secret
|
||
comes back with the store.
|
||
|
||
### Replicas follow load between min and max
|
||
|
||
A model's `max` was a ceiling nothing reached: every model ran exactly its
|
||
minimum however long its queue. Now, when requests are waiting at the engines —
|
||
every slot of every replica busy — at one or more per replica for a minute, one
|
||
replica is added, up to `max`; ten minutes with nothing waiting takes one of
|
||
those away, down to `min`. One step at a time, and not while the previous one
|
||
is still starting, so a spike that is over before a replica loads does not
|
||
leave several behind. Each decision is logged with the load that made it.
|
||
`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
|
||
|
||
### A model from the registry in one command
|
||
|
||
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
|
||
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
|
||
mirror fetch it and checks what arrived against the registry's published
|
||
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
|
||
header, adds it to the catalog, waits for a replica — stopping early with the
|
||
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
|
||
is shown as it happens. It runs in the control plane as a job, so a download of
|
||
hours does not depend on a terminal staying open. `--on-demand` adds it without
|
||
starting a replica.
|
||
|
||
### Upgrading from one command
|
||
|
||
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
|
||
machine and then every node in turn. Each agent is told the version over its
|
||
existing stream, fetches the release for its own platform, checks it against
|
||
the release's checksums, keeps the old binary beside the new and restarts,
|
||
while its engines keep serving. A node that does not come back on the new
|
||
version, ready, with its replicas, stops the upgrade before the next one.
|
||
Running it again resumes. Agents from before this have to be moved by hand
|
||
once.
|
||
|
||
### Following a request
|
||
|
||
Every response carries `X-Request-Id` — the client's own when it sends a plain
|
||
one, otherwise one the gateway makes — and the engine receives it too. One
|
||
access log line per request records it with the model asked for and served,
|
||
the replica, status, time queued, time to the first token, total time,
|
||
tokens, and the key, project and user. Usage records keep the id, replica,
|
||
queue time and time to first token, and `llmgw ctl request ID` reads one back:
|
||
the answer to "why was this slow?".
|
||
|
||
### Rate-limit headers
|
||
|
||
Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
|
||
`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
|
||
tenancy tree will refuse first, so SDKs and editors can slow down before they
|
||
are refused.
|
||
|
||
### Memory sized from the model
|
||
|
||
The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
|
||
weights, plus the KV cache for the context pool from its layers, KV heads and
|
||
head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
|
||
to 5.25 GB across the models on the live mirror, which the old "weights plus
|
||
half again plus two" could not see.
|
||
|
||
### Security
|
||
|
||
- **Alertmanager answered anyone on the network**, including requests to
|
||
silence alerts. The monitoring README now binds it to loopback, where
|
||
Prometheus beside it is its only client.
|
||
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
|
||
64 MiB of argon2, with no bound on how many ran at once, and the per-address
|
||
throttle grew by one entry per address tried. Hashes now run a few at a time
|
||
and the rest wait; the throttle expires old entries, and when it is still
|
||
full it refuses new addresses rather than dropping live ones — which would
|
||
have let a flood of made-up addresses reset the throttle on a real one.
|
||
- **Any key holder could grow Prometheus without bound** by asking for made-up
|
||
model names, each of which became a metric label. Requests are counted under
|
||
the catalog's name for their model, or "unknown".
|
||
- **A SQLite backup left a full copy of the store readable** by local users
|
||
while it ran. The snapshot is now made in a private directory, and
|
||
llmgw-backup.service sets `UMask=0077`.
|
||
- **An engine archive could write outside its directory through chained
|
||
links.** Archives are unpacked through `os.Root`, and a link may only point
|
||
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
|
||
follows that rule.
|
||
- **Onboarding made every node trust the cluster's root for everything.** It
|
||
was added to the system store, so a certificate the control plane signed
|
||
would pass for any site with every TLS client on the node. The agent trusts
|
||
it through its own configuration only; onboarding no longer adds it, and
|
||
takes it out of a node that an older version onboarded.
|
||
- **Any node could manage a mirror running mutual TLS**, since one root signs
|
||
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
|
||
the certificate with that common name, and the mirror warns when mutual TLS
|
||
runs without one.
|
||
- **The administrator's password stayed in `init.log`.** It is shown once, on
|
||
the screen, and the log keeps a placeholder.
|
||
- **Auth off on a public address was silent.** The server now warns at start
|
||
when the API listens beyond loopback without keys.
|
||
- **The actor recorded with a change was whatever the caller sent**, line
|
||
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
|
||
records the person who ran sudo rather than root.
|
||
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
|
||
reaches it. It is off. The head itself still has no authentication, and
|
||
SGLang still takes its key as a flag; docs/install.md says what to do about
|
||
both. The monitoring README says how to keep Prometheus's queries to the
|
||
control plane.
|
||
|
||
### Fixes
|
||
|
||
- Tests that slept a fixed time and then asserted now wait for their
|
||
condition. They are the likeliest cause of CI's intermittent failures on a
|
||
loaded runner.
|
||
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
|
||
`ctl restart` did on its first real use.
|
||
- The status each failure is answered with — 499, 502, 503 with and without
|
||
Retry-After, 504, 429 — is now tested case by case.
|
||
|
||
### Releases
|
||
|
||
Small fixes are published as release candidates of the next version —
|
||
v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
|
||
notes come from this section, and they are marked as pre-releases. Onboarding
|
||
treats a release candidate as a release and downloads it.
|
||
|
||
## v0.10.3 — 2026-09-30
|
||
|
||
Security fixes from a review of v0.10.2. Three of them were confirmed on the
|
||
live cluster before being fixed.
|
||
|
||
### Engines demand a key only the router holds
|
||
|
||
Engines listened on every address of their node with no key. A request from a
|
||
neighbouring node reached the model directly, past the gateway's
|
||
authentication, quotas and accounting. Each replica's engine is now started
|
||
with a key derived from the replica id and a secret in the store. The router,
|
||
its metrics scrape, the agent's readiness check and `ctl bench` send it.
|
||
llama.cpp and vLLM receive it through their environment, not their command
|
||
line. Engines started before this keep running without a key until their
|
||
replica is next started.
|
||
|
||
### The mirror's management API answers only who it was told to
|
||
|
||
Without `--api-allow`, the management API answered anyone who could reach it,
|
||
including every node. It now answers this machine only; the control plane is
|
||
let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
|
||
add `--api-allow <control plane address>` to its unit.** Downloads started
|
||
through the API go to http and https only, and never to loopback, private or
|
||
link-local addresses (`--allow-private-sources` to permit that). Model,
|
||
version and file names can no longer be `..` or start with a dot.
|
||
|
||
### The admin listener refuses what a browser page could send
|
||
|
||
The admin API has no login and trusts whoever reaches it. A page open in an
|
||
operator's browser, reaching it over an ssh tunnel, could post a model
|
||
definition. The listener now requires Host to be an address or localhost,
|
||
refuses cross-site requests, and takes changes only as `application/json`.
|
||
A profile's `env` may no longer set variables that make an engine run other
|
||
code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
|
||
They are refused where a model is defined and dropped again before an engine
|
||
starts.
|
||
|
||
### Fixes
|
||
|
||
- The agent installs only engine versions that look like one, whatever the
|
||
control plane sent; a version of `../x` could have installed outside
|
||
`/opt/llmgw/engines`.
|
||
|
||
## v0.10.2 — 2026-09-29
|
||
|
||
- After `server init`'s screen closes, only the administrator's password and
|
||
the next steps are printed, not the whole run again under the screen that
|
||
just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
|
||
only, and printed in full when init fails.
|
||
- The screens' spinner and text field are llmgw's own rather than from
|
||
bubbles, one dependency fewer. Every llmgw command still asks the terminal
|
||
for its colours as it starts, which bubbletea v1 does on purpose; a terminal
|
||
that never answers waits for its timeout.
|
||
|
||
## v0.10.1 — 2026-09-29
|
||
|
||
### `llmgw ctl fabric-test`: two nodes, rail by rail
|
||
|
||
`llmgw doctor` checks a node on its own, and every check can pass while two
|
||
nodes still cannot reach each other: a cable into the wrong switch, a
|
||
partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
|
||
`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
|
||
one rail at a time, through the agents — no SSH between nodes, and perftest is
|
||
installed the first time — and grades each against its link's rate: ok at 80%,
|
||
slow at 50%, fail below or when nothing got through.
|
||
|
||
Agents now say which requests they answer beyond the usual (`features`), so
|
||
one too old for the test is refused at once instead of waited for.
|
||
|
||
Written against perftest's real output and tested with it, but not yet run
|
||
over real InfiniBand or RoCE: this cluster has neither.
|
||
|
||
### The mirror on its own machine, in two commands
|
||
|
||
`llmgw mirror init --control-plane <address>` sets a machine up as the model
|
||
mirror: the user, `/srv/models`, and a service serving files to every node on
|
||
:8081 and the management API on :8082. `server init --mirror
|
||
http://<mirror>:8082` uses it instead of running one beside the control plane.
|
||
It took a unit written by hand and an edit to `server.yaml` before.
|
||
|
||
The management API can fetch and delete models; `mirror serve --api-allow`
|
||
makes it answer only the addresses given and the machine itself, and `mirror
|
||
init` gives it the control plane's. With `--no-node`, the starter model is
|
||
fetched and catalogued anyway, unpinned, and starts on the first node added.
|
||
|
||
Tried on three empty VMs: mirror, control plane and a node, and a request
|
||
answered by the starter model on the node.
|
||
|
||
### `server init` asks, and `node add` takes several nodes
|
||
|
||
- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
|
||
administrator's email, the other names nodes will reach the machine by
|
||
(showing the addresses it already found), SQLite or PostgreSQL, and whether
|
||
to run the mirror, a node and the starter model here. Then its steps fill
|
||
in as a list, downloads updating in place, and the administrator's password
|
||
stays in a box until the end — and is printed again after, with the whole
|
||
run, so it is in the terminal's history. The last screen says how to add a
|
||
node and to back up the authority's key. `--plain`, or no terminal, prints
|
||
lines as before.
|
||
- **`llmgw node add a b c`** adds each in turn and ends with which were added.
|
||
A fabric named for one node's adapters is recognised on the next by what the
|
||
adapters report, and not asked again.
|
||
|
||
Both tried on empty VMs: init from the form to a starter model answering, and
|
||
two nodes added to it in one command in 40 seconds.
|
||
|
||
## v0.10.0 — 2026-09-29
|
||
|
||
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
|
||
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
|
||
installed by the nodes themselves; people in many places at once; and a chat
|
||
page, editor setup and a model explorer for the people using it.
|
||
|
||
### Independent InfiniBand fabrics, told apart and drawn
|
||
|
||
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
|
||
with several independent fabrics, two given one name would have had a replica
|
||
spread across both — started, then hung on its first collective. Now:
|
||
|
||
- **Identity.** Each active InfiniBand port reports the master subnet manager
|
||
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
|
||
asked once a minute through one port per fabric. Ports whose managers are
|
||
listed together are one physical fabric, so a failover does not split one in
|
||
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
|
||
installs `infiniband-diags` where there are adapters and no `sminfo`.
|
||
- **Placement** divides a declared fabric by that identity and forms a replica
|
||
on one part only. A node whose adapters for one name are cabled to two
|
||
fabrics is refused; a node that cannot say is kept apart from those that can.
|
||
- **Subnet managers.** The cluster page lists each fabric's master, its
|
||
priority, its standbys and its master changes over the last day, and warns
|
||
when a fabric has no standby, when its ports disagree on the master, when a
|
||
port is up with no subnet manager (INIT), and when one name covers two
|
||
fabrics or two names one fabric.
|
||
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
|
||
each heartbeat: state, physical state, rate, error and data counters. A port
|
||
going down takes its fabric off the node within seconds, and back when it
|
||
returns — it used to wait for the agent to restart.
|
||
- **Maps.** Every five minutes one node on each fabric
|
||
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
|
||
the fabric: switches in their tiers, adapters under their leaves, cables
|
||
coloured by rate, degraded ones — trained below the rest of their kind —
|
||
marked in red, rail alignment, and the cluster's own adapters live, with
|
||
their traffic running along the cables. It zooms, pans, highlights what a
|
||
switch or adapter touches, and opens detail on a click. Placement prefers
|
||
the nodes of one leaf and puts nodes with a degraded cable last.
|
||
|
||
### NVLink, on every build and across nodes
|
||
|
||
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
|
||
which was the only source of NVLink peers: on every NVLink machine the
|
||
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
|
||
peers, each card's NUMA node, and how near each network adapter is.
|
||
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
|
||
within it, then — for a multi-node shard — nearness to the fabric's adapters,
|
||
then NUMA nodes spanned. It used to grow one card's peers, which took cards
|
||
joined to that one but not to each other.
|
||
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
|
||
found from each GPU's fabric state, ClusterUUID and CliqueId, with
|
||
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
|
||
domain first, on nodes whose every card is joined and whose IMEX is up, and
|
||
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
|
||
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
|
||
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
|
||
|
||
### Multi-node launches fixed before they meet hardware
|
||
|
||
- **vLLM launches multi-node its own way by default.** An empty
|
||
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
|
||
it now means `mp`, every node running `vllm serve` with `--nnodes`,
|
||
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
|
||
documents. The `mp` path used to give workers no rank and the leader's
|
||
address as their own. Set `launch_mode: ray` to keep Ray.
|
||
- **NCCL bootstraps over the node's own interface.** The agent's default of
|
||
`eth0` meant it never looked for the interface holding its management
|
||
address, and the control plane sent one name for every node. A node on
|
||
`ens3` or `bond0` had NCCL bind nothing.
|
||
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
|
||
address; NCCL's default is a v1 link-local one that does not route.
|
||
- A Ray worker now waits for its head before joining.
|
||
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
|
||
|
||
### `llmgw doctor`
|
||
|
||
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
|
||
wrong and how to fix it, most serious first: NVSwitch without the fabric
|
||
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
|
||
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
|
||
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
|
||
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
|
||
shard on each fabric would be given. `--json` writes every fact it read, to
|
||
send back; it exits non-zero when something fails.
|
||
|
||
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
|
||
model, where llama.cpp runs several times slower and nothing else looks wrong —
|
||
and whether vLLM can be installed on the node, and why not.
|
||
|
||
### vLLM installs itself on GPU nodes
|
||
|
||
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
|
||
that can run it, with nothing done by hand. A Python 3.12 environment is built
|
||
by uv — itself a GitHub release checked against its checksum — from PyPI,
|
||
resolved as PyPI stood a day after that vLLM was published and installed with
|
||
`--require-hashes`, so every node gets the same files whenever it is built. The
|
||
resolved list stays in the environment as `llmgw-requirements.txt`.
|
||
|
||
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
|
||
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
|
||
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
|
||
25 GB free while it installs. A node that lacks one says which on the
|
||
cluster page; a node with no GPU skips vLLM without a word.
|
||
- gcc is installed with it, since Triton compiles its kernel launcher on the
|
||
first model load.
|
||
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
|
||
version.
|
||
|
||
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
|
||
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
|
||
itself was not started there.
|
||
|
||
A node is asked only for engines its agent says it can install. An agent older
|
||
than this release says nothing, and is asked for llama.cpp alone, so a cluster
|
||
half way through its upgrade does not show every node as failing vLLM.
|
||
|
||
### `llmgw node add`: a node in one command
|
||
|
||
Adding a node was a bash script from the repository, not installed anywhere,
|
||
that needed the node's name, the control plane's address and a dozen flags.
|
||
It is now part of the binary:
|
||
|
||
```
|
||
sudo llmgw node add ubuntu@10.0.0.21
|
||
```
|
||
|
||
run on the control plane. The node id is the machine's hostname and the
|
||
address its agent dials is this machine's on the route to it, checked against
|
||
the server certificate. It installs llmgw on the node — the release for its
|
||
architecture, fetched and checked there, which works on RHEL too — runs
|
||
`llmgw doctor` there and stops on a problem, issues and installs the
|
||
certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
|
||
the node to join and install llama.cpp.
|
||
|
||
In a terminal it is a checklist that fills in, and asks what a script cannot:
|
||
which fabric each group of active adapters is on — grouped by the subnet
|
||
manager or RoCE network they report — and whether to go on past a problem.
|
||
Elsewhere it prints one line per step. Running it again on a node is safe.
|
||
|
||
`llmgw node remove` forgets a node that is gone for good, which otherwise
|
||
stays listed as lost until the control plane restarts; only a lost node, or an
|
||
unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
|
||
longer lists vLLM as skipped on every CPU node.
|
||
|
||
Tried on an empty VM, from the terminal screen and plain: joined with
|
||
llama.cpp ready in under a minute, and 17 seconds to add it again.
|
||
|
||
### People belong to many places, or to none
|
||
|
||
A person was in exactly one project, chosen at approval, and could not use the
|
||
gateway before that. Now:
|
||
|
||
- **Memberships.** A person can be a member of any number of organizations,
|
||
teams and projects, or of none. A member of a team or organization may use
|
||
every project in it. `llmgw users join/leave`, and the people page adds and
|
||
removes them from either side. Approving no longer requires a project.
|
||
- **Personal keys.** A key can belong to a person with no project. On
|
||
`/profile` a person chooses what a new key spends against: one of their
|
||
projects, or Personal. `llmgw keys create --for <person>` does the same.
|
||
`auth.personal` sets default limits for personal use, or turns it off.
|
||
- **Limits for people.** `--scope user` limits a person across every key,
|
||
project and the chat page. `--per-member` on an organization, team or
|
||
project gives each member their own share there. Both are on the quotas
|
||
page, and are enforced alongside the tree, each in its own pool.
|
||
- **The chat page** asks what to spend against when a person has more than one
|
||
choice, and checks it on every message.
|
||
- **Spend is recorded with who spent it**, so a person's budget survives a
|
||
restart, personal use and the chat page included.
|
||
|
||
Leaving a project revokes the keys the person made there, unless another
|
||
membership still reaches it. A personal key keeps no conversations or notes:
|
||
those are kept per project.
|
||
|
||
### An organization's own page
|
||
|
||
A member of an organization can be made its **manager** (`users join --org …
|
||
--role manager`, or the star on the people page). A manager gets `/org`: the
|
||
organization's usage over time by team, project or person, its teams and
|
||
projects with limits and spend, and its members — read-only, and only what was
|
||
spent in its projects. Administrators can open every organization's page.
|
||
|
||
### The chat page
|
||
|
||
- A model that is not running shows a progress bar while it loads — waiting
|
||
for a node, downloading, starting — and the message is sent once it is up,
|
||
rather than failing after 45 seconds with "the model is loading".
|
||
- A finished reply is rendered as markdown, by the server under the model
|
||
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
||
- Stop, while a reply streams; its tokens, speed and time to first token under
|
||
it; a system prompt and a temperature; and the conversation kept in the
|
||
browser across a reload — the gateway still keeps none.
|
||
|
||
### Connect an editor, and the Responses API
|
||
|
||
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
||
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
||
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
||
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
||
no secret — the key comes from the environment — keeps a copy of whatever it
|
||
replaces, and leaves the rest of the file alone.
|
||
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
||
chat completions and back — instructions, messages, function and custom
|
||
tools with their calls and outputs, usage, streamed or not — down the same
|
||
path as any request.
|
||
|
||
### Your own use
|
||
|
||
The account page (was "Your keys") shows the last thirty days of a person's own
|
||
use and, for each place they spend, a bar for every budget that applies — their
|
||
own, or a project's shared with others, said as such.
|
||
|
||
### The mirror page
|
||
|
||
- **Explore models** opens on what is trending on Hugging Face, with views for
|
||
new and most downloaded, filters for task and size, and each repository's
|
||
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
||
base model and license. "Only what these nodes can run" is on by default.
|
||
- A repository opens beneath its row with its model card, and a download
|
||
started there starts in place, its progress followed in the table above.
|
||
- What the mirror holds is grouped by model family.
|
||
- An importance matrix published beside a model's quantisations is no longer
|
||
offered as a model.
|
||
|
||
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
|
||
demand, or stopped — and it can be changed there, without the CLI. The serve
|
||
form asks the same: on demand loads the model when somebody asks for it and
|
||
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
||
mode and says to change it there, because the file is read on every start.
|
||
|
||
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
||
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
||
The architecture was in the first kilobytes of the file all along. Now:
|
||
|
||
- The mirror page's registry listing shows each file's architecture and, for
|
||
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
||
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
||
- A fetch reads the file's header first — a range request, not a download —
|
||
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
||
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
||
|
||
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
||
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
||
not refused, only not checked.
|
||
|
||
### Each node's CPU and memory, and a page per node
|
||
|
||
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
||
used and available, and each engine's cores and resident memory. The cluster
|
||
page's node table shows CPU and RAM as bars; each node has a page with its
|
||
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
||
from Prometheus.
|
||
|
||
### The admin pages
|
||
|
||
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
||
it refreshes in place instead of reloading, and its usage table names people
|
||
and projects instead of ids.
|
||
- The usage page's legend gives each line's total, not its last hour.
|
||
- The people page keeps an account on one line and its rarer actions in a menu.
|
||
- Every page fits a phone: a wide table scrolls in its own box.
|
||
- Static files are named by their content, so a deploy's new styles and
|
||
scripts reach browsers at once instead of a day later.
|
||
|
||
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
|
||
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
||
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
||
left for an administrator, Chat, Account and Sign out on the right for
|
||
everyone — with the current page marked. The chat and key pages name the
|
||
person's project by its path rather than its id.
|
||
|
||
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
|
||
its organization held it to sixty requests a minute. Each limit now shows the
|
||
value it is held to from above, muted and marked ↑, with where it comes from on
|
||
hover; a larger value set below a stricter one says what caps it, for rate and
|
||
token limits as well as budgets. Spend and budget share a column with a meter
|
||
that turns red past 90%. The tree is indented like the people page, opening a
|
||
form or saving one no longer reloads the page, and the explanations moved under
|
||
"How limits combine". Projects are placed under their team by id rather than by
|
||
the start of their path, which misplaced them when two organizations shared a
|
||
name.
|
||
|
||
### The project's website, in every gateway
|
||
|
||
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
|
||
the reverse proxy configurations, and the release's files to download. It is
|
||
served without a key like `/`, whose footer links to it, and says nothing about
|
||
the installation. Its fonts are in the binary, so it makes no request elsewhere.
|
||
The published website is rendered from the same template.
|
||
|
||
### Metrics
|
||
|
||
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
||
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
||
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
||
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
||
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
||
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
||
|
||
### Fixes
|
||
|
||
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
||
rewriting read it as the start of a string. Comments are now skipped.
|
||
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
||
- Usage by team, project or organization, and the spend on the quotas page,
|
||
left out every request made on the chat page: they were joined through the
|
||
key, and the chat page has none. They now count where a request was spent.
|
||
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
||
a test now drives every session page through the authentication middleware.
|
||
|
||
### Upgrading
|
||
|
||
Upgrade the control plane first, then the agents: an older agent is served as
|
||
before and simply reports none of the new measurements. Existing agent units
|
||
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
||
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
||
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
||
start with `mp` unless `launch_mode: ray` is set.
|
||
|
||
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
|
||
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
||
the keys table; it was run against a copy of a production PostgreSQL store
|
||
before release. `llmgw users move` is gone: use `join` and `leave`.
|
||
|
||
vLLM is installed on every GPU node that can run it, once its agent is
|
||
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
|
||
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
|
||
replaced by `llmgw node add`; it still works, for a machine holding the
|
||
authority but not this llmgw.
|
||
|
||
## v0.9.2 — 2026-09-28
|
||
|
||
### A new control plane serves a model by itself
|
||
|
||
`llmgw server init` now also runs a model mirror on the same machine, makes the
|
||
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
|
||
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
|
||
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
|
||
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
|
||
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
|
||
and every step is skipped on a rerun when it is already done.
|
||
|
||
Not yet run end to end on a real machine.
|
||
|
||
## v0.9.1 — 2026-09-28
|
||
|
||
### The mirror page is one line per model
|
||
|
||
What the mirror holds used to take two table rows per file — seven columns and
|
||
the address under them — so a sharded model filled the screen. Now each version
|
||
is one line: model and version, the file (or "4 files" for a set), total size,
|
||
and which catalog models use it. The address moved into a "copy address" button;
|
||
the checksum is in the file's tooltip. A version already in use is not offered
|
||
for serving again, and removing one warns that the catalog uses it. The header
|
||
counts versions rather than files.
|
||
|
||
## v0.9.0 — 2026-09-28
|
||
|
||
### Managing people and the tenancy tree
|
||
|
||
Organizations, teams and projects can now be renamed, moved and removed, and an
|
||
approved person can be moved to another project or removed — none of which
|
||
could be done at all before, from the web or the CLI.
|
||
|
||
- **`/admin/users`** shows the whole tree, each project opening onto its people
|
||
and keys, with approve, move, suspend, reinstate and remove for people,
|
||
revoke for keys, and add, rename, move and remove at every level. A search
|
||
over every account sits below it. Changes update the page in place (htmx,
|
||
embedded in the binary) and it still works with scripts off.
|
||
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
|
||
- **`llmgw users move`** and **`llmgw users delete`**.
|
||
- **`llmgw keys list --all`** lists every key with its project and owner.
|
||
- Every command that takes a project, team or organization accepts an id, a
|
||
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
|
||
be omitted when there is only one. Users are named by id or email.
|
||
|
||
Moving a person takes the keys they made with them; keys an operator issued stay
|
||
with the project. Only empty things are removed, and a project that ever had a
|
||
key is never removed, because usage history refers to it.
|
||
|
||
### Changes
|
||
|
||
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
|
||
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
|
||
built from the web has no project of that name. With one project it is
|
||
chosen; with several the command lists them.
|
||
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
|
||
rather than micro-units. `--budget 50000000` now means fifty million units;
|
||
divide existing scripts' figures by 1 000 000.
|
||
|
||
### Fixes
|
||
|
||
- `keys init` could silently move an existing team or project to another
|
||
organization or team by redeclaring it. It now refuses.
|
||
- An account rejected while pending and then reinstated became active with no
|
||
project, and failed at its first key. Reinstating one is refused; approving
|
||
gives it a project. Rejecting a signup now removes it.
|
||
|
||
## v0.8.8 — 2026-09-28
|
||
|
||
### Fixes
|
||
|
||
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
|
||
executable file not found". The PostgreSQL project's own packages (PGDG)
|
||
provide `postgresql-server` too, so dnf can answer that name with
|
||
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
|
||
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
|
||
reads the data directory from it, and runs `initdb` itself when the directory
|
||
is empty. A PostgreSQL that was already installed is used as it is.
|
||
|
||
### Known gaps
|
||
|
||
Not yet run on a real machine. Three arrangements are not handled: a
|
||
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
|
||
and says so), one listening on a port other than 5432, and several PGDG
|
||
versions where the highest is not the one on 5432.
|
||
|
||
## v0.8.7 — 2026-09-28
|
||
|
||
### PostgreSQL in one command
|
||
|
||
```
|
||
sudo llmgw server init --postgres
|
||
```
|
||
|
||
Installs PostgreSQL with the machine's own package manager when there is none,
|
||
creates the `llmgw` role and database with a generated password kept in
|
||
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
|
||
already running on SQLite it stops the gateway, copies the store across so
|
||
every account and key survives, points `auth.store` at PostgreSQL and starts
|
||
the gateway again; if the move fails, the gateway is started again on SQLite.
|
||
Given on the first run, there is never a SQLite store.
|
||
|
||
`server.yaml` is edited in place with its comments intact, and the previous
|
||
file is kept beside it. A PostgreSQL that init did not install is left sized as
|
||
it was, since something else may depend on it.
|
||
|
||
The PostgreSQL installation itself has not yet run on a real machine; the move,
|
||
the configuration edit and the rerun behaviour are tested.
|
||
|
||
### Fixes
|
||
|
||
- **A new control plane with no models did not start.** A configuration with
|
||
no static endpoints and no models was refused as having nothing to serve, so
|
||
under systemd the unit restarted every two seconds and nothing listened on
|
||
8080. It now runs as a control plane with an empty catalog.
|
||
|
||
## v0.8.6 — 2026-09-28
|
||
|
||
### Reverse proxy configurations
|
||
|
||
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
|
||
port, and both are attached to every release beside the binaries. They do not
|
||
buffer streamed replies, their read timeout outlasts a model's wait before its
|
||
first token, their body limit matches the router's, and they set
|
||
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
|
||
cookies `Secure` from the first and shows the second as the base URL on its
|
||
landing page.
|
||
|
||
## v0.8.5 — 2026-09-28
|
||
|
||
### A control plane in one command
|
||
|
||
```
|
||
sudo llmgw server init --admin you@example.com
|
||
```
|
||
|
||
Creates the llmgw user and its directories, the certificate authority, the
|
||
control plane's certificate for the machine's hostname and every address it
|
||
has, `server.yaml`, the default project, and an administrator whose password is
|
||
generated and printed once — then installs the systemd unit, starts it and
|
||
waits until it answers. What used to be two steps of install.md is now this.
|
||
|
||
Each step is skipped when its result exists, so running it again finishes an
|
||
install that stopped and changes nothing on one that did not. `--host` adds an
|
||
address the machine cannot see of itself (a DNS alias, a NAT or floating
|
||
address) and reissues the certificate keeping the old ones. It never overwrites
|
||
`server.yaml` or an authority, and certificates issued by an authority kept on
|
||
another machine are left as they are.
|
||
|
||
### Fixes
|
||
|
||
- **The configuration install.md showed could not start.** An empty catalog
|
||
without `control.enabled: true` is refused as having nothing to serve. The
|
||
documentation and the file `init` writes both set it.
|
||
|
||
## v0.8.4 — 2026-09-28
|
||
|
||
### Node logs on the admin page
|
||
|
||
Each node on the cluster page links to its logs — the agent's own, and the
|
||
output of every engine running there — with an option to follow them. `llmgw
|
||
ctl logs NODE` reads the same. They come over the agent's existing stream, so
|
||
no ssh, jump host or root is needed to see what a node did. The agent keeps its
|
||
last 2000 lines in memory; each engine its last few hundred.
|
||
|
||
## v0.8.3 — 2026-09-28
|
||
|
||
Found by building a node from nothing with v0.8.2.
|
||
|
||
### Fixes
|
||
|
||
- **A new node's first engine install failed on apt's lock.** cloud-init and
|
||
unattended-upgrades run apt in a machine's first minutes, and
|
||
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
|
||
package command held up by a lock now waits for up to ten minutes, and says
|
||
so; a failed install is retried after a minute rather than ten.
|
||
- **A new node reported no disk** until its first download created the weights
|
||
directory. It is created when the agent starts.
|
||
- **Onboarding took one control plane address.** It takes several, comma
|
||
separated, so a new node fails over like the rest.
|
||
|
||
## v0.8.2 — 2026-09-28
|
||
|
||
A node no longer runs whatever engine somebody once put on it. It installs its
|
||
own.
|
||
|
||
### Engines are installed by the node
|
||
|
||
`engines:` in server.yaml names the engines every node has and their versions,
|
||
defaulting to the llama.cpp build this release was checked against (b10991).
|
||
Each agent converges on it: it picks the build for its hardware — CUDA with its
|
||
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
|
||
packages that build needs with the machine's own package manager, downloads the
|
||
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
|
||
anything that would land outside its directory, and runs it. `ldd` names a
|
||
missing library or a system too old for the build ("libstdc++.so.6 is too old
|
||
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
|
||
again every ten minutes.
|
||
|
||
An engine is reported only once installed and seen to run, or when the agent's
|
||
configuration names it on purpose. A node that has installed nothing yet is not
|
||
a candidate for any model, where before it was a candidate for all of them.
|
||
`ctl nodes`, the cluster page and placement's refusals say where each
|
||
installation stands, and warn about a CPU without AVX2.
|
||
|
||
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
|
||
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
|
||
installed this way and must still be named in the agent's configuration.
|
||
|
||
### Onboarding
|
||
|
||
Onboarding installs llmgw from the CI release for the node's architecture,
|
||
checked against the release's checksums, and writes an agent configuration of
|
||
only the control plane's addresses and the TLS files.
|
||
|
||
### Fixes
|
||
|
||
- A race on the weights fingerprint, which two heartbeat loops touched during a
|
||
failover.
|
||
|
||
## v0.8.1 — 2026-09-28
|
||
|
||
Nothing here changes a configuration format: every file that worked with
|
||
v0.8.0 still works. What changes is how much of it needs writing, and how much
|
||
of running the cluster can be done from a page rather than a shell.
|
||
|
||
### A node describes itself
|
||
|
||
An agent file used to name its node, its address, the path to llama-server and
|
||
its weight directories — different on every node and kept true by hand. The
|
||
agent now finds these out: its id is the hostname, its address is the one on
|
||
the route to the control plane, llama-server is looked for on PATH and in
|
||
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
|
||
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
|
||
Memory is re-read on every heartbeat and installed engines once a minute, so a
|
||
VM given more RAM or a node given an engine becomes a candidate without an
|
||
agent restart; the control plane logs both. The agent file shrinks to the
|
||
control plane's addresses and the TLS files, the same on every node.
|
||
|
||
### Models from the mirror, without writing YAML
|
||
|
||
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
|
||
and serve a mirrored version from its row — name, source, engine and an
|
||
estimated memory requirement filled in. A model can say what it is for, and
|
||
`/v1/models` shows it. Placement no longer hands the same memory to two models
|
||
or puts an engine on a node that lacks it, and says when a replica runs an
|
||
edited profile.
|
||
|
||
### Tenancy, quotas and usage on pages
|
||
|
||
The tenancy tree, quotas and budgets are managed from the admin pages; an
|
||
organization's budget bounds its teams, a group's quota counts against the
|
||
group, and a budget that cannot be spent against says so. `/admin/usage`
|
||
charts requests, tokens and spend over time by organization, team, project or
|
||
person, and a request is priced by the model that served it.
|
||
|
||
### Watching the cluster
|
||
|
||
The cluster page has its own charts, backed by a year of Prometheus history,
|
||
with alerts that were made to fire before being trusted — including for the
|
||
disk the metrics share with the models.
|
||
|
||
### Chat
|
||
|
||
The picker offers each model once rather than once per alias, never asks for
|
||
more tokens than the model allows, and offers a model that loads on demand —
|
||
sending it a message is what loads it.
|
||
|
||
## v0.8.0 — 2026-09-27
|
||
|
||
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
||
an existing installation onto it without reissuing every key, and the two
|
||
things a second control plane needs so that running one is safe rather than
|
||
clever.
|
||
|
||
### A second control plane, and a way to fail over to it
|
||
|
||
Leadership is a lease in the store, so two control planes against one database
|
||
cannot both place work — which they otherwise would, both seeing a model
|
||
wanting replicas and none running, leaving twice the replicas each holding GPUs
|
||
the other believes are free.
|
||
|
||
A standby also refuses agent streams. Commands travel down the stream the agent
|
||
opened, so a standby holding one is a node the leader cannot command, and the
|
||
leader's placements would fail with "no agent connected" for a node that is
|
||
plainly connected. Agents therefore take a list of addresses rather than one,
|
||
trying each in turn: a standby refusing, a stopped control plane refusing the
|
||
connection and a crashed one never answering are the same thing from a node.
|
||
|
||
Measured on the cluster. Leadership moved three seconds after the leader was
|
||
stopped — it released the lease on the way out rather than letting it lapse —
|
||
and all five agents had registered with the standby six seconds later. The
|
||
standby adopted every replica rather than restarting any: six replicas, all
|
||
ready, engines that never stopped serving.
|
||
|
||
Everything is unset-means-leader, so a single control plane and every
|
||
deployment that existed before this behave exactly as they did.
|
||
|
||
What this does not do is move client traffic. The standby served inference
|
||
correctly on its own port while the public address returned 502, because
|
||
nothing in front of them moved — that needs a load balancer or a shared
|
||
address, and it is the remaining piece.
|
||
|
||
### PostgreSQL
|
||
|
||
The store runs on PostgreSQL, chosen by what the address looks like: a path is
|
||
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
|
||
address out of a file so a DSN carrying a password stays out of a configuration
|
||
that gets copied and pasted into issues. Every existing installation keeps
|
||
working untouched.
|
||
|
||
One set of queries rather than one per database, because the SQL here is almost
|
||
entirely portable and two copies would drift one query at a time. The rewrite
|
||
of placeholders is a wrapper around the handle rather than a change at sixty
|
||
call sites — which found six places reaching past it, including all three
|
||
manual transactions and two prepared statements, every one of which worked on
|
||
SQLite and would have failed on PostgreSQL.
|
||
|
||
Exactly one query is not the same sentence in both: the note search, MATCH
|
||
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
|
||
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
|
||
to keep in step and no trigger to forget. It uses the 'simple' configuration
|
||
rather than English — these are an agent's own notes, in whatever language it
|
||
chose, and stemming English rules over Turkish would quietly stop finding
|
||
things.
|
||
|
||
`llmgw keys move-store` copies an existing store rather than starting a fresh
|
||
one, because keys are stored hashed: a migration that drops them cannot
|
||
recreate them, and every person and every script would need a new one.
|
||
|
||
Testing it against a real database found three things nothing else would have.
|
||
`schema_migrations` is created in Go rather than in a migration file, so it
|
||
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
|
||
and a millisecond timestamp does not fit. The copy order followed the order the
|
||
migrations create tables in, and `users` is created after `api_keys` even
|
||
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
|
||
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
|
||
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
|
||
difference pgx cannot see, so it arrived as NULL.
|
||
|
||
The driver is pgx, which is pure Go: the whole binary is built with
|
||
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
|
||
and a cgo driver would take that away from every deployment, including the ones
|
||
staying on SQLite.
|
||
|
||
## v0.7.0 — 2026-09-25
|
||
|
||
What a cluster needs to be lived with rather than demonstrated: somewhere to
|
||
keep a conversation, somewhere to keep a note, and an honest answer about the
|
||
disk underneath both.
|
||
|
||
### Memory notes for agents, and threads without API keys
|
||
|
||
Notes an agent writes for itself and reads back in later conversations, offered
|
||
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
|
||
`forget`). A note is a key, text and tags, searched by words or tag, private to
|
||
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
|
||
tools into chat requests: an agent that is not given the server never sees
|
||
memory.
|
||
|
||
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
|
||
and the retention is required, as it is for threads.
|
||
|
||
On a gateway with no authentication, a `threads:` block (`store`,
|
||
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
|
||
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
|
||
nothing refuses it with an explanation instead. The shared pool is for one
|
||
trusted machine: anyone who can reach the port and knows an id can read it.
|
||
|
||
### Conversations that can move between models
|
||
|
||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||
llmgw replays the stored history to whichever model is serving, which is what
|
||
lets a conversation begun on a large model continue on a small one — including
|
||
when a spent budget moved it there and the caller never chose.
|
||
|
||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||
stores no message content anywhere, as it never has, and that default is worth
|
||
more than the feature. The retention is required rather than optional, because
|
||
a conversation store with no expiry is a transcript of everything a team ever
|
||
typed kept forever by omission, and it is measured from the last turn so a
|
||
conversation in progress does not expire underneath it.
|
||
|
||
History that does not fit the serving model is trimmed oldest-first and the
|
||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||
are never dropped — losing one silently changes what the model is for while it
|
||
carries on answering — and neither is the turn the caller just sent.
|
||
|
||
Threads are scoped to a project, and reading, writing or deleting one that
|
||
belongs to another answers exactly as one that never existed: two answers would
|
||
let a caller map which ids are real in projects they cannot read.
|
||
|
||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||
accumulates nothing on the hottest path in the system, and one with a thread
|
||
stops at a cap rather than growing to it.
|
||
|
||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||
operator: the person who said the thing should not have to ask someone to read
|
||
it before it can go.
|
||
|
||
### Disk, and taking back only what llmgw put there
|
||
|
||
Disk was the one resource nothing reported. `statfs` was already being called
|
||
on every model directory, but only to ask whether the filesystem was a network
|
||
one — the free space in the same struct was discarded, so the only way to find
|
||
out why a node was filling up was to ssh to it.
|
||
|
||
Nodes now report it, per filesystem rather than per configured directory: free
|
||
space belongs to the disk, and two model directories on one disk are one pool
|
||
of bytes. Shared filesystems are kept out of a node's own figure, because space
|
||
on a filer belongs to everyone mounting it and twenty nodes would otherwise
|
||
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
|
||
Prometheus, and a node that reported nothing reads as unknown everywhere — an
|
||
older agent must not look like a full disk.
|
||
|
||
A profile may declare `min_disk_gb`, and a node without the room is no longer a
|
||
candidate, with a reason `explain-placement` prints. Skipped on a node that
|
||
already holds the weights: it needs no room for them, and without that the node
|
||
best placed to run a model is the one most likely to be rejected.
|
||
|
||
`ctl weights` lists model files llmgw downloaded for models no longer in the
|
||
catalog, and `--collect` deletes them. The design is mostly about what it will
|
||
not do. Only files llmgw fetched are candidates, and that is a calculation
|
||
rather than a heuristic — the cache name is a hash of the source, so the wanted
|
||
set is computable exactly. Anything an operator put on a node is left alone
|
||
however unreferenced it looks. Each node checks every path again before
|
||
unlinking it, refusing anything outside its cache directory or without the
|
||
cache name, so a control plane that is wrong cannot talk a node into losing a
|
||
model that took a week to copy.
|
||
|
||
### Fixes
|
||
|
||
- **A capability was lost at an indirection.** The controller is handed a
|
||
forwarding dispatcher, and reaches past its interface by type assertion to
|
||
find what lies beyond scheduling. Deleting weights was implemented on the
|
||
agent service and not on the forwarder, so it was simply absent at runtime —
|
||
with the whole suite green, because the dispatchers the tests use do not
|
||
implement it either and its absence there is the expected case.
|
||
- **`ctl tool-check` blamed the engine for what the model did.** It had one
|
||
verdict for a failed attempt and told operators not to try another model.
|
||
Measured: the engine renders the tool prompt correctly, the coder models do
|
||
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
|
||
the same engine returns a populated `tool_calls` array. A failed attempt is
|
||
now two verdicts, told apart by whether the reply used the wrapper.
|
||
|
||
## v0.6.0 — 2026-09-18
|
||
|
||
Phase 4: a replica can be bigger than one machine, and the fabric it spans
|
||
decides where it is allowed to go. Around that, everything a cluster needs
|
||
before anyone other than its operator is let near it — keys, budgets, a way in
|
||
through a browser, and enough visibility to answer "why is nothing scheduling".
|
||
|
||
### Multi-node replicas
|
||
|
||
A replica is now a set of shards with ranks, a leader and a bootstrap address,
|
||
rather than one process. Two invariants are enforced rather than documented: a
|
||
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
|
||
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
|
||
several times slower, and nothing says which.
|
||
|
||
A replica missing a shard is not ready. It was previously derived from the
|
||
shards that had reported, so a two-node replica with one node up looked
|
||
servable and took traffic into a model that could not answer.
|
||
|
||
Adoption after a restart happens one agent at a time, because that is how
|
||
agents reconnect. Merging them as they arrive is what keeps a multi-node
|
||
replica whole instead of being torn into fragments that are then killed as
|
||
strays — and an adopted engine is health-checked before it is called ready,
|
||
rather than being assumed alive because its process exists.
|
||
|
||
### Fabrics are distribution domains too
|
||
|
||
An isolated fabric is not only a compute boundary: weights present in one are
|
||
not present in the other. Node weight inventories are reported and summarised
|
||
per fabric, so "the model is on the cluster" is never mistaken for "the model
|
||
is reachable from the nodes that would run it". A fabric name that matches
|
||
nothing a node actually has is refused at registration rather than becoming an
|
||
unschedulable replica hours later.
|
||
|
||
### Who may use it
|
||
|
||
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
|
||
still no cgo). Rate limits and budgets are enforced on the request path; usage
|
||
is written off it, in batches, so accounting never adds latency to inference.
|
||
Budgets survive a restart — counters are rebuilt from the store, because a
|
||
limit anyone can clear by bouncing the process is not a limit.
|
||
|
||
A spent budget is now served a cheaper model where the catalog offers one,
|
||
rather than refused. For a team trying models out, being dropped to something
|
||
smaller is a far better answer than a wall, and the response says which model
|
||
answered.
|
||
|
||
People can sign up, be approved, and issue and revoke their own keys. A key is
|
||
shown once; only its hash is kept.
|
||
|
||
### The browser surface
|
||
|
||
A landing page that says how to connect, a read-only cluster dashboard, and an
|
||
in-browser chat that never puts an API key in the browser. The dashboard shows
|
||
who is using the cluster and what they are running — per person and per model,
|
||
with requests, tokens, errors, spend and latency.
|
||
|
||
All of it is served from the client-facing port behind a session, deliberately
|
||
not from the admin port, which also serves pprof — and pprof can dump the heap
|
||
that holds every API key in plaintext.
|
||
|
||
### Models without a restart
|
||
|
||
The catalog is live. Models can be added and removed from the command line or
|
||
the API, take effect immediately, and outlive a restart; configuration still
|
||
wins over the store, and the whole catalog is validated before anything is
|
||
written.
|
||
|
||
A model is only listed once it can actually serve. It was listed as soon as it
|
||
was known, so the first thing a new user did was ask for a model that returned
|
||
503.
|
||
|
||
### Loading on demand, and letting go
|
||
|
||
A model whose scaling allows it is started when someone asks for it, and
|
||
stopped when nobody has for a while. An idle replica is evicted to make room
|
||
for one that was asked for, with guards that exist because each is a way to
|
||
lose work quietly: only on-demand models, only single-node replicas, never one
|
||
that has never been used, and never one younger than its minimum lifetime.
|
||
|
||
Unknown is not idle. A replica with no recorded use is left alone rather than
|
||
treated as idle since the epoch.
|
||
|
||
### Knowing whether it will run before it runs
|
||
|
||
- `ctl plan` works out whether a model fits the cluster and how it would be
|
||
split, in the precision the chosen engine can actually serve — recommending
|
||
a quantisation the engine cannot load would cause the failure the command
|
||
exists to prevent.
|
||
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
|
||
installed. Flags drift between releases, and the alternative is finding out
|
||
during a model load.
|
||
- Placement now knows a model's context length and memory footprint, and node
|
||
RAM, so a CPU profile is not placed on a machine that cannot hold it.
|
||
- Placement waits for the fleet to settle after a restart rather than deciding
|
||
against a half-reported cluster.
|
||
|
||
### Weights
|
||
|
||
Nodes report the model files they can see, without llmgw owning the storage:
|
||
a shared filer, an image exported over the fabric, rsync or a copy made by hand
|
||
all look the same to placement, which only cares whether the files are there.
|
||
Reporting nothing means "not asked to look", not "has nothing".
|
||
|
||
Where the simple arrangement is not enough, a node can fetch a model by URL —
|
||
single-flight, resumable, written to a temporary name and renamed atomically,
|
||
so a killed download never leaves something that looks like weights.
|
||
|
||
### Seeing the fleet
|
||
|
||
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
|
||
scrape time rather than from a background goroutine, and served on their own
|
||
endpoint rather than through the one that also exposes pprof. The same view is
|
||
in the dashboard, so nothing else needs installing to see it.
|
||
|
||
### Onboarding a node
|
||
|
||
Adding a node is one command. There are systemd units, a private CA and a
|
||
from-nothing install guide that starts with a machine that has no CUDA and no
|
||
binary.
|
||
|
||
### llama.cpp, and CPU profiles
|
||
|
||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||
can all be exercised on ordinary machines.
|
||
|
||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||
allocation to stop two of them sharing a machine's cores.
|
||
|
||
### Metric mappings are per engine
|
||
|
||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||
and a single global mapping left half of it unmeasurable.
|
||
|
||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||
violation. `Running` and `Waiting` can never be omitted.
|
||
|
||
### Mutual TLS on the agent stream
|
||
|
||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||
node id does not match its certificate's common name. Public authorities cannot
|
||
supply these: they stopped issuing the `clientAuth` usage.
|
||
|
||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||
agent turn it on.
|
||
|
||
### Fixes
|
||
|
||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||
context and then asked that context whether it was shutting down, so every
|
||
stream error read as a clean close: no message, and the backoff reset.
|
||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||
level, so a cluster could look healthy while every routing decision was made
|
||
blind. It is now a warning, once, with a matching line when it clears.
|
||
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
|
||
over about 2 KB was undercounted, so tokens-per-minute limits let through far
|
||
more than they were set to.
|
||
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
|
||
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
|
||
bfloat16 does not fit two 80 GB cards. It does.
|
||
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
|
||
already written the response header, which is the one point at which headers
|
||
stop being changeable. A client had no way to tell a fallback from the model
|
||
it asked for.
|
||
- **Fleet metrics published nothing.** They were registered after the registry
|
||
had been built, so the collector existed and was never scraped.
|
||
- **GPU discovery broke on older drivers.** A field added for utilisation was
|
||
required rather than optional, so a driver that does not report it produced no
|
||
inventory at all instead of an inventory without that number.
|
||
- **Replica ids and ports were reused** while the processes holding them were
|
||
still alive.
|
||
|
||
## v0.5.0 — 2026-09-16
|
||
|
||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||
control plane places replicas, and `ctl` makes all of it visible.
|
||
|
||
This release covers both phases because their exit criteria depended on the same
|
||
piece — the agent stream — and neither was demonstrable without it.
|
||
|
||
### ctl
|
||
|
||
```
|
||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||
```
|
||
|
||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||
the answer when a node looks idle but nothing schedules onto it.
|
||
- `explain-placement` prints every candidate considered and the reason it was
|
||
rejected. "Unschedulable" on its own says nothing about which of six
|
||
constraints bit.
|
||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||
is placed elsewhere, and the original is stopped only once the replacement is
|
||
serving. Bounded by a grace period, because draining the only node that can
|
||
host a model would otherwise never finish and leave it undrainable.
|
||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||
|
||
The management API is on the admin listener, not the client-facing one. The two
|
||
have different audiences: one serves inference to whoever holds an API key, the
|
||
other lets a caller cordon nodes.
|
||
|
||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||
not have. The failure path already ships an engine log tail with every status
|
||
report, so the case that matters most is covered; reading a healthy shard's
|
||
output on demand is not.
|
||
|
||
### Phases 2 and 3, in detail
|
||
|
||
Written as the work landed, which is why it reads as a log rather than as
|
||
notes. Everything here shipped in v0.5.0.
|
||
|
||
### Added
|
||
|
||
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
|
||
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
|
||
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
|
||
without IB is not an error — it is a single-node island.
|
||
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
|
||
would force the whole binary, server included, to be built with a C
|
||
toolchain. Health flags anything that should keep a card out of placement:
|
||
uncorrected ECC errors, and memory already held by another process.
|
||
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
|
||
always match the marketing name: an L40S is sold as 48GB and reports 44.99
|
||
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
|
||
box, because the number decides whether a model fits.
|
||
- **Engine adapters** for vLLM and SGLang: command and environment
|
||
construction, kept free of process handling so it is testable without GPUs.
|
||
vLLM supports both Ray and multiprocessing launch modes, because which one
|
||
works depends on the version.
|
||
- **NCCL environment generation**, the part plan §5.3 marks critical.
|
||
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
|
||
NCCL pick an adapter on another fabric where the peer is unreachable, and the
|
||
job hangs rather than failing. Bootstrap is pinned to the management
|
||
interface for the same reason.
|
||
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
|
||
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
|
||
compute capability 8.9 rather than silently falling back to a slower path.
|
||
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
|
||
process group and are stopped as a group, because engines fork workers that
|
||
hold GPU memory and outlive a signal sent only to the leader. Bounded log
|
||
tail, a state machine that separates Starting from Loading (a 70B model takes
|
||
minutes, and an operator needs to know that is normal), and orphan adoption
|
||
through pid files fingerprinted with the command line — pids get reused, and
|
||
adopting an unrelated process is worse than leaking the original.
|
||
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
|
||
says the weights loaded. Routing on `/health` alone sends traffic into a model
|
||
that is still loading.
|
||
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
|
||
as a table or JSON. Run it before adding a node: it is the fastest way to find
|
||
a card another process is using, or a port with no subnet manager.
|
||
|
||
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
|
||
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
|
||
a tensor-parallel group off the slow path inside a node. `Discover` falls back
|
||
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
|
||
topology. CI compiles the tagged build so it cannot rot unnoticed.
|
||
|
||
### Not verified
|
||
|
||
The NVML path compiles but has never run against a driver — there is no GPU
|
||
machine to try it on. Treat the NVLink domains it reports as unverified until
|
||
someone runs `llmgw agent inventory` on a real node.
|
||
|
||
- **Control plane wiring** (`internal/control`): Ready replicas become router
|
||
endpoints, with no static configuration involved. The router keeps live state
|
||
for endpoints that survive an update — inflight counts, breaker state, drain
|
||
rate — so a replica that was already serving does not start from cold because
|
||
another one moved. The sink is an interface, so when the router and control
|
||
plane split into separate processes this becomes a watch stream and nothing
|
||
above it changes.
|
||
|
||
### Phase 3 exit criterion, measured in `test/e2e`
|
||
|
||
A node hosting a live replica is killed while four clients stream continuously.
|
||
The control plane marks it unreachable, then lost, rebuilds the replica on a
|
||
survivor, and republishes. Across three runs: **~370 requests completed, zero
|
||
failures**, and no replica remained on the dead node.
|
||
|
||
This proves the control logic, not the transport: the agent fleet is simulated,
|
||
and the real path still needs the stream.
|
||
|
||
### A real bug the race detector caught
|
||
|
||
`startReplica` read the replica's fields after publishing it to the map, and
|
||
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
|
||
status reports, which arrive from another goroutine and mutate exactly those
|
||
fields. Under the simulated fleet the reports came back fast enough to hit the
|
||
window every time. Everything is snapshotted under the lock now and the
|
||
functions work from values.
|
||
|
||
### The agent stream
|
||
|
||
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
|
||
carrying registration, heartbeats and shard status upward, and start, stop and
|
||
drain commands downward. The agent dials out, because GPU nodes sit behind
|
||
firewalls far more often than a control plane does and an outbound connection
|
||
needs no inbound rule on every node in the fleet.
|
||
- **Registration carries the whole inventory**, so there is no separate
|
||
discovery call and a reconnecting agent refreshes what the scheduler knows.
|
||
The fabric labels travel with it — they cannot be inferred from the wire, so
|
||
losing them in transit would silently disable multi-node placement.
|
||
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
|
||
the reconciler's dispatcher: it holds the open streams, so it is the only
|
||
thing that can deliver a command to a node.
|
||
- **Agent** (`internal/agent`) validates a profile before launching anything —
|
||
an engine that refuses a profile does so after minutes of loading — then
|
||
supervises the process and reports every transition, including the engine's
|
||
output tail on failure.
|
||
- **Reconnects forever with jittered backoff.** A control plane restart must not
|
||
require touching every GPU node, and a fleet must not reconnect in lockstep
|
||
and knock the control plane over again. Engines keep serving throughout: the
|
||
router talks to them directly.
|
||
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
|
||
any model declares a profile. The static endpoint list remains for
|
||
router-only runs against sim-engines.
|
||
|
||
### Verified with real binaries
|
||
|
||
Server and agent as separate processes over a real stream: the agent registers,
|
||
a replica is placed, an engine is launched through the vLLM adapter's actual
|
||
argument construction, streaming traffic is served through the router, and
|
||
killing the agent moves the node through unreachable to lost with placement
|
||
explaining exactly why it can no longer run anything.
|
||
|
||
### Surviving a control plane restart
|
||
|
||
Not persisting replica state left one hole that had to be closed before it
|
||
mattered: restarting the control plane orphaned live engines. They kept holding
|
||
GPUs the inventory believed were free, so the next placement failed to allocate
|
||
with an error naming the new process rather than the old one.
|
||
|
||
The fix is on the agent, not in a database, and it cannot disagree with reality
|
||
the way a stale row can:
|
||
|
||
- The supervisor now writes the full shard spec, not just a pid and command
|
||
line. A pid identifies a process; the spec is what lets it be described back
|
||
to the control plane and matched to a replica.
|
||
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
|
||
waited on — it is not this process's child — so it is watched by polling, and
|
||
stopped by signalling its group directly. Its earlier output is gone, and the
|
||
log says so rather than presenting emptiness as silence.
|
||
- Every `Register`, not only the first, reports what the agent is running. A
|
||
reconnect may be a control plane restart and has to recover the same way.
|
||
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
|
||
Anything whose model or profile left the catalog is stopped: nothing will
|
||
route to it and it is holding cards.
|
||
- An engine the agent cannot identify — a corrupt state file — is killed. A
|
||
process holding GPUs that nothing can describe is worse than no process,
|
||
because placement will hand those GPUs out again.
|
||
|
||
One mechanism covers agent restarts and control plane restarts.
|
||
|
||
### A test that was passing for no reason
|
||
|
||
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
|
||
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
|
||
tracking hijacked connections. The agent never reconnected and the assertion —
|
||
on the connection map, which still held a stale entry — passed anyway. Tests now
|
||
sever connections through their own listener and count opened streams.
|
||
|
||
### Multi-node launch
|
||
|
||
Placement and environment generation had landed early, which made multi-node
|
||
look further along than it was: a multi-node replica could not come up at all.
|
||
Three things were missing, and all three came down to two assumptions that do
|
||
not hold once a replica spans nodes.
|
||
|
||
**Not every rank serves HTTP.** The agent health-checked every shard at its own
|
||
port, but only the leader answers. Workers timed out, reported Failed, and took
|
||
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
|
||
worker is judged by staying alive instead. That is weaker evidence on purpose —
|
||
the replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined.
|
||
|
||
**A shard is not always one process.** `Engine.Commands(spec)` returns a
|
||
sequence: every command but the last is a sidecar kept alive for the shard's
|
||
lifetime and torn down with it, and the last one's exit is the shard's exit.
|
||
vLLM under Ray is why — the leader runs a Ray head and then the server against
|
||
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
|
||
had been written and tested but never called, so `launch_mode: ray` would have
|
||
run `vllm serve` against a cluster that did not exist.
|
||
|
||
A command can also require the rendezvous point to be listening before the next
|
||
one starts, bounded by a timeout so a head that never appears fails the shard
|
||
rather than hanging it. Sidecars are killed with the shard: a Ray head left
|
||
behind holds the port the next replica needs.
|
||
|
||
**Start order was backwards.** Workers were started first, on the theory that
|
||
the head should wait for them. Both engines put the rendezvous point on rank 0,
|
||
so a worker started first has nothing to connect to and spends its startup
|
||
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
|
||
needs, so the leader can start immediately and waits on its own.
|
||
|
||
None of this is verified against real engines. It is tested with stub processes;
|
||
the Ray sequencing and which launch mode a given vLLM version wants can only be
|
||
confirmed on hardware.
|
||
|
||
### Still missing for phase 2
|
||
|
||
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
|
||
network.
|
||
|
||
### Phase 3: placement
|
||
|
||
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
|
||
their fabric attachments, and what is allocated where. A re-registering agent
|
||
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
|
||
without doing so either. A GPU with an ECC fault or another process on it is
|
||
not counted as free. Replicas are released whole, because they fail whole.
|
||
- **Placement** (`internal/control/placement`): picks the highest-priority
|
||
profile that fits and explains every candidate it rejected. Two constraints
|
||
are enforced structurally rather than checked afterwards — there is no code
|
||
path that can combine nodes from two fabrics, and only GPUs of the profile's
|
||
class are ever considered.
|
||
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
|
||
large models that cannot be split. Multi-node replicas get the opposite
|
||
treatment: they are the least movable thing in the cluster, so they take the
|
||
emptiest machines.
|
||
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
|
||
domain where the topology is known. Without the NVML build it falls back to
|
||
the lowest free indices, which is usually right but only by accident of how
|
||
vendors number slots.
|
||
- **Property-based invariant tests** (plan §13): 400 randomly generated
|
||
clusters, checking on every placement that a multi-node replica lives on one
|
||
fabric, that each shard receives only that fabric's adapters, and that no
|
||
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
|
||
The table-driven tests cover the cases someone thought of; this covers the
|
||
ones nobody did.
|
||
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
|
||
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
|
||
rather than at request time — a typo in a fallback should stop a deployment,
|
||
not surface during the outage when the fallback is finally needed. The
|
||
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
|
||
load instead of minutes into model loading.
|
||
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
|
||
loop. It places one replica per pass, because placement depends on what is
|
||
already allocated and starting several from one snapshot would hand them the
|
||
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
|
||
vLLM under Ray needs its workers attached before the head serves.
|
||
- **Replica failure is whole-replica failure.** One dead shard makes the others
|
||
useless — they wait on collectives that will never complete while holding
|
||
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
|
||
exponential backoff carried on the model rather than the replica it just
|
||
destroyed.
|
||
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
|
||
blip must not tear down replicas that are still serving, and rebuilding a
|
||
multi-node replica costs tens of minutes of model loading.
|
||
- **Scale-down drains before stopping**, so in-flight requests finish instead of
|
||
being cut off mid-stream.
|
||
|
||
## v0.3.0 — 2026-09-16
|
||
|
||
Flow control. A batch workload no longer starves interactive traffic.
|
||
|
||
### Added
|
||
|
||
- **Admission control** (`internal/router/admission`), inserted between quota
|
||
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||
still answers *where* it goes. Keeping them apart is the point: an engine
|
||
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||
no amount of endpoint selection rescues the request behind it.
|
||
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||
nearly full cache, or the reverse.
|
||
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||
"flow up to the healthy limit"; above 1 means never wait.
|
||
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||
one caller sending a thousand requests occupies one slot in the rotation
|
||
rather than a thousand.
|
||
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||
(504), because sending the GPU work the client has already abandoned helps
|
||
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||
`x-llmgw-queue-ms` when a request waited.
|
||
|
||
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||
|
||
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||
With admission switched off the same load pushes it to **55–60x**. Both
|
||
directions are asserted, so the test fails if admission regresses *and* if the
|
||
load generator ever stops being a real load.
|
||
|
||
### Two lags, two different fixes
|
||
|
||
Getting this right took longer than the queueing itself:
|
||
|
||
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||
count does not — but it includes *running* requests, and treating those as
|
||
queue depth makes a healthy engine look overloaded every interval. What is
|
||
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||
the metric contract, so it is learned as the high-water mark of observed
|
||
`running`.
|
||
- Between admission granting a request and the router dispatching it, the
|
||
request is invisible to every signal. A simultaneous burst therefore passed
|
||
the check en masse — exactly the batch arrival pattern this layer exists to
|
||
survive. The controller now counts its own outstanding leases against
|
||
dispatched inflight.
|
||
|
||
### Known limitations
|
||
|
||
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||
Anyone who can reach the port can promote themselves to platinum.
|
||
- The tenant key is the session id, for the same reason.
|
||
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||
|
||
## v0.2.0 — 2026-09-16
|
||
|
||
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||
engine replicas with an OpenAI-compatible API.
|
||
|
||
### Added
|
||
|
||
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||
propagates cancellation: when a client disconnects, the upstream request is
|
||
aborted so the GPU stops working. Separate deadlines for dial, response
|
||
headers, first token, inter-token idle and total duration — a wedged engine is
|
||
indistinguishable from a slow one without the idle watchdog.
|
||
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||
reached the client; once bytes are out, splicing two responses together would
|
||
be worse than the short stream. Models can name fallback models, and the
|
||
response says which one served it via `x-llmgw-served-model`.
|
||
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||
Only routing-relevant fields are parsed; the rest of the body passes through
|
||
untouched.
|
||
- **Usage accounting that does not depend on the client.** The router can ask
|
||
engines for usage the client never requested, then drop that chunk on the way
|
||
back, so token counts are recorded either way.
|
||
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||
duplicate endpoint ids and aliases that would shadow another model — both
|
||
would otherwise misroute silently at request time.
|
||
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||
without seeing what the scorers actually produced. Served on a separate admin
|
||
listener alongside health and pprof.
|
||
|
||
### Phase 1 exit criterion, measured in `test/e2e`
|
||
|
||
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||
- Every client disconnecting frees the KV blocks on every engine.
|
||
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||
|
||
### Known limitations
|
||
|
||
- No admission control, so a batch workload can still starve interactive
|
||
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||
|
||
## v0.1.0 — 2026-09-16
|
||
|
||
First tagged version. It is useful for exactly one thing: developing and testing
|
||
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
||
returns "not implemented".
|
||
|
||
### Added
|
||
|
||
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
||
that models the behaviours a router has to cope with:
|
||
|
||
- continuous batching with a bounded concurrent sequence count
|
||
- KV cache block accounting, queueing when blocks run out, including
|
||
head-of-line blocking
|
||
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
||
effect on time to first token
|
||
- per-token latency that grows with batch size
|
||
- dropping a request and freeing its blocks when the client disconnects
|
||
- `/metrics` under vLLM's real metric names
|
||
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
||
|
||
**Routing primitives** — libraries, not yet wired into a server:
|
||
|
||
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
||
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
||
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
||
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
||
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
||
detection and drain rate derived from engine counters.
|
||
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
||
endpoints missing any of the mandatory three. The metric name mapping is
|
||
overridable, because names drift between engine versions.
|
||
- `router/breaker`: cooldown and circuit breaking in one component, with
|
||
exponential backoff and half-open probes.
|
||
|
||
### Known limitations
|
||
|
||
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
||
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
||
- No admission control, so a batch workload can still starve interactive traffic
|
||
once the router exists. That is what v0.3.0 is for.
|
||
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
||
|
||
### Notes
|
||
|
||
- The project language is English throughout: code, comments, tests, commits and
|
||
the design document.
|
||
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
||
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|