env is an allowlist of tuning families less path, plugin, library, address and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as flags. args refuse --trust-remote-code, --log-file, --slot-save-path, --tool-parser-plugin, --middleware, --allowed-local-media-path and others that read, write or load, and what the agent sets itself. Checked in the catalog and again on the node before an engine starts. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2715 lines
144 KiB
Markdown
2715 lines
144 KiB
Markdown
# Changelog
|
||
|
||
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
||
is only released once its **exit criterion** is met, not when its code lands.
|
||
|
||
| Version | Phase | Scope | State |
|
||
|---|---|---|---|
|
||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
|
||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
|
||
| v0.7.0 | — | Conversations, agent memory, disk management | released |
|
||
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
|
||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
|
||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||
|
||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||
APIs change between minors without a deprecation period.
|
||
|
||
## Unreleased
|
||
|
||
### Security
|
||
|
||
- **Built with Go 1.26.6.** Eleven standard-library vulnerabilities reachable
|
||
from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto)
|
||
are fixed by it. `make vuln` runs govulncheck; `make check` runs it with the
|
||
tests and vet.
|
||
- **Releases are signed, and only signed releases are installed.** The build
|
||
now carries the release signing key (`internal/release/trusted.pub`); the
|
||
control plane upgrading itself, an agent told to upgrade and a node being
|
||
added all refuse a release whose checksums are not signed with it. The CI
|
||
release job stops rather than publish an unsigned release. Adding a node
|
||
no longer has the node fetch the release and compare it with a checksum
|
||
from the same server: the control plane fetches it for the node's
|
||
architecture, checks the signature and the checksum, and hands it over.
|
||
Releases are fetched over https only (loopback excepted), and
|
||
`LLMGW_RELEASE_KEY` replaces the trusted key in development builds only.
|
||
- **The agent port is not opened without TLS by accident.** A control plane
|
||
with no `agent_tls` refused nothing: its agent port was plaintext, on every
|
||
interface, and any machine reaching it could register as any node — be
|
||
given replicas and so be sent prompts. It now refuses to start unless
|
||
`agent_insecure: true` is set, and then listens on loopback unless
|
||
`agent_listen` names an address. `server init` writes `agent_tls`, as
|
||
before.
|
||
- **A model definition cannot run code on its nodes.** A profile's `env` and
|
||
`args` reach a process the agent runs as root. The environment was checked
|
||
against a short denylist that missed NCCL's and CUDA's plugin and
|
||
injection libraries; arguments were not checked at all, so
|
||
`--trust-remote-code`, llama.cpp's `--log-file` or `--slot-save-path`, or
|
||
vLLM's `--tool-parser-plugin` were taken. `env` is now an allowlist of the
|
||
families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any
|
||
name for a path, plugin, library, address or credential, and llama.cpp's
|
||
`LLAMA_ARG_*` are checked as the flags they are; `args` refuse what reads,
|
||
writes or loads files, loads code, or moves what the agent sets (host,
|
||
port, key, model). Both are refused where a model is defined and again on
|
||
the node before the engine starts.
|
||
|
||
### Everything that can be changed on a running cluster, from the console
|
||
|
||
`sudo llmgw` only watched the cluster and edited the configuration files;
|
||
changing a model's replicas still meant `llmgw ctl`. Now Enter on a row opens
|
||
what can be done to it, as forms and questions:
|
||
|
||
- **Models**: replicas (min, max, idle timeout), class, owner, resources, LoRA
|
||
adapters, restart, retry when parked, why it is placed where it is, remove;
|
||
A adds one from a definition file.
|
||
- **Nodes**: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its
|
||
agent, delete model files nothing uses, remove.
|
||
- **Replicas**: restart, replaced or in place.
|
||
- **Quotas** (new): capacity quotas, set and removed.
|
||
- **Mirror** (new): what it holds and which models use it; download into it,
|
||
remove from it.
|
||
- **People** (new): accounts — approve, suspend, reinstate, the cluster
|
||
dashboard, memberships, limits, a new password, delete, create; API keys —
|
||
issue (shown once), limits, revoke; organizations, teams and projects — add,
|
||
rename, move, remove, limits for the level and for each member, capacity
|
||
quotas, how long conversations and agent notes are kept.
|
||
- **Serve a model** (new view): GGUF models on Hugging Face — trending, most
|
||
downloaded, most liked or newest, or searched with / — then one of the
|
||
chosen repository's files, a name and a replica count, and it is fetched,
|
||
checked, added and started. The registry is mirror.registry's when set.
|
||
- **Do**: upgrade the cluster, restart every replica, back up now.
|
||
|
||
The people side runs the same commands as `llmgw users`, `keys` and `org`,
|
||
against the same store, so their checks and messages are the ones shown. A
|
||
sidebar taller than the terminal now scrolls with the current entry.
|
||
|
||
### New weights in a model's repository, found and moved to
|
||
|
||
A repository on Hugging Face can publish new weights under the same file
|
||
names — a fixed chat template, a better quantisation — and nothing said so. A
|
||
model served from the registry now records its origin: the repository, the
|
||
commit, and each file's checksum there; a model served before is worked out
|
||
from the mirror's record of where it fetched each file. Every
|
||
`mirror.update_check` (six hours by default; 0 turns it off) the control
|
||
plane asks whether those files changed. A new commit that leaves them alone
|
||
is not an update.
|
||
|
||
- `llmgw ctl model updates [--check]` lists what each repository has.
|
||
- `llmgw ctl model update MODEL` fetches the new files into the mirror beside
|
||
the old ones, checks them against the registry's checksums and the nodes'
|
||
llama.cpp, points the catalog at them, replaces the replicas one at a
|
||
time and asks one to say hello.
|
||
- `llmgw ctl model rollback MODEL` goes back to the files the update
|
||
replaced, which stay on the mirror.
|
||
- The console shows UPDATE on Models and offers all three on a model's row.
|
||
|
||
A model from `server.yaml` is reported but not changed: its source is edited
|
||
there.
|
||
|
||
### The monitor: the cluster drawn like btop
|
||
|
||
The console has a **Monitor** under Watch: CPU, memory, GPU busy and memory,
|
||
requests, tokens, time to first token, in flight and queued, network and
|
||
disk, as Braille charts filled and coloured by height, over 5 minutes, an
|
||
hour, a day or a week (1–4 or R); then each node's CPU, memory and network,
|
||
each card's busy, memory, heat and power, and each model's requests, as
|
||
sparklines. A node's and a model's own pages carry their charts too.
|
||
|
||
The history is read from the Prometheus in `monitoring.prometheus`, through
|
||
the admin API (`POST /api/v1/metrics/range`); without one, the charts draw
|
||
what the console has seen since it opened.
|
||
|
||
The control plane now lists the cluster's machines for Prometheus' service
|
||
discovery, at `/prometheus/targets` on its metrics port — nodes, itself and
|
||
the store's members, each labelled with its llmgw id — and `llmgw node add`
|
||
installs `prometheus-node-exporter` on the machine (`--no-node-exporter`
|
||
leaves it off). `deploy/monitoring/prometheus.yml` scrapes them as the
|
||
`nodes` job.
|
||
|
||
### Serving and updating run in the background
|
||
|
||
Serving a model or moving one to new weights from the console starts a job
|
||
on the control plane and leaves the console free at once, instead of
|
||
closing it to follow a download that can take an hour. A new **Jobs** view
|
||
shows each job's progress and steps, and the sidebar how many are running.
|
||
`llmgw ctl model serve --detach` and `llmgw ctl model update --detach` do the
|
||
same; `llmgw ctl jobs` lists the jobs and `llmgw ctl jobs ID` follows one.
|
||
|
||
### The file that suits the hardware, and how to spread it
|
||
|
||
`llmgw ctl model plan REPO` now plans the files the repository actually has
|
||
against the fleet as it is, before anything is downloaded: the model's shape
|
||
comes from one GGUF file's header, read from the registry, or from its
|
||
config.json. Every file is listed with the best way it runs here — one card;
|
||
several of a node's cards (llama.cpp splits by layer, vLLM by tensor
|
||
parallelism, dividing the KV heads); several nodes' cards over one fabric,
|
||
pipeline-parallel between them (vLLM); or a CPU node's memory — what each
|
||
card or node would hold and whether that hardware is free now. Best first:
|
||
on a GPU, the fewest cards, then the better quantisation up to eight bits;
|
||
whether the cards are free now only breaks a tie. Busy is not unfit: a model
|
||
for cards that are in use is added and starts when there is room, the job
|
||
saying so rather than failing. `--precision` keeps the old plan in the abstract.
|
||
|
||
`llmgw ctl model serve` without `--quant` takes the plan's file, and without
|
||
`--cpu` or `--gpu-class`/`--gpus`/`--nodes` the plan's way of running it —
|
||
a GGUF model on GPUs is now given a GPU profile rather than always the CPU.
|
||
`--fallbacks` writes the next best ways too, lower in priority, so placement
|
||
uses one of them when the first is busy. The console's Serve a model shows
|
||
the plan for each file, marks the best, and asks for the way and the
|
||
fallback.
|
||
|
||
### Models of several files: vision projectors, original weights
|
||
|
||
- A vision model's projector (`mmproj`) is fetched beside its GGUF weights
|
||
and loaded with them: `llmgw ctl model serve REPO --projector FILE`, or the
|
||
choice offered by the console. The model is marked as taking images. Nodes
|
||
load it once their agent is this version or later.
|
||
- `--format safetensors` serves a repository's original weights, with their
|
||
config.json, tokenizer and shard index, on vLLM across `--tp` GPUs of
|
||
`--gpu-class` (the fleet's one class when there is one). In the console,
|
||
**F** on Serve a model switches to them.
|
||
- The mirror keeps where it fetched each file beside it, and reports that
|
||
and the checksum after it restarts. The console's Mirror view shows one
|
||
row per version of a model, with its files on Enter.
|
||
- An update watches every file of a model, the projector and the tokenizer
|
||
included; one whose files the mirror cannot trace is found on the
|
||
registry by name, commit and size, or given with `llmgw ctl model origin`.
|
||
- `llmgw_model_update_available` is published, and `ModelUpdateAvailable`
|
||
alerts on it.
|
||
|
||
### The manual, in the binary
|
||
|
||
`llmgw docs` reads every document — installation, operation, the
|
||
configuration reference, the README, the changes — and a page for every
|
||
command, built from the commands' own help, like `info`: topics on the left,
|
||
T for a page's contents, / to find, n for the next match. Piped, it prints
|
||
the page. In the console it is **Help › Manual**, and **?** opens the new
|
||
page about the console itself from any screen.
|
||
|
||
## v0.11.1 — 2026-10-03
|
||
|
||
### The console's overview no longer spills over the screen
|
||
|
||
On a cluster with a GPU, `sudo llmgw` printed the overview's GPU table onto the
|
||
terminal under the console every three seconds, scrolling across the full
|
||
screen. The summary's tables were written to standard output whatever they
|
||
were given; they are now written where the summary is, which for the console
|
||
is its panel. The test cluster the console was tried on had no GPU, which is
|
||
why it was not seen there.
|
||
|
||
## v0.11.0 — 2026-10-03
|
||
|
||
A minor version: the store can be replicated and the control plane doubled,
|
||
both set up by llmgw itself; `sudo llmgw` opens setup screens and a console;
|
||
usage is read from sums kept as it is written. Released without candidates.
|
||
|
||
**Upgrading.** Migrations 0015 (usage sums, filled from the request table
|
||
already recorded — once, at the first start) and 0016 (control planes) run by
|
||
themselves. An agent older than this ignores the list of control planes and
|
||
keeps its configured one. Nothing in `server.yaml` or `agent.yaml` changes
|
||
meaning; `auth.usage_retention` and `router.tenant_label` are new and optional.
|
||
|
||
The GPU and fabric work below began from the field report in issue #2, after
|
||
a deployment on GPU nodes.
|
||
|
||
### GPU metrics that move, and charts of them
|
||
|
||
A GPU's utilisation and memory in use were read once, when the agent
|
||
started, and published from then on: a card busy since looked as idle as it
|
||
was at registration, and one idle since as busy. They are now sampled with
|
||
the temperatures and power every heartbeat (ten seconds at most), with the
|
||
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
|
||
An agent too old to send them still has its registration figures published,
|
||
and a figure the driver does not give is left out rather than shown as zero.
|
||
|
||
- **New series**: `llmgw_gpu_memory_utilization_percent`,
|
||
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
|
||
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
|
||
sample.
|
||
- **The node page** shows each card's utilisation, memory, clock and ECC
|
||
errors in its table, and charts of utilisation, memory, temperature and
|
||
power, one line per card.
|
||
- **The cluster page** charts the fleet's GPUs by class: average utilisation
|
||
and memory in use.
|
||
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
|
||
utilisation, memory, memory bandwidth, temperature, power against the
|
||
limit, clock, and what holds cards back.
|
||
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
|
||
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
|
||
- **The cluster page's GPU map**: every card a square, a row per node,
|
||
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
|
||
with what holds it and its figures under the pointer; free cards dashed.
|
||
- **Cards held and doing nothing**: the replicas whose every card reads under
|
||
5% busy, with when they were last asked — the first place to find room.
|
||
- **What each model gets from its cards**, on the cluster page and in
|
||
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
|
||
draw and average utilisation, over the last quarter hour.
|
||
- **What wears a card out**, read every minute: XID errors from the kernel's
|
||
log (counted from when the agent started), memory rows remapped or pages
|
||
retired and whether either waits for a reset, a card out of spare rows,
|
||
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
|
||
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
|
||
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
|
||
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
|
||
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
|
||
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
|
||
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
|
||
`GPUNVLinkErrorsClimbing`.
|
||
- **GPU time by owner** on the usage page: the cards each owner's models
|
||
held over the window, charted and summed into GPU-hours, whether or not the
|
||
models were asked anything — the time a GPU quota is about. From the
|
||
metrics store, as far back as it keeps history.
|
||
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
|
||
each card and whose quota it counts against, for joining the GPU series to
|
||
models and tenants.
|
||
|
||
### Usage is read from sums kept as it is written
|
||
|
||
Every chart and total — the usage tab, quotas, the dashboard, a person's own
|
||
page, and budgets restored at start — summed the request table again on each
|
||
view: a row per request, scanned whole. Usage is now also summed as it is
|
||
written, in the same transaction, into five-minute slices, hours and days per
|
||
key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
|
||
0015 fills them from what is already recorded).
|
||
|
||
- **Charts** read the coarsest of the three their slices are made of: a day in
|
||
20-minute slices from the five-minute table, a week or a month from hours.
|
||
- **Totals and budgets** read the rows up to the next five-minute slice, then
|
||
slices, hours and days — exact, not rounded to a slice. On three million
|
||
requests over thirty days, a month's total per person went from 705 ms to
|
||
18 ms on PostgreSQL.
|
||
- **Latency and time to first token** are kept as histograms in the sums, so a
|
||
percentile per team or model can be read after the rows are gone.
|
||
- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
|
||
that, hourly. Totals, charts and budgets are unchanged by it — they read the
|
||
sums — but `ctl request` finds nothing older. Unset keeps every record, as
|
||
before.
|
||
|
||
### Alerts per tenant and per budget
|
||
|
||
- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
|
||
and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
|
||
tokens and time to first token per tenant. The tenant is the level
|
||
`router.tenant_label` names — `team` by default, or `org`, `project`, or
|
||
`none` — named by its path, "acme/platform". One level rather than every
|
||
project, so a cluster of thousands of projects does not become a Prometheus
|
||
of millions of series.
|
||
- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
|
||
over the budget for every organization, team, project and person given one,
|
||
refreshed every minute.
|
||
- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
|
||
`BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
|
||
`TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
|
||
`TenantSlowFirstToken`.
|
||
- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
|
||
(uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
|
||
refused by a limit, first-token p95, every budget's share spent, and who used
|
||
most in the last day — with a tenant chooser at the top.
|
||
|
||
### How each team's requests fared, not only what they cost
|
||
|
||
- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
|
||
per organization, team, project, person or model, read from the histograms
|
||
kept with the usage sums: on the usage tab's table, and in a branch's or
|
||
person's panel on the people page, beside the error rate and requests
|
||
refused by a limit.
|
||
- **Refused requests are counted against their owner.** A request a rate,
|
||
token or budget limit refused was counted only in Prometheus, by model; a
|
||
team running into its limit looked like a team that had gone quiet. It is
|
||
now recorded as a 429 against its key, person and project — no tokens, no
|
||
cost — and shown as "Limited".
|
||
- **Usage by model** is a grouping of the usage tab, by the model that
|
||
answered.
|
||
### The console watches the cluster and holds every setting
|
||
|
||
`sudo llmgw` on a control plane is a console in three groups down the left:
|
||
|
||
- **Watch**, refreshed every three seconds: the overview; nodes, with CPU,
|
||
memory, GPUs free, disk and agent version, and a node's detail — load,
|
||
engines, each card's busy, memory, temperature and power; every GPU in the
|
||
fleet and which model holds it; models, with a model's profiles, warnings
|
||
and replicas; replicas and where they run; the last day's use by team,
|
||
project or organization; and a node's agent log.
|
||
- **Settings**: every key of `server.yaml`, block by block, with what applies
|
||
when the file sets nothing and what each key is for, from its comment in
|
||
the code. Enter changes a key or opens a block; lists and maps of blocks
|
||
(endpoints, models, prices) take entries added and removed. A save is read
|
||
by the same loader the service uses first — nothing is written that would
|
||
not load — keeps the file's comments, leaves the old one beside it, and
|
||
offers to restart the service.
|
||
- **Do**: add a node, replicate the store, leave.
|
||
|
||
On a node, `sudo llmgw` opens the same settings for its `agent.yaml`.
|
||
|
||
The keys come from the configuration structs themselves, through the
|
||
generator that writes `docs/reference.md`, so a setting added to the code is on
|
||
the screen without anyone listing it, and a test fails if the two drift.
|
||
That generator told blocks apart by name alone, so `tlsconf.Config`,
|
||
`engines.Config` and `supervisor.Config` were each taken for the agent's
|
||
`Config` and dropped: the reference had no TLS, engine or supervisor settings
|
||
at all. It now has 37 blocks, six more than before.
|
||
|
||
### A second control plane with `llmgw server add`, and the screens as windows
|
||
|
||
- **`sudo llmgw server add [user@]host [--vip 10.0.0.160/24]`**, run on a
|
||
control plane, makes another machine a standby: this llmgw copied over, a
|
||
certificate from the cluster's authority, the same configuration and store
|
||
address, the machine let into the store (for a store under Patroni, one
|
||
change through Patroni's API), and the service started. Every node is told
|
||
of it within half a minute and fails over to it untouched. `--vip` keeps a
|
||
floating address on whichever control plane leads, with keepalived on both.
|
||
Tried on new machines: switching the leader off, the standby led after 14
|
||
seconds and the floating address answered from it after 23.
|
||
- **Nodes learn every control plane** from the leader (a new `ControlPlanes`
|
||
message, kept in `/var/lib/llmgw-agent/control-planes.json`), after the
|
||
ones in their configuration.
|
||
- **A node notices a dead control plane within a quarter of a minute.** A
|
||
leader whose machine stopped dead left its nodes' streams open until the
|
||
kernel gave up retransmitting, about fifteen minutes; the agent now pings
|
||
the stream after ten quiet seconds, and bounds dialling too.
|
||
- **`GET /leader`**: 200 on the control plane that leads, 503 on a standby,
|
||
open like `/ready` — what keepalived follows.
|
||
- **The store's access rules live in Patroni's shared configuration** for a
|
||
cluster set up by `server init --patroni`, so another control plane is let
|
||
in without touching each member.
|
||
- **The screens are windows**: full screen, a title bar, the steps or actions
|
||
down the left, the content in a bordered panel, the keys along the bottom,
|
||
questions and errors as dialogs over it — the setup, the console, adding a
|
||
node and one-machine init alike. Keyboard only.
|
||
|
||
### Machines nothing may log into: `llmgw join`
|
||
|
||
For sites that allow no ssh into a machine, the setup screens offer a fourth
|
||
way to reach one: a join code. The setup listens on port 7444 while it waits
|
||
and shows a code for each such machine, with the commands that fetch llmgw
|
||
onto it; there, `sudo llmgw join <code>` dials out — nothing dials in —
|
||
checks by the certificate fingerprint in the code that it reached the setup
|
||
that made it, admits itself with the code's secret, and runs what the setup
|
||
sends, printing each step, until the setup says it is done. A code is good
|
||
for one machine and an hour, and is under 90 characters so it can be copied
|
||
whole from one screen into another.
|
||
|
||
To the setup a joined machine is one more way of running a script there:
|
||
the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one
|
||
machine joining by code and another logged into with a password, in one
|
||
setup.
|
||
|
||
### Setting up from nothing in screens: `sudo llmgw`
|
||
|
||
Run with nothing after it in a terminal, as root, `llmgw` opens full-screen
|
||
setup on a machine with nothing installed, and a console on a control plane.
|
||
In a pipe or a script it prints its help, as before.
|
||
|
||
- **Setup** asks one machine or three. For three — this one as the control
|
||
plane and etcd's third vote, two more for the store under Patroni — it asks
|
||
how to reach the other two: the ssh agent, a key file, or a password, and a
|
||
sudo password where sudo asks for one. Passwords stay in memory for the run;
|
||
a sudo password grants the user passwordless sudo until the install
|
||
finishes, and the grant is removed after, failed or not.
|
||
- **Every machine is checked before anything is installed**: login, system
|
||
(Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its
|
||
role, clock against this one, NTP, the ports its role needs, internet, and
|
||
whether it reaches the other machines. Failures stop it, with what to do;
|
||
warnings let it go on.
|
||
- **Then the plan, and the command line that does the same** from a script,
|
||
then the install with its progress, and the administrator's password.
|
||
- **The console** shows `ctl status`, refreshed every five seconds, and offers
|
||
to add a node or, for a store on one machine, to replicate it.
|
||
- Machines are reached with Go's ssh client rather than the system's, so a
|
||
password can be used; a host key is recorded on first sight and refused if it
|
||
changes.
|
||
|
||
Tried end to end on three new machines whose user logs in and sudoes only with
|
||
a password.
|
||
|
||
### A store that survives losing its machine
|
||
|
||
`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
|
||
under Patroni on two machines, its leader lock in a three-member etcd, and
|
||
llmgw given every member with `target_session_attrs=read-write`. Measured: a
|
||
planned switchover cost one failed lease renewal; switching off the primary's
|
||
machine promoted the standby in 24 seconds with the gateway serving
|
||
throughout. Example configurations are beside it.
|
||
|
||
**`llmgw server init --patroni user@host --patroni user@host`** sets it up:
|
||
PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's
|
||
third vote — the role and database, and the store already in use moved in.
|
||
Every step looks first; run against a cluster that is up, it changes nothing.
|
||
Tried from scratch on three new machines (two minutes, then a failover by
|
||
switching the primary off) and against the reference cluster (nothing
|
||
touched).
|
||
|
||
**Watching it**: for a store of several hosts the control plane publishes
|
||
`llmgw_store_member`, `llmgw_store_replica_lag_bytes` and
|
||
`llmgw_store_patroni_up` from each member's Patroni, a member that stopped
|
||
showing as `missing` rather than vanishing; `llmgw ctl status` has a STORE
|
||
line; and four alerts — `StoreHasNoLeader`, `StorePatroniUnreachable`,
|
||
`StoreReplicaNotStreaming`, `StoreReplicaBehind`.
|
||
|
||
**Making an administrator on PostgreSQL** (`init --admin`, `users promote`)
|
||
failed: the flag was sent as a bool to an integer column, which SQLite takes
|
||
and PostgreSQL refuses. Every existing PostgreSQL store got its administrator
|
||
by being copied from SQLite, which is why nobody had met it.
|
||
|
||
`llmgw server backup` reads a store address of several hosts. It took the
|
||
list for one host name and reached the database only by falling through to
|
||
the second; it now hands libpq the list, and dumps from a standby when there
|
||
is one.
|
||
|
||
### People, limits and usage are one page
|
||
|
||
Managing a team meant its members on the people page, its quota on the quotas
|
||
page and its spend on the usage page, each linking to the others. They are now
|
||
one page in three tabs — **Organizations & people**, **Limits**, **Usage** —
|
||
each at the address it had before, so links and bookmarks still land. The menu
|
||
has one entry for them, People.
|
||
|
||
Choosing an organization, team, project or person in the tree, or a person in
|
||
the account list, opens a panel beside it with everything about that one
|
||
thing: its own limits and what it is held to from above, spend against its
|
||
budget, its per-member limit, its capacity quota, and its requests, tokens
|
||
and spend over the last week, drawn. Its limits can be set or changed there,
|
||
and the answer comes back to the same panel.
|
||
|
||
The administrator's menu no longer has an Organizations entry beside it: `/org`
|
||
and `/org/<id>` send an administrator to that organization's panel here. The
|
||
page stays as it was for an organization's managers, who cannot see this one.
|
||
|
||
### The chat page says whether a GPU or a CPU answered
|
||
|
||
A model can have replicas on a card and on CPUs at once, and the two answer
|
||
at very different speeds; the chat page gave no way to tell which one a reply
|
||
came from. The picker now says where each model runs — `GPU`, `CPU` or
|
||
`GPU + CPU` — and the line under each reply names what answered it: `GPU
|
||
6000-24g` or `CPU`, beside its tokens a second.
|
||
|
||
Every answer through the gateway carries the same as `x-llmgw-hardware`:
|
||
`cpu`, or the GPU class. An endpoint configured by hand, with no pool, sends
|
||
none.
|
||
|
||
### A card shared with something else is judged every heartbeat
|
||
|
||
A card with another process on it when its agent started was kept out of
|
||
placement, and stayed out — "5014 MiB already allocated by another process" —
|
||
after that process let go of it, until the agent was restarted. Found on a
|
||
node shared with ComfyUI. The other way round, a tenant arriving after
|
||
registration was never seen, and a card the agent's own engine held when the
|
||
agent restarted read as someone else's.
|
||
|
||
The agent now reads every card's memory per process on each heartbeat,
|
||
leaves out what its own engines hold, and sends the rest; the control plane
|
||
keeps a card out of placement on that reading, and lets it back in when it
|
||
falls under 512 MiB. `llmgw_gpu_memory_used_bytes` reads the same figure. An
|
||
agent too old to send it keeps the reading from registration.
|
||
|
||
### A GPU placement no longer runs on the CPU unnoticed
|
||
|
||
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
|
||
where the loader looks), came up on the CPU, answered at CPU speed, and was
|
||
reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||
|
||
- **Installing** the CUDA build asks it which devices it sees
|
||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||
own `ldd` could not say it: the backend is opened at run time.
|
||
- **After the upgrade**, engines the agent finds already running are held to
|
||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||
is stopped, and its replacement starts through the check. The installed
|
||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||
sees no GPU.
|
||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||
included). One that does not is stopped and reported failed: "placed on GPU
|
||
0 but computing on the CPU", with the devices the engine lists. This covers
|
||
a hand-installed binary (`engines.llamacpp.bin`), which nothing else checks.
|
||
- **`llmgw ctl engine-check llamacpp`**, run on a GPU node, now also fails
|
||
when the engine sees no GPU, and otherwise prints the devices it computes on.
|
||
|
||
Shards are also given `CUDA_DEVICE_ORDER=PCI_BUS_ID`: placement numbers GPUs
|
||
as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||
node with two kinds of GPU, a shard could get a different card from the one it
|
||
was given.
|
||
|
||
### A model of many files, mirrored whole for vLLM
|
||
|
||
vLLM loads a directory — shards, config.json, the tokenizer — and does not
|
||
read GGUF. Mirroring one could go wrong several ways: the explorer offered the
|
||
shards alone, a node could be handed a version still downloading or missing a
|
||
shard, and a repository's page given as a model's source was fetched as one
|
||
file of HTML.
|
||
|
||
- **The explorer fetches a safetensors set with what the engine reads beside
|
||
it**: config.json, generation_config.json, the tokenizer files, the shard
|
||
index, the quantisation config. Never code: a `.py` would be the engine
|
||
running what the repository ships, which is to be chosen on purpose.
|
||
- **A version is offered to nodes only whole.** The mirror refuses its
|
||
manifest (409, with what is missing) while a file is still downloading, or
|
||
while a shard its `model.safetensors.index.json` names, or a part of a
|
||
split (`-00002-of-00004`), is absent. A node reports "not complete on the
|
||
mirror yet" and is tried again.
|
||
- **Checksums go with the files.** The mirror keeps each file's sha256 beside
|
||
it and the manifest carries it; a node checks every file of a set against
|
||
it before using the set, and fetches one that differs again.
|
||
- **`llmgw mirror add` takes several URLs** into one version.
|
||
- **A Hugging Face repository page given as a source** is read as the
|
||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||
itself (the node then needs to reach Hugging Face), and refused for one
|
||
llama.cpp runs, which needs a file.
|
||
|
||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||
such a model, as before.
|
||
|
||
### Signing up with an address that has an account
|
||
|
||
On PostgreSQL, signing up with an address already registered showed
|
||
"something went wrong; try again": the store recognised a duplicate by the
|
||
word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under
|
||
code 23505. It now knows both, and the address is answered as on SQLite —
|
||
with the same page as a new sign-up, so the page cannot be used to find out
|
||
who has an account. That page now also says what to do if the address had
|
||
one: sign in, or ask an operator to reset the password.
|
||
|
||
An error the page does not explain says the account was not created and
|
||
gives a reference (`err-…`), logged beside the error, for an operator to find
|
||
it; the database's own words stay in the log.
|
||
|
||
### An adapter no fabric uses does not keep a node out
|
||
|
||
`llmgw doctor` failed a node for a RoCE port with no RoCE v2 GID entry, or for
|
||
`ib_uverbs` not being loaded, whether or not any fabric used them; on a node
|
||
added for single-node inference, with no fabric given, that refused it for
|
||
RDMA nobody had asked for.
|
||
|
||
- Those are failures only for ports a declared fabric uses, and only when a
|
||
fabric is declared. Otherwise they are warnings saying nothing uses them yet.
|
||
- Which fabric an undeclared InfiniBand port is on, and whether it has a
|
||
subnet manager LID, are notes rather than warnings; `node add` still warns
|
||
once that adapters are up with no fabric given.
|
||
- `node add --fabric` hands its fabrics to doctor (`doctor --fabric
|
||
name:mlx5_0,…`), so they are checked as declared even though agent.yaml is
|
||
written afterwards.
|
||
|
||
### A restart with no room beside it
|
||
|
||
`ctl restart` places each replacement beside the replica it replaces. With
|
||
no room for it — one GPU, every card that fits held by the replica itself —
|
||
the old replica was stopped only after the drain grace period, five minutes
|
||
of "restart replacing" with nothing saying why, which read as a restart that
|
||
never happened; scaling to zero and back was the way out. And a replacement
|
||
that fit nowhere even with the old replica gone was stopped for all the same,
|
||
leaving the model with nothing. Now:
|
||
|
||
- **The replica is the only thing in the way**, found by placing the
|
||
replacement on a copy of the inventory without it: stopped at once, the
|
||
replacement taking its place.
|
||
- **The replacement fits nowhere even then**: the restart is given up and the
|
||
replica keeps serving. `ctl replicas` shows `restart refused:` and why.
|
||
- **Room being made** — a lower class preempted, an idle on-demand replica
|
||
stopping — is waited for, as before.
|
||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||
|
||
### Every GPU node gets engine builds it can run, by itself
|
||
|
||
Which build of an engine a node installs follows its hardware — the CPU
|
||
architecture, the driver, and the oldest GPU on it — from what each project
|
||
publishes and how it was compiled. Each engine has a list of builds, best
|
||
first; the node takes the first its hardware meets, and the first of those
|
||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||
about costs its download, and the next one is tried.
|
||
|
||
| | llama.cpp | vLLM |
|
||
|---|---|---|
|
||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||
| No NVIDIA GPU | the CPU build | none |
|
||
|
||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||
starts at 7.5.
|
||
|
||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||
column shows the version installed, and the installation's detail says why
|
||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||
vLLM is not placed on a V100 by accident.
|
||
- **The install computes on the GPU before it counts**: a small tensor
|
||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||
at once.
|
||
- **Whether an engine computes on its GPUs is read from every process
|
||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||
`CUDA_VISIBLE_DEVICES`.
|
||
|
||
### Engines per node, and nodes that do not run them all
|
||
|
||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||
and `--ignore-checks` passed over every other problem with it.
|
||
|
||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||
node joins without it and serves what it does run. `node add` checks the
|
||
vLLM the control plane installs, not the binary's own default.
|
||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||
advertised there, and not checked by doctor.
|
||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||
or labels a different version of an engine, or none — for exceptions; the
|
||
build each GPU generation needs is chosen without it (above).
|
||
- **The requirements follow the version**, as the table above sets out; the
|
||
threshold had been CUDA 13's for every version.
|
||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||
its own beside it, which was then used first.
|
||
- The node page shows each GPU's compute capability.
|
||
|
||
### Serving from the mirror says where a model will run
|
||
|
||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||
"Profile name" field, pre-filled "cpu", read as though renaming it would move
|
||
the model to a GPU. Placement is decided by `gpu_class`, which the form never
|
||
set. Now:
|
||
|
||
- **Runs on** lists the fleet's GPU classes with how many cards are free, and
|
||
for llama.cpp the CPU. On a fleet with GPUs the class with the most free
|
||
cards is chosen.
|
||
- **GPUs per replica** sets how many cards a replica takes on one node
|
||
(`gpus_per_node` for llama.cpp, `tp` for vLLM), and **parallel requests**
|
||
how many a llama.cpp replica serves at once.
|
||
- The profile is named for where it runs (`t4`, `t4-x4`, `cpu`) rather than
|
||
typed.
|
||
- A line under the form says what the model will run as — engine, CPU or how
|
||
many of which GPU, memory, context pool and how it is divided — and turns
|
||
red when the CPU is chosen with GPUs free. The message after adding repeats
|
||
it.
|
||
- Choosing the CPU while GPUs are free is refused unless **run on the CPU
|
||
anyway** is ticked.
|
||
|
||
### llama.cpp on several GPUs of one node
|
||
|
||
A llama.cpp profile takes several of a node's GPUs with `gpus_per_node: N`:
|
||
placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||
`nodes` above one, is refused.
|
||
|
||
## v0.10.6 — 2026-10-01
|
||
|
||
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
Tried on the PoC cluster before release: LoRA adapters asked for by name
|
||
through the gateway, two on one model, each answering as trained and the model
|
||
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
|
||
requests on the preferred replica, 4 on the other); and a bronze request
|
||
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
|
||
live: a model served from an external provider, which needs a provider key.
|
||
|
||
### Capacity from the admin pages
|
||
|
||
What v0.10.5 made settable from the command line is now on the pages too:
|
||
|
||
- **Quotas** has a Capacity section: every scope's guaranteed and most GPUs,
|
||
cores and memory beside what it holds, with a form to set, change or remove
|
||
one. It goes through the same checks as `llmgw ctl quota`.
|
||
- **A model's card** on the mirror page sets its class, its owner (when there
|
||
are quotas to count against), and the memory and cores a replica reserves,
|
||
beside its replica counts. Only what changed is applied, and each change is
|
||
said.
|
||
- **Room counts what a class may take.** The card, `llmgw ctl model target` and
|
||
the headroom API count the replicas a model could get by stopping those its
|
||
class outranks, separately from free room, and name whose they would be. A
|
||
critical model's card no longer refuses a replica that preemption would make
|
||
room for.
|
||
|
||
### LoRA adapters
|
||
|
||
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
|
||
nodes fetch them with the model's weights and llama.cpp loads them all at start,
|
||
applied to nothing (`--lora-scaled FILE:0`); a request that names an
|
||
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
|
||
so one replica serves the model and all its adapters. Adapter names share the
|
||
namespace of models, are listed in `/v1/models`, and follow key allowlists on
|
||
their own — a key allowed an adapter does not reach the model under it. Adapter
|
||
files are kept from weight cleanup like the model's own. Not yet for vLLM or
|
||
SGLang profiles, whose adapters are loaded differently.
|
||
|
||
### Finding and attaching adapters from the mirror page
|
||
|
||
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
|
||
held model's card, *find adapters on Hugging Face* — those made for the model
|
||
its repository was made from, since a GGUF repository is a quantisation and
|
||
adapters are made for the original.
|
||
- **Opening a repository** reads the first bytes of each GGUF file and marks it
|
||
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
|
||
the adapter merged in about as often as an adapter, and only the file says.
|
||
- **Held adapter files** are marked in the list, and a model's card lists its
|
||
adapters, offers the held adapter files of its architecture to attach under a
|
||
name and scale, and removes them. A file that is not an adapter, or is for
|
||
another architecture, is refused before it reaches the catalog.
|
||
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
|
||
lists and changes them from the command line, through
|
||
`POST /api/v1/models/{name}/adapters`, with the same checks.
|
||
|
||
### Models served outside the cluster
|
||
|
||
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
|
||
model, api_key_file}` and no profiles. Nothing is placed for it; the router
|
||
sends its requests to the provider, rewritten to the provider's name for the
|
||
model and with its key, read from the file on every publish so a rotated key is
|
||
picked up. Named in another model's `fallbacks`, it is used only when the
|
||
cluster has no capacity left. It is never scraped for metrics, and never reads
|
||
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
|
||
model — and since its prompts leave the cluster, the allowlist is how to decide
|
||
who may send them (docs/install.md). `ctl models` and the cluster page mark it
|
||
external.
|
||
|
||
### Pool spillover
|
||
|
||
A model served from pools of different priority — H100 before A100, say — was
|
||
routed to the preferred pool however full it was. Now a pool whose replicas are
|
||
saturated, by the same measure admission uses (queue past its healthy depth, or
|
||
KV cache past its ceiling), passes requests to the next pool that is not. When
|
||
every pool is saturated the preferred one is kept: a pool that is just as full
|
||
gains nothing.
|
||
|
||
### Displacement: a higher band is not refused while a lower one waits
|
||
|
||
Each band's queue was bounded, and nothing bounded them together, so a full
|
||
gateway turned an interactive request away while batch requests sat queued
|
||
behind it. `admission.queue_capacity_total` now bounds everything queued —
|
||
by default half of every band's capacity together. Reached, a request is not
|
||
refused while a lower band has any queued: the newest request of the lowest
|
||
band is turned away instead, with a 503 and `Retry-After: 5`, so it backs off
|
||
and returns. The newest, because the oldest have waited longest and are
|
||
nearest their TTL. Counted as `displaced` in the admission rejection metric.
|
||
|
||
### Releases can be signed
|
||
|
||
`llmgw release keygen` makes an ed25519 key pair. With the private key in the
|
||
CI secret `RELEASE_SIGNING_KEY`, releases carry `checksums.txt.sig`; with the
|
||
public key in `internal/release/trusted.pub`, every binary built after refuses
|
||
a release not signed with it, wherever it installs one. Off until then, and
|
||
docs/install.md says in which order to turn it on.
|
||
|
||
### Restarts borrow a lower class's room
|
||
|
||
A restart places the replacement beside the replica it replaces, and where
|
||
there is no free room it now makes some the way any placement does: by
|
||
stopping a replica its class outranks, which gets its room back once the old
|
||
replica has gone. The in-place restart — the model unserved while its
|
||
replacement loads — is left for when nothing of a lower class is running. On a
|
||
cluster where every model is standard, that is every full one: mark the models
|
||
that can wait `opportunistic` (`llmgw ctl model class`, or the model's card).
|
||
|
||
## v0.10.5 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
|
||
format check and has no binaries.
|
||
|
||
After upgrading:
|
||
|
||
- **Declare what CPU models need.** A CPU model added from now on must say its
|
||
memory; one stored before this still loads and is flagged in `ctl models`
|
||
until `llmgw ctl model resources MODEL --memory GB --cores N` is run. Cores
|
||
are then reserved per replica and become the engine's thread count, for
|
||
replicas started after the change: `llmgw ctl restart --model MODEL`.
|
||
- **Upgrade the mirror on its own host**, now with one command:
|
||
`sudo llmgw upgrade v0.10.5 --service llmgw-mirror` (by hand once, for a
|
||
mirror older than this). `llmgw ctl upgrade` says so at the end.
|
||
|
||
### CPU cores are reserved, and memory has to be declared
|
||
|
||
The first step of resource allocation for larger clusters: what a replica
|
||
holds of a node is reserved, not assumed.
|
||
|
||
- **Cores.** A profile's `cpu_cores` is reserved out of its node's CPUs, as
|
||
memory is, and is what its engine computes on (llama.cpp's `--threads`). A
|
||
node holds as many CPU replicas as it has cores for, and two on one node no
|
||
longer spin against each other. Agents report their CPU count; a node that
|
||
does not, and a profile that declares no cores, constrain nothing.
|
||
- **Memory is required of new CPU models.** A model with no GPU is all host
|
||
memory, and one declaring none looked to placement as if it fitted
|
||
everywhere, any number of times. Adding one without `min_memory_gb` is now
|
||
refused. Models stored before this still load, and say what is missing.
|
||
- **Declared against measured.** Each model shows what one replica reserves
|
||
and the most one of its replicas uses — in `ctl models` (RESERVES) and on the
|
||
cluster page — with a warning when nothing is declared or when it uses more
|
||
than a tenth over what it reserves.
|
||
- **Setting them.** `llmgw ctl model resources MODEL --memory GB --cores N`
|
||
(and `POST /api/v1/models/{name}/resources`) for a model added at runtime.
|
||
`ctl model serve` and the mirror's serve form fill in cores — half the
|
||
smallest node the model fits on — beside the memory they already estimate.
|
||
Running replicas keep their old figures until restarted.
|
||
|
||
The commands are `llmgw ctl model resources` and `llmgw ctl model class`; in
|
||
rc.7 the first was briefly `llmgw ctl resources`.
|
||
|
||
### Capacity classes, and preemption
|
||
|
||
A model has a class — `critical`, `standard` or `opportunistic` — saying what
|
||
its capacity is worth against another's when there is not enough for both.
|
||
Unset, an on-demand model is opportunistic and every other model standard,
|
||
which is how the catalog behaved before. `llmgw ctl model class MODEL CLASS`
|
||
sets it.
|
||
|
||
When a model cannot be placed, replicas are stopped to make room for it, by
|
||
these rules:
|
||
|
||
- A lower class gives way to a higher one.
|
||
- Within a class, only replicas above a model's minimum — room it grew into
|
||
under load — and only for a model still below its own minimum.
|
||
- A critical model's minimum is never taken, nor a replica that has only just
|
||
loaded, nor one spanning several nodes.
|
||
- Idle on-demand replicas go first, then what lower classes grew into, then
|
||
the same within the class, then lower classes' minimums; the longest unused
|
||
first within each, and of equally idle ones the smallest.
|
||
- The set is tried on a copy of the inventory first, at most three replicas,
|
||
and nothing is stopped unless it makes the room. Of a set that does, each
|
||
replica whose stopping turns out not to be needed is left running, so an
|
||
idle replica tried first on the wrong node is not stopped with the one that
|
||
made the room.
|
||
|
||
A replica stopped this way is drained and given 30 seconds to finish what it
|
||
is serving. Its model is held back for up to five minutes, or until the model
|
||
it gave way to has its replicas, so it does not take the room straight back.
|
||
`ctl models` shows each model's class and "gave room to X" while held; the
|
||
cluster page shows the class and how many replicas were lost; the metric is
|
||
`llmgw_model_preempted_total{model,class}`, and every preemption is logged.
|
||
|
||
### Capacity quotas for the tenancy tree
|
||
|
||
An organization, team or project can be guaranteed GPUs, CPU cores and
|
||
memory, and limited to at most some amount of each. What it holds is what the
|
||
replicas of the models it owns hold — and those of the teams and projects
|
||
beneath it. `llmgw ctl model owner MODEL team:Acme/ml` says whose a model is;
|
||
one with no owner is shared and counts against nobody.
|
||
|
||
- **A guarantee is kept.** A replica within its owner's guarantee is never
|
||
taken for another scope's model, whatever the classes: a guarantee a higher
|
||
class elsewhere could take would not be one.
|
||
- **Beyond it is borrowing.** Free room is used; when a scope within its own
|
||
guarantee needs room, what others hold beyond theirs is taken back first,
|
||
whatever their class — except a critical model's minimum.
|
||
- **A limit is a ceiling.** A model whose owner would go past a limit anywhere
|
||
up the tree gets no further replica, and says which limit; headroom and the
|
||
mirror card stop there too.
|
||
|
||
`llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16` (also `--cores`,
|
||
`--memory` and their limits), `llmgw ctl quota list` with what each scope holds,
|
||
`llmgw ctl quota clear`. Scopes are named by path or ID; a name that could mean
|
||
two is refused. Changes take effect within half a minute. Metrics:
|
||
`llmgw_capacity_holds`, `llmgw_capacity_guarantee` and `llmgw_capacity_limit`
|
||
by scope and resource; alert: CapacityAtLimit.
|
||
|
||
### Sized by what a replica was measured to serve
|
||
|
||
What `llmgw ctl bench MODEL --save` measured — the most requests at once one
|
||
replica served while meeting its SLO, and its throughput there — now drives
|
||
decisions rather than only being printed:
|
||
|
||
- **Scaling up** when requests per replica pass what was measured, not only
|
||
when the engines' queues fill: an engine takes requests into its slots long
|
||
after latency has gone.
|
||
- **Scaling down** only when the replicas left would stay under 70% of what
|
||
they were measured to hold, so a replica taken back does not put the rest
|
||
straight over.
|
||
- **Choosing a profile.** Among profiles of equal priority, the one measured
|
||
to serve the most per GPU — or per core — is tried first. Priority set by an
|
||
operator still wins.
|
||
- **Sizing.** `llmgw ctl model target MODEL --tokens-per-sec N` says how many
|
||
replicas that takes, how many run, and whether the rest fit, with the reason
|
||
when they do not; `--apply` sets it if they do. Headroom reports what one
|
||
more replica would serve.
|
||
|
||
A model never measured behaves exactly as before. Measurements are read from
|
||
the store every half minute.
|
||
|
||
### Upgrading a machine the control plane does not reach
|
||
|
||
`sudo llmgw upgrade VERSION --service UNIT` upgrades the machine it runs on —
|
||
the mirror's host above all, whose service runs unprivileged and cannot replace
|
||
its own binary as an agent does. The release's checksums are checked, the old
|
||
binary kept, the unit restarted and watched until it stays up.
|
||
|
||
### Nodes report their own heat and power
|
||
|
||
The first step of facility-aware placement (docs/facility-aware-placement.md),
|
||
observe-only: nothing is decided on these yet. Every agent reads, at most every
|
||
ten seconds, and sends with its heartbeat:
|
||
|
||
- **GPUs** from nvidia-smi: temperature, HBM temperature where the driver has
|
||
it, power draw and the enforced limit, and what is holding the clocks down —
|
||
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
|
||
at registration; it is now current.
|
||
- **CPUs** from hwmon, which is what lm_sensors reads, with nothing to install:
|
||
the package temperature, and the board's inlet air sensor where it has one.
|
||
- **CPU power** from RAPL's energy counters.
|
||
|
||
A sensor a machine does not have is not reported, never shown as zero. New
|
||
metrics: `llmgw_gpu_power_watts`, `llmgw_gpu_power_limit_watts`,
|
||
`llmgw_gpu_memory_temperature_celsius`, `llmgw_gpu_throttled{reason}`,
|
||
`llmgw_node_cpu_temperature_celsius`, `llmgw_node_cpu_power_watts`,
|
||
`llmgw_node_inlet_temperature_celsius`. Each node's page shows them, with a GPU
|
||
table of temperature, power against its limit, and what is throttling it.
|
||
|
||
### The mirror page in tabs
|
||
|
||
/admin/mirror was one long page: what the mirror holds, the Hugging Face
|
||
explorer and the fetch form, one under another, with every visit waiting on
|
||
Hugging Face for a list it might not show. It is now three tabs — Held (with
|
||
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
|
||
URL — each with its own address. Only the tab shown does its work, and only
|
||
Held reloads while something downloads. A download started from the explorer
|
||
links to Held, where its progress is; a fetch the mirror refused comes back on
|
||
the form it was sent from.
|
||
|
||
### A held model opens to where it runs, and how many
|
||
|
||
Clicking a model under Held opens a card beneath it: each replica with its
|
||
node, state, age and the CPU and memory its engine uses, and how many more
|
||
replicas the cluster has room for now, on which nodes, and why not more. The
|
||
replicas it always runs and the most it grows to under load are changed there.
|
||
Room is found by trying the scheduler itself on a copy of the inventory — the
|
||
same decisions the next passes would make, not an estimate beside them — and a
|
||
number of always-running replicas it cannot place is refused with the reason,
|
||
rather than accepted and left short. A ceiling above the room is taken, and
|
||
said. The same answer is at `GET /api/v1/models/{name}/headroom` on the admin
|
||
API.
|
||
|
||
Each row also says how its model runs without being opened: replicas ready of
|
||
those it should have, how many are still starting, and the nodes they are on —
|
||
in amber when short, in red when parked or when it does not fit. It is one line
|
||
per model beside its mode, and nodes drop the part of the name every node
|
||
shares (node05 for llmgw-node05, the full name on hover).
|
||
|
||
### One line per record, in every table
|
||
|
||
The same rule across the admin pages. Headers do not wrap; a record is one
|
||
line; free text is cut to its column with the whole of it on hover rather than
|
||
given a row of its own; a row across the whole table is only a note or an open
|
||
form. On the cluster page that means a node is one line (its address and logs
|
||
beside its name, not under it), a model's description is a column rather than a
|
||
second row, node names drop the prefix they all share, and the GPUs, classes,
|
||
fabrics and fallbacks columns appear only when some row has something in them —
|
||
on a CPU-only cluster they were three columns of dashes pushing the table past
|
||
the page. On People, a person in many teams shows two and a count, the rest on
|
||
hover.
|
||
|
||
## v0.10.4 — 2026-09-30
|
||
|
||
Published first as release candidates v0.10.4-rc.1 to rc.9.
|
||
|
||
After upgrading, run `llmgw ctl restart --all` once: CPU replicas take the new
|
||
thread count below only when they start.
|
||
|
||
### CPU replicas no longer fight over the CPUs
|
||
|
||
llama.cpp computes on every CPU of the machine unless told otherwise, and the
|
||
agent did not tell it. Two busy CPU replicas on one node then spin against each
|
||
other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to
|
||
0.8 each with both busy, in two rounds of the same test. The agent now starts a
|
||
CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy
|
||
make 5.5 each. `llamacpp.threads` in the agent's configuration sets it for a
|
||
node — all of them suits a node that only ever runs one CPU replica — and a
|
||
profile's own `--threads` still wins. Measured on VMs whose host is
|
||
oversubscribed; the proportions, not the numbers, are the point.
|
||
|
||
### Tracing
|
||
|
||
With `tracing.endpoint` set (or `OTEL_EXPORTER_OTLP_ENDPOINT`), each inference
|
||
request is traced over OTLP/HTTP: a span for the request, carrying its
|
||
`X-Request-Id`, with admission, scheduling and each attempt at an engine under
|
||
it. A caller's own `traceparent` is continued, and the engine is sent one for
|
||
its attempt. `tracing.sample_ratio` traces a share of the requests that arrive
|
||
without a decision. Off, it costs nothing.
|
||
|
||
### Restarting replicas
|
||
|
||
`llmgw ctl restart REPLICA... | --model M | --all` replaces replicas with new
|
||
ones. Each replacement is started beside the replica it replaces, which keeps
|
||
serving until the new one is ready; replicas of one model go one at a time.
|
||
Where no second copy fits anywhere, the old replica is stopped after the drain
|
||
grace period and the new one starts in its place. This is how running engines
|
||
pick up the keys introduced in v0.10.3, an edited profile, or a new engine
|
||
version.
|
||
|
||
### A model that keeps failing is parked, and everybody is told
|
||
|
||
A model whose replicas failed at every start was retried forever: kev-0.5b,
|
||
whose architecture the nodes' llama.cpp did not know, was started 155 times in
|
||
eight hours. After five failures in a row a model is now parked — not placed
|
||
again until its definition changes or `llmgw ctl retry MODEL` is run — and
|
||
`ctl status`, `ctl models` and the admin API say so with the last error.
|
||
|
||
A request for a model that will not come up — parked, or fitting on no node —
|
||
is answered at once with the reason, instead of "the model is loading; retry
|
||
shortly", which a client would retry forever. Each request still makes the
|
||
control plane try placing it afresh, so a model that fits once room is made is
|
||
not refused on an old answer. "no placement for model" is logged when its
|
||
reason changes or every ten minutes, not on every pass: it was 250 identical
|
||
lines in two days for one model.
|
||
|
||
### Metrics and alerts for what goes wrong
|
||
|
||
New series: `llmgw_model_replicas_desired`, `llmgw_model_start_failures`,
|
||
`llmgw_model_parked`, `llmgw_model_unplaceable`, `llmgw_node_engine_install`,
|
||
`llmgw_control_plane_leader` and `llmgw_certificate_expiry_timestamp_seconds`.
|
||
`llmgw_model_replicas_ready` is now published at 0 for every model that should
|
||
be running; it was published only while a replica was ready, so
|
||
ModelHasNothingServing could never fire. New alerts: ModelParked,
|
||
ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed,
|
||
ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.
|
||
|
||
`ctl status` counts a model against the replicas it should have now rather
|
||
than its minimum, so an on-demand model that was asked for and fitted nowhere
|
||
no longer reads as satisfied.
|
||
|
||
### Backups
|
||
|
||
`llmgw server backup` writes the store and `/etc/llmgw` — configuration,
|
||
secrets, and the certificate authority's key — to one archive readable by root
|
||
only, keeping the newest 14. `deploy/systemd/llmgw-backup.timer` runs it
|
||
nightly. The control plane publishes the newest archive's time as
|
||
`llmgw_backup_last_success_timestamp_seconds`, and BackupTooOld fires when it is
|
||
more than 26 hours old or there has never been one. docs/install.md now says how
|
||
to restore, including why running engines survive it: the engine-key secret
|
||
comes back with the store.
|
||
|
||
### Replicas follow load between min and max
|
||
|
||
A model's `max` was a ceiling nothing reached: every model ran exactly its
|
||
minimum however long its queue. Now, when requests are waiting at the engines —
|
||
every slot of every replica busy — at one or more per replica for a minute, one
|
||
replica is added, up to `max`; ten minutes with nothing waiting takes one of
|
||
those away, down to `min`. One step at a time, and not while the previous one
|
||
is still starting, so a spike that is over before a replica loads does not
|
||
leave several behind. Each decision is logged with the load that made it.
|
||
`llmgw ctl scale MODEL --min 1 --max 3` turns it on for a model.
|
||
|
||
### A model from the registry in one command
|
||
|
||
`llmgw ctl model serve OWNER/REPO` takes a model from the registry to a replica
|
||
that answers. It picks the GGUF file (`--quant`, Q4_K_M by default), has the
|
||
mirror fetch it and checks what arrived against the registry's published
|
||
sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its
|
||
header, adds it to the catalog, waits for a replica — stopping early with the
|
||
reason if it fits nowhere or keeps failing — and asks it to say hello. Each step
|
||
is shown as it happens. It runs in the control plane as a job, so a download of
|
||
hours does not depend on a terminal staying open. `--on-demand` adds it without
|
||
starting a replica.
|
||
|
||
### Upgrading from one command
|
||
|
||
`sudo llmgw ctl upgrade VERSION`, run on the control plane, upgrades that
|
||
machine and then every node in turn. Each agent is told the version over its
|
||
existing stream, fetches the release for its own platform, checks it against
|
||
the release's checksums, keeps the old binary beside the new and restarts,
|
||
while its engines keep serving. A node that does not come back on the new
|
||
version, ready, with its replicas, stops the upgrade before the next one.
|
||
Running it again resumes. Agents from before this have to be moved by hand
|
||
once.
|
||
|
||
### Following a request
|
||
|
||
Every response carries `X-Request-Id` — the client's own when it sends a plain
|
||
one, otherwise one the gateway makes — and the engine receives it too. One
|
||
access log line per request records it with the model asked for and served,
|
||
the replica, status, time queued, time to the first token, total time,
|
||
tokens, and the key, project and user. Usage records keep the id, replica,
|
||
queue time and time to first token, and `llmgw ctl request ID` reads one back:
|
||
the answer to "why was this slow?".
|
||
|
||
### Rate-limit headers
|
||
|
||
Responses carry OpenAI's `x-ratelimit-limit-*`, `x-ratelimit-remaining-*` and
|
||
`x-ratelimit-reset-*` for requests and tokens, from whichever level of the
|
||
tenancy tree will refuse first, so SDKs and editors can slow down before they
|
||
are refused.
|
||
|
||
### Memory sized from the model
|
||
|
||
The mirror's serve form sizes a llama.cpp model's memory from its GGUF header:
|
||
weights, plus the KV cache for the context pool from its layers, KV heads and
|
||
head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB
|
||
to 5.25 GB across the models on the live mirror, which the old "weights plus
|
||
half again plus two" could not see.
|
||
|
||
### Security
|
||
|
||
- **Alertmanager answered anyone on the network**, including requests to
|
||
silence alerts. The monitoring README now binds it to loopback, where
|
||
Prometheus beside it is its only client.
|
||
- **Login and signup could be made to use gigabytes.** Each attempt hashes with
|
||
64 MiB of argon2, with no bound on how many ran at once, and the per-address
|
||
throttle grew by one entry per address tried. Hashes now run a few at a time
|
||
and the rest wait; the throttle expires old entries, and when it is still
|
||
full it refuses new addresses rather than dropping live ones — which would
|
||
have let a flood of made-up addresses reset the throttle on a real one.
|
||
- **Any key holder could grow Prometheus without bound** by asking for made-up
|
||
model names, each of which became a metric label. Requests are counted under
|
||
the catalog's name for their model, or "unknown".
|
||
- **A SQLite backup left a full copy of the store readable** by local users
|
||
while it ran. The snapshot is now made in a private directory, and
|
||
llmgw-backup.service sets `UMask=0077`.
|
||
- **An engine archive could write outside its directory through chained
|
||
links.** Archives are unpacked through `os.Root`, and a link may only point
|
||
at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already
|
||
follows that rule.
|
||
- **Onboarding made every node trust the cluster's root for everything.** It
|
||
was added to the system store, so a certificate the control plane signed
|
||
would pass for any site with every TLS client on the node. The agent trusts
|
||
it through its own configuration only; onboarding no longer adds it, and
|
||
takes it out of a node that an older version onboarded.
|
||
- **Any node could manage a mirror running mutual TLS**, since one root signs
|
||
every node's certificate. `llmgw mirror serve --api-client NAME` answers only
|
||
the certificate with that common name, and the mirror warns when mutual TLS
|
||
runs without one.
|
||
- **The administrator's password stayed in `init.log`.** It is shown once, on
|
||
the screen, and the log keeps a placeholder.
|
||
- **Auth off on a public address was silent.** The server now warns at start
|
||
when the API listens beyond loopback without keys.
|
||
- **The actor recorded with a change was whatever the caller sent**, line
|
||
breaks included. It is cut to a short printable name, and `sudo llmgw ctl`
|
||
records the person who ran sudo rather than root.
|
||
- **vLLM's Ray head ran its dashboard**, whose job API runs code for anyone who
|
||
reaches it. It is off. The head itself still has no authentication, and
|
||
SGLang still takes its key as a flag; docs/install.md says what to do about
|
||
both. The monitoring README says how to keep Prometheus's queries to the
|
||
control plane.
|
||
|
||
### Fixes
|
||
|
||
- Tests that slept a fixed time and then asserted now wait for their
|
||
condition. They are the likeliest cause of CI's intermittent failures on a
|
||
loaded runner.
|
||
- `ctl` reported a change the API had queued (202 Accepted) as an error, as
|
||
`ctl restart` did on its first real use.
|
||
- The status each failure is answered with — 499, 502, 503 with and without
|
||
Retry-After, 504, 429 — is now tested case by case.
|
||
|
||
### Releases
|
||
|
||
Small fixes are published as release candidates of the next version —
|
||
v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release
|
||
notes come from this section, and they are marked as pre-releases. Onboarding
|
||
treats a release candidate as a release and downloads it.
|
||
|
||
## v0.10.3 — 2026-09-30
|
||
|
||
Security fixes from a review of v0.10.2. Three of them were confirmed on the
|
||
live cluster before being fixed.
|
||
|
||
### Engines demand a key only the router holds
|
||
|
||
Engines listened on every address of their node with no key. A request from a
|
||
neighbouring node reached the model directly, past the gateway's
|
||
authentication, quotas and accounting. Each replica's engine is now started
|
||
with a key derived from the replica id and a secret in the store. The router,
|
||
its metrics scrape, the agent's readiness check and `ctl bench` send it.
|
||
llama.cpp and vLLM receive it through their environment, not their command
|
||
line. Engines started before this keep running without a key until their
|
||
replica is next started.
|
||
|
||
### The mirror's management API answers only who it was told to
|
||
|
||
Without `--api-allow`, the management API answered anyone who could reach it,
|
||
including every node. It now answers this machine only; the control plane is
|
||
let in with `--api-allow` or mutual TLS. **Upgrading a mirror set up by hand:
|
||
add `--api-allow <control plane address>` to its unit.** Downloads started
|
||
through the API go to http and https only, and never to loopback, private or
|
||
link-local addresses (`--allow-private-sources` to permit that). Model,
|
||
version and file names can no longer be `..` or start with a dot.
|
||
|
||
### The admin listener refuses what a browser page could send
|
||
|
||
The admin API has no login and trusts whoever reaches it. A page open in an
|
||
operator's browser, reaching it over an ssh tunnel, could post a model
|
||
definition. The listener now requires Host to be an address or localhost,
|
||
refuses cross-site requests, and takes changes only as `application/json`.
|
||
A profile's `env` may no longer set variables that make an engine run other
|
||
code (`LD_*`, `PATH`, `PYTHON*` start-up variables, `BASH_ENV` and similar).
|
||
They are refused where a model is defined and dropped again before an engine
|
||
starts.
|
||
|
||
### Fixes
|
||
|
||
- The agent installs only engine versions that look like one, whatever the
|
||
control plane sent; a version of `../x` could have installed outside
|
||
`/opt/llmgw/engines`.
|
||
|
||
## v0.10.2 — 2026-09-29
|
||
|
||
- After `server init`'s screen closes, only the administrator's password and
|
||
the next steps are printed, not the whole run again under the screen that
|
||
just showed it. The whole run is kept in `/var/lib/llmgw/init.log`, root's
|
||
only, and printed in full when init fails.
|
||
- The screens' spinner and text field are llmgw's own rather than from
|
||
bubbles, one dependency fewer. Every llmgw command still asks the terminal
|
||
for its colours as it starts, which bubbletea v1 does on purpose; a terminal
|
||
that never answers waits for its timeout.
|
||
|
||
## v0.10.1 — 2026-09-29
|
||
|
||
### `llmgw ctl fabric-test`: two nodes, rail by rail
|
||
|
||
`llmgw doctor` checks a node on its own, and every check can pass while two
|
||
nodes still cannot reach each other: a cable into the wrong switch, a
|
||
partition key, a RoCE network with no route. `llmgw ctl fabric-test A B` runs
|
||
`ib_write_bw` from each of A's adapters on a fabric to the same adapter on B,
|
||
one rail at a time, through the agents — no SSH between nodes, and perftest is
|
||
installed the first time — and grades each against its link's rate: ok at 80%,
|
||
slow at 50%, fail below or when nothing got through.
|
||
|
||
Agents now say which requests they answer beyond the usual (`features`), so
|
||
one too old for the test is refused at once instead of waited for.
|
||
|
||
Written against perftest's real output and tested with it, but not yet run
|
||
over real InfiniBand or RoCE: this cluster has neither.
|
||
|
||
### The mirror on its own machine, in two commands
|
||
|
||
`llmgw mirror init --control-plane <address>` sets a machine up as the model
|
||
mirror: the user, `/srv/models`, and a service serving files to every node on
|
||
:8081 and the management API on :8082. `server init --mirror
|
||
http://<mirror>:8082` uses it instead of running one beside the control plane.
|
||
It took a unit written by hand and an edit to `server.yaml` before.
|
||
|
||
The management API can fetch and delete models; `mirror serve --api-allow`
|
||
makes it answer only the addresses given and the machine itself, and `mirror
|
||
init` gives it the control plane's. With `--no-node`, the starter model is
|
||
fetched and catalogued anyway, unpinned, and starts on the first node added.
|
||
|
||
Tried on three empty VMs: mirror, control plane and a node, and a request
|
||
answered by the starter model on the node.
|
||
|
||
### `server init` asks, and `node add` takes several nodes
|
||
|
||
- **`sudo llmgw server init`** in a terminal, without `--admin`, asks for the
|
||
administrator's email, the other names nodes will reach the machine by
|
||
(showing the addresses it already found), SQLite or PostgreSQL, and whether
|
||
to run the mirror, a node and the starter model here. Then its steps fill
|
||
in as a list, downloads updating in place, and the administrator's password
|
||
stays in a box until the end — and is printed again after, with the whole
|
||
run, so it is in the terminal's history. The last screen says how to add a
|
||
node and to back up the authority's key. `--plain`, or no terminal, prints
|
||
lines as before.
|
||
- **`llmgw node add a b c`** adds each in turn and ends with which were added.
|
||
A fabric named for one node's adapters is recognised on the next by what the
|
||
adapters report, and not asked again.
|
||
|
||
Both tried on empty VMs: init from the form to a starter model answering, and
|
||
two nodes added to it in one command in 40 seconds.
|
||
|
||
## v0.10.0 — 2026-09-29
|
||
|
||
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
|
||
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
|
||
installed by the nodes themselves; people in many places at once; and a chat
|
||
page, editor setup and a model explorer for the people using it.
|
||
|
||
### Independent InfiniBand fabrics, told apart and drawn
|
||
|
||
A fabric was whatever `agent.yaml` called it, and nothing checked. At a site
|
||
with several independent fabrics, two given one name would have had a replica
|
||
spread across both — started, then hung on its first collective. Now:
|
||
|
||
- **Identity.** Each active InfiniBand port reports the master subnet manager
|
||
it sees (`sminfo`) and every subnet manager on its fabric (`saquery SMIR`),
|
||
asked once a minute through one port per fabric. Ports whose managers are
|
||
listed together are one physical fabric, so a failover does not split one in
|
||
two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent
|
||
installs `infiniband-diags` where there are adapters and no `sminfo`.
|
||
- **Placement** divides a declared fabric by that identity and forms a replica
|
||
on one part only. A node whose adapters for one name are cabled to two
|
||
fabrics is refused; a node that cannot say is kept apart from those that can.
|
||
- **Subnet managers.** The cluster page lists each fabric's master, its
|
||
priority, its standbys and its master changes over the last day, and warns
|
||
when a fabric has no standby, when its ports disagree on the master, when a
|
||
port is up with no subnet manager (INIT), and when one name covers two
|
||
fabrics or two names one fabric.
|
||
- **Ports while running.** The agent re-reads every InfiniBand and RoCE port on
|
||
each heartbeat: state, physical state, rate, error and data counters. A port
|
||
going down takes its fabric off the node within seconds, and back when it
|
||
returns — it used to wait for the agent to restart.
|
||
- **Maps.** Every five minutes one node on each fabric
|
||
runs `ibnetdiscover`, and the control plane parses it. `/admin/fabrics` draws
|
||
the fabric: switches in their tiers, adapters under their leaves, cables
|
||
coloured by rate, degraded ones — trained below the rest of their kind —
|
||
marked in red, rail alignment, and the cluster's own adapters live, with
|
||
their traffic running along the cables. It zooms, pans, highlights what a
|
||
switch or adapter touches, and opens detail on a click. Placement prefers
|
||
the nodes of one leaf and puts nodes with a degraded cable last.
|
||
|
||
### NVLink, on every build and across nodes
|
||
|
||
- **The release binary sees NVLink.** It is built without cgo, so without NVML,
|
||
which was the only source of NVLink peers: on every NVLink machine the
|
||
portable agent reported none. Discovery now reads `nvidia-smi topo -m`: NVLink
|
||
peers, each card's NUMA node, and how near each network adapter is.
|
||
- **GPUs are chosen by it.** Every set of the right size is scored: NVLink pairs
|
||
within it, then — for a multi-node shard — nearness to the fabric's adapters,
|
||
then NUMA nodes spanned. It used to grow one card's peers, which took cards
|
||
joined to that one but not to each other.
|
||
- **Multi-node NVLink domains** — an NVL72 rack, an NVLink Switch system — are
|
||
found from each GPU's fabric state, ClusterUUID and CliqueId, with
|
||
`nvidia-imex` and its channels. A replica spanning nodes is placed inside one
|
||
domain first, on nodes whose every card is joined and whose IMEX is up, and
|
||
its shards get `NCCL_MNNVL_ENABLE=1`; InfiniBand is the fallback.
|
||
`/admin/nvlink` draws each domain as a rack of trays and says what keeps a
|
||
tray out. A node's page shows its GPU-by-GPU NVLink matrix.
|
||
|
||
### Multi-node launches fixed before they meet hardware
|
||
|
||
- **vLLM launches multi-node its own way by default.** An empty
|
||
`engines.vllm.launch_mode` meant `ray`, never checked against a real cluster;
|
||
it now means `mp`, every node running `vllm serve` with `--nnodes`,
|
||
`--node-rank` and `--master-addr`, the workers `--headless`, as vLLM
|
||
documents. The `mp` path used to give workers no rank and the leader's
|
||
address as their own. Set `launch_mode: ray` to keep Ray.
|
||
- **NCCL bootstraps over the node's own interface.** The agent's default of
|
||
`eth0` meant it never looked for the interface holding its management
|
||
address, and the control plane sent one name for every node. A node on
|
||
`ens3` or `bond0` had NCCL bind nothing.
|
||
- **RoCE** shards get `NCCL_IB_GID_INDEX`, the RoCE v2 entry with an IPv4
|
||
address; NCCL's default is a v1 link-local one that does not route.
|
||
- A Ray worker now waits for its head before joining.
|
||
- The agent unit sets `LimitMEMLOCK=infinity`, which RDMA needs.
|
||
|
||
### `llmgw doctor`
|
||
|
||
Checks a node for what llmgw needs of its GPUs and fabric, and says what is
|
||
wrong and how to fix it, most serious first: NVSwitch without the fabric
|
||
manager, GPUs not joined to their NVLink domain, IMEX, `nvidia_peermem` and
|
||
`ib_uverbs`, each port's link and subnet manager, a fabric with no standby,
|
||
RoCE without a v2 GID, the agent's declared fabrics against the adapters, the
|
||
interface NCCL will use, the unit's memlock limit, and the NCCL environment a
|
||
shard on each fabric would be given. `--json` writes every fact it read, to
|
||
send back; it exits non-zero when something fails.
|
||
|
||
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
|
||
model, where llama.cpp runs several times slower and nothing else looks wrong —
|
||
and whether vLLM can be installed on the node, and why not.
|
||
|
||
### vLLM installs itself on GPU nodes
|
||
|
||
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
|
||
that can run it, with nothing done by hand. A Python 3.12 environment is built
|
||
by uv — itself a GitHub release checked against its checksum — from PyPI,
|
||
resolved as PyPI stood a day after that vLLM was published and installed with
|
||
`--require-hashes`, so every node gets the same files whenever it is built. The
|
||
resolved list stays in the environment as `llmgw-requirements.txt`.
|
||
|
||
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
|
||
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
|
||
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
|
||
25 GB free while it installs. A node that lacks one says which on the
|
||
cluster page; a node with no GPU skips vLLM without a word.
|
||
- gcc is installed with it, since Triton compiles its kernel launcher on the
|
||
first model load.
|
||
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
|
||
version.
|
||
|
||
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
|
||
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
|
||
itself was not started there.
|
||
|
||
A node is asked only for engines its agent says it can install. An agent older
|
||
than this release says nothing, and is asked for llama.cpp alone, so a cluster
|
||
half way through its upgrade does not show every node as failing vLLM.
|
||
|
||
### `llmgw node add`: a node in one command
|
||
|
||
Adding a node was a bash script from the repository, not installed anywhere,
|
||
that needed the node's name, the control plane's address and a dozen flags.
|
||
It is now part of the binary:
|
||
|
||
```
|
||
sudo llmgw node add ubuntu@10.0.0.21
|
||
```
|
||
|
||
run on the control plane. The node id is the machine's hostname and the
|
||
address its agent dials is this machine's on the route to it, checked against
|
||
the server certificate. It installs llmgw on the node — the release for its
|
||
architecture, fetched and checked there, which works on RHEL too — runs
|
||
`llmgw doctor` there and stops on a problem, issues and installs the
|
||
certificate, writes `agent.yaml` and the unit, starts the agent, and waits for
|
||
the node to join and install llama.cpp.
|
||
|
||
In a terminal it is a checklist that fills in, and asks what a script cannot:
|
||
which fabric each group of active adapters is on — grouped by the subnet
|
||
manager or RoCE network they report — and whether to go on past a problem.
|
||
Elsewhere it prints one line per step. Running it again on a node is safe.
|
||
|
||
`llmgw node remove` forgets a node that is gone for good, which otherwise
|
||
stays listed as lost until the control plane restarts; only a lost node, or an
|
||
unreachable one with nothing placed on it, is removed. `llmgw ctl nodes` no
|
||
longer lists vLLM as skipped on every CPU node.
|
||
|
||
Tried on an empty VM, from the terminal screen and plain: joined with
|
||
llama.cpp ready in under a minute, and 17 seconds to add it again.
|
||
|
||
### People belong to many places, or to none
|
||
|
||
A person was in exactly one project, chosen at approval, and could not use the
|
||
gateway before that. Now:
|
||
|
||
- **Memberships.** A person can be a member of any number of organizations,
|
||
teams and projects, or of none. A member of a team or organization may use
|
||
every project in it. `llmgw users join/leave`, and the people page adds and
|
||
removes them from either side. Approving no longer requires a project.
|
||
- **Personal keys.** A key can belong to a person with no project. On
|
||
`/profile` a person chooses what a new key spends against: one of their
|
||
projects, or Personal. `llmgw keys create --for <person>` does the same.
|
||
`auth.personal` sets default limits for personal use, or turns it off.
|
||
- **Limits for people.** `--scope user` limits a person across every key,
|
||
project and the chat page. `--per-member` on an organization, team or
|
||
project gives each member their own share there. Both are on the quotas
|
||
page, and are enforced alongside the tree, each in its own pool.
|
||
- **The chat page** asks what to spend against when a person has more than one
|
||
choice, and checks it on every message.
|
||
- **Spend is recorded with who spent it**, so a person's budget survives a
|
||
restart, personal use and the chat page included.
|
||
|
||
Leaving a project revokes the keys the person made there, unless another
|
||
membership still reaches it. A personal key keeps no conversations or notes:
|
||
those are kept per project.
|
||
|
||
### An organization's own page
|
||
|
||
A member of an organization can be made its **manager** (`users join --org …
|
||
--role manager`, or the star on the people page). A manager gets `/org`: the
|
||
organization's usage over time by team, project or person, its teams and
|
||
projects with limits and spend, and its members — read-only, and only what was
|
||
spent in its projects. Administrators can open every organization's page.
|
||
|
||
### The chat page
|
||
|
||
- A model that is not running shows a progress bar while it loads — waiting
|
||
for a node, downloading, starting — and the message is sent once it is up,
|
||
rather than failing after 45 seconds with "the model is loading".
|
||
- A finished reply is rendered as markdown, by the server under the model
|
||
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
||
- Stop, while a reply streams; its tokens, speed and time to first token under
|
||
it; a system prompt and a temperature; and the conversation kept in the
|
||
browser across a reload — the gateway still keeps none.
|
||
|
||
### Connect an editor, and the Responses API
|
||
|
||
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
||
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
||
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
||
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
||
no secret — the key comes from the environment — keeps a copy of whatever it
|
||
replaces, and leaves the rest of the file alone.
|
||
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
||
chat completions and back — instructions, messages, function and custom
|
||
tools with their calls and outputs, usage, streamed or not — down the same
|
||
path as any request.
|
||
|
||
### Your own use
|
||
|
||
The account page (was "Your keys") shows the last thirty days of a person's own
|
||
use and, for each place they spend, a bar for every budget that applies — their
|
||
own, or a project's shared with others, said as such.
|
||
|
||
### The mirror page
|
||
|
||
- **Explore models** opens on what is trending on Hugging Face, with views for
|
||
new and most downloaded, filters for task and size, and each repository's
|
||
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
||
base model and license. "Only what these nodes can run" is on by default.
|
||
- A repository opens beneath its row with its model card, and a download
|
||
started there starts in place, its progress followed in the table above.
|
||
- What the mirror holds is grouped by model family.
|
||
- An importance matrix published beside a model's quantisations is no longer
|
||
offered as a model.
|
||
|
||
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
|
||
demand, or stopped — and it can be changed there, without the CLI. The serve
|
||
form asks the same: on demand loads the model when somebody asks for it and
|
||
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
||
mode and says to change it there, because the file is read on every start.
|
||
|
||
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
||
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
||
The architecture was in the first kilobytes of the file all along. Now:
|
||
|
||
- The mirror page's registry listing shows each file's architecture and, for
|
||
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
||
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
||
- A fetch reads the file's header first — a range request, not a download —
|
||
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
||
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
||
|
||
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
||
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
||
not refused, only not checked.
|
||
|
||
### Each node's CPU and memory, and a page per node
|
||
|
||
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
||
used and available, and each engine's cores and resident memory. The cluster
|
||
page's node table shows CPU and RAM as bars; each node has a page with its
|
||
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
||
from Prometheus.
|
||
|
||
### The admin pages
|
||
|
||
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
||
it refreshes in place instead of reloading, and its usage table names people
|
||
and projects instead of ids.
|
||
- The usage page's legend gives each line's total, not its last hour.
|
||
- The people page keeps an account on one line and its rarer actions in a menu.
|
||
- Every page fits a phone: a wide table scrolls in its own box.
|
||
- Static files are named by their content, so a deploy's new styles and
|
||
scripts reach browsers at once instead of a day later.
|
||
|
||
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
|
||
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
||
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
||
left for an administrator, Chat, Account and Sign out on the right for
|
||
everyone — with the current page marked. The chat and key pages name the
|
||
person's project by its path rather than its id.
|
||
|
||
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
|
||
its organization held it to sixty requests a minute. Each limit now shows the
|
||
value it is held to from above, muted and marked ↑, with where it comes from on
|
||
hover; a larger value set below a stricter one says what caps it, for rate and
|
||
token limits as well as budgets. Spend and budget share a column with a meter
|
||
that turns red past 90%. The tree is indented like the people page, opening a
|
||
form or saving one no longer reloads the page, and the explanations moved under
|
||
"How limits combine". Projects are placed under their team by id rather than by
|
||
the start of their path, which misplaced them when two organizations shared a
|
||
name.
|
||
|
||
### The project's website, in every gateway
|
||
|
||
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
|
||
the reverse proxy configurations, and the release's files to download. It is
|
||
served without a key like `/`, whose footer links to it, and says nothing about
|
||
the installation. Its fonts are in the binary, so it makes no request elsewhere.
|
||
The published website is rendered from the same template.
|
||
|
||
### Metrics
|
||
|
||
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
||
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
||
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
||
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
||
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
||
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
||
|
||
### Fixes
|
||
|
||
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
||
rewriting read it as the start of a string. Comments are now skipped.
|
||
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
||
- Usage by team, project or organization, and the spend on the quotas page,
|
||
left out every request made on the chat page: they were joined through the
|
||
key, and the chat page has none. They now count where a request was spent.
|
||
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
||
a test now drives every session page through the authentication middleware.
|
||
|
||
### Upgrading
|
||
|
||
Upgrade the control plane first, then the agents: an older agent is served as
|
||
before and simply reports none of the new measurements. Existing agent units
|
||
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
||
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
||
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
||
start with `mp` unless `launch_mode: ray` is set.
|
||
|
||
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
|
||
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
||
the keys table; it was run against a copy of a production PostgreSQL store
|
||
before release. `llmgw users move` is gone: use `join` and `leave`.
|
||
|
||
vLLM is installed on every GPU node that can run it, once its agent is
|
||
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
|
||
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
|
||
replaced by `llmgw node add`; it still works, for a machine holding the
|
||
authority but not this llmgw.
|
||
|
||
## v0.9.2 — 2026-09-28
|
||
|
||
### A new control plane serves a model by itself
|
||
|
||
`llmgw server init` now also runs a model mirror on the same machine, makes the
|
||
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
|
||
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
|
||
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
|
||
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
|
||
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
|
||
and every step is skipped on a rerun when it is already done.
|
||
|
||
Not yet run end to end on a real machine.
|
||
|
||
## v0.9.1 — 2026-09-28
|
||
|
||
### The mirror page is one line per model
|
||
|
||
What the mirror holds used to take two table rows per file — seven columns and
|
||
the address under them — so a sharded model filled the screen. Now each version
|
||
is one line: model and version, the file (or "4 files" for a set), total size,
|
||
and which catalog models use it. The address moved into a "copy address" button;
|
||
the checksum is in the file's tooltip. A version already in use is not offered
|
||
for serving again, and removing one warns that the catalog uses it. The header
|
||
counts versions rather than files.
|
||
|
||
## v0.9.0 — 2026-09-28
|
||
|
||
### Managing people and the tenancy tree
|
||
|
||
Organizations, teams and projects can now be renamed, moved and removed, and an
|
||
approved person can be moved to another project or removed — none of which
|
||
could be done at all before, from the web or the CLI.
|
||
|
||
- **`/admin/users`** shows the whole tree, each project opening onto its people
|
||
and keys, with approve, move, suspend, reinstate and remove for people,
|
||
revoke for keys, and add, rename, move and remove at every level. A search
|
||
over every account sits below it. Changes update the page in place (htmx,
|
||
embedded in the binary) and it still works with scripts off.
|
||
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
|
||
- **`llmgw users move`** and **`llmgw users delete`**.
|
||
- **`llmgw keys list --all`** lists every key with its project and owner.
|
||
- Every command that takes a project, team or organization accepts an id, a
|
||
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
|
||
be omitted when there is only one. Users are named by id or email.
|
||
|
||
Moving a person takes the keys they made with them; keys an operator issued stay
|
||
with the project. Only empty things are removed, and a project that ever had a
|
||
key is never removed, because usage history refers to it.
|
||
|
||
### Changes
|
||
|
||
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
|
||
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
|
||
built from the web has no project of that name. With one project it is
|
||
chosen; with several the command lists them.
|
||
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
|
||
rather than micro-units. `--budget 50000000` now means fifty million units;
|
||
divide existing scripts' figures by 1 000 000.
|
||
|
||
### Fixes
|
||
|
||
- `keys init` could silently move an existing team or project to another
|
||
organization or team by redeclaring it. It now refuses.
|
||
- An account rejected while pending and then reinstated became active with no
|
||
project, and failed at its first key. Reinstating one is refused; approving
|
||
gives it a project. Rejecting a signup now removes it.
|
||
|
||
## v0.8.8 — 2026-09-28
|
||
|
||
### Fixes
|
||
|
||
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
|
||
executable file not found". The PostgreSQL project's own packages (PGDG)
|
||
provide `postgresql-server` too, so dnf can answer that name with
|
||
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
|
||
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
|
||
reads the data directory from it, and runs `initdb` itself when the directory
|
||
is empty. A PostgreSQL that was already installed is used as it is.
|
||
|
||
### Known gaps
|
||
|
||
Not yet run on a real machine. Three arrangements are not handled: a
|
||
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
|
||
and says so), one listening on a port other than 5432, and several PGDG
|
||
versions where the highest is not the one on 5432.
|
||
|
||
## v0.8.7 — 2026-09-28
|
||
|
||
### PostgreSQL in one command
|
||
|
||
```
|
||
sudo llmgw server init --postgres
|
||
```
|
||
|
||
Installs PostgreSQL with the machine's own package manager when there is none,
|
||
creates the `llmgw` role and database with a generated password kept in
|
||
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
|
||
already running on SQLite it stops the gateway, copies the store across so
|
||
every account and key survives, points `auth.store` at PostgreSQL and starts
|
||
the gateway again; if the move fails, the gateway is started again on SQLite.
|
||
Given on the first run, there is never a SQLite store.
|
||
|
||
`server.yaml` is edited in place with its comments intact, and the previous
|
||
file is kept beside it. A PostgreSQL that init did not install is left sized as
|
||
it was, since something else may depend on it.
|
||
|
||
The PostgreSQL installation itself has not yet run on a real machine; the move,
|
||
the configuration edit and the rerun behaviour are tested.
|
||
|
||
### Fixes
|
||
|
||
- **A new control plane with no models did not start.** A configuration with
|
||
no static endpoints and no models was refused as having nothing to serve, so
|
||
under systemd the unit restarted every two seconds and nothing listened on
|
||
8080. It now runs as a control plane with an empty catalog.
|
||
|
||
## v0.8.6 — 2026-09-28
|
||
|
||
### Reverse proxy configurations
|
||
|
||
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
|
||
port, and both are attached to every release beside the binaries. They do not
|
||
buffer streamed replies, their read timeout outlasts a model's wait before its
|
||
first token, their body limit matches the router's, and they set
|
||
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
|
||
cookies `Secure` from the first and shows the second as the base URL on its
|
||
landing page.
|
||
|
||
## v0.8.5 — 2026-09-28
|
||
|
||
### A control plane in one command
|
||
|
||
```
|
||
sudo llmgw server init --admin you@example.com
|
||
```
|
||
|
||
Creates the llmgw user and its directories, the certificate authority, the
|
||
control plane's certificate for the machine's hostname and every address it
|
||
has, `server.yaml`, the default project, and an administrator whose password is
|
||
generated and printed once — then installs the systemd unit, starts it and
|
||
waits until it answers. What used to be two steps of install.md is now this.
|
||
|
||
Each step is skipped when its result exists, so running it again finishes an
|
||
install that stopped and changes nothing on one that did not. `--host` adds an
|
||
address the machine cannot see of itself (a DNS alias, a NAT or floating
|
||
address) and reissues the certificate keeping the old ones. It never overwrites
|
||
`server.yaml` or an authority, and certificates issued by an authority kept on
|
||
another machine are left as they are.
|
||
|
||
### Fixes
|
||
|
||
- **The configuration install.md showed could not start.** An empty catalog
|
||
without `control.enabled: true` is refused as having nothing to serve. The
|
||
documentation and the file `init` writes both set it.
|
||
|
||
## v0.8.4 — 2026-09-28
|
||
|
||
### Node logs on the admin page
|
||
|
||
Each node on the cluster page links to its logs — the agent's own, and the
|
||
output of every engine running there — with an option to follow them. `llmgw
|
||
ctl logs NODE` reads the same. They come over the agent's existing stream, so
|
||
no ssh, jump host or root is needed to see what a node did. The agent keeps its
|
||
last 2000 lines in memory; each engine its last few hundred.
|
||
|
||
## v0.8.3 — 2026-09-28
|
||
|
||
Found by building a node from nothing with v0.8.2.
|
||
|
||
### Fixes
|
||
|
||
- **A new node's first engine install failed on apt's lock.** cloud-init and
|
||
unattended-upgrades run apt in a machine's first minutes, and
|
||
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
|
||
package command held up by a lock now waits for up to ten minutes, and says
|
||
so; a failed install is retried after a minute rather than ten.
|
||
- **A new node reported no disk** until its first download created the weights
|
||
directory. It is created when the agent starts.
|
||
- **Onboarding took one control plane address.** It takes several, comma
|
||
separated, so a new node fails over like the rest.
|
||
|
||
## v0.8.2 — 2026-09-28
|
||
|
||
A node no longer runs whatever engine somebody once put on it. It installs its
|
||
own.
|
||
|
||
### Engines are installed by the node
|
||
|
||
`engines:` in server.yaml names the engines every node has and their versions,
|
||
defaulting to the llama.cpp build this release was checked against (b10991).
|
||
Each agent converges on it: it picks the build for its hardware — CUDA with its
|
||
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
|
||
packages that build needs with the machine's own package manager, downloads the
|
||
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
|
||
anything that would land outside its directory, and runs it. `ldd` names a
|
||
missing library or a system too old for the build ("libstdc++.so.6 is too old
|
||
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
|
||
again every ten minutes.
|
||
|
||
An engine is reported only once installed and seen to run, or when the agent's
|
||
configuration names it on purpose. A node that has installed nothing yet is not
|
||
a candidate for any model, where before it was a candidate for all of them.
|
||
`ctl nodes`, the cluster page and placement's refusals say where each
|
||
installation stands, and warn about a CPU without AVX2.
|
||
|
||
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
|
||
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
|
||
installed this way and must still be named in the agent's configuration.
|
||
|
||
### Onboarding
|
||
|
||
Onboarding installs llmgw from the CI release for the node's architecture,
|
||
checked against the release's checksums, and writes an agent configuration of
|
||
only the control plane's addresses and the TLS files.
|
||
|
||
### Fixes
|
||
|
||
- A race on the weights fingerprint, which two heartbeat loops touched during a
|
||
failover.
|
||
|
||
## v0.8.1 — 2026-09-28
|
||
|
||
Nothing here changes a configuration format: every file that worked with
|
||
v0.8.0 still works. What changes is how much of it needs writing, and how much
|
||
of running the cluster can be done from a page rather than a shell.
|
||
|
||
### A node describes itself
|
||
|
||
An agent file used to name its node, its address, the path to llama-server and
|
||
its weight directories — different on every node and kept true by hand. The
|
||
agent now finds these out: its id is the hostname, its address is the one on
|
||
the route to the control plane, llama-server is looked for on PATH and in
|
||
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
|
||
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
|
||
Memory is re-read on every heartbeat and installed engines once a minute, so a
|
||
VM given more RAM or a node given an engine becomes a candidate without an
|
||
agent restart; the control plane logs both. The agent file shrinks to the
|
||
control plane's addresses and the TLS files, the same on every node.
|
||
|
||
### Models from the mirror, without writing YAML
|
||
|
||
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
|
||
and serve a mirrored version from its row — name, source, engine and an
|
||
estimated memory requirement filled in. A model can say what it is for, and
|
||
`/v1/models` shows it. Placement no longer hands the same memory to two models
|
||
or puts an engine on a node that lacks it, and says when a replica runs an
|
||
edited profile.
|
||
|
||
### Tenancy, quotas and usage on pages
|
||
|
||
The tenancy tree, quotas and budgets are managed from the admin pages; an
|
||
organization's budget bounds its teams, a group's quota counts against the
|
||
group, and a budget that cannot be spent against says so. `/admin/usage`
|
||
charts requests, tokens and spend over time by organization, team, project or
|
||
person, and a request is priced by the model that served it.
|
||
|
||
### Watching the cluster
|
||
|
||
The cluster page has its own charts, backed by a year of Prometheus history,
|
||
with alerts that were made to fire before being trusted — including for the
|
||
disk the metrics share with the models.
|
||
|
||
### Chat
|
||
|
||
The picker offers each model once rather than once per alias, never asks for
|
||
more tokens than the model allows, and offers a model that loads on demand —
|
||
sending it a message is what loads it.
|
||
|
||
## v0.8.0 — 2026-09-27
|
||
|
||
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
||
an existing installation onto it without reissuing every key, and the two
|
||
things a second control plane needs so that running one is safe rather than
|
||
clever.
|
||
|
||
### A second control plane, and a way to fail over to it
|
||
|
||
Leadership is a lease in the store, so two control planes against one database
|
||
cannot both place work — which they otherwise would, both seeing a model
|
||
wanting replicas and none running, leaving twice the replicas each holding GPUs
|
||
the other believes are free.
|
||
|
||
A standby also refuses agent streams. Commands travel down the stream the agent
|
||
opened, so a standby holding one is a node the leader cannot command, and the
|
||
leader's placements would fail with "no agent connected" for a node that is
|
||
plainly connected. Agents therefore take a list of addresses rather than one,
|
||
trying each in turn: a standby refusing, a stopped control plane refusing the
|
||
connection and a crashed one never answering are the same thing from a node.
|
||
|
||
Measured on the cluster. Leadership moved three seconds after the leader was
|
||
stopped — it released the lease on the way out rather than letting it lapse —
|
||
and all five agents had registered with the standby six seconds later. The
|
||
standby adopted every replica rather than restarting any: six replicas, all
|
||
ready, engines that never stopped serving.
|
||
|
||
Everything is unset-means-leader, so a single control plane and every
|
||
deployment that existed before this behave exactly as they did.
|
||
|
||
What this does not do is move client traffic. The standby served inference
|
||
correctly on its own port while the public address returned 502, because
|
||
nothing in front of them moved — that needs a load balancer or a shared
|
||
address, and it is the remaining piece.
|
||
|
||
### PostgreSQL
|
||
|
||
The store runs on PostgreSQL, chosen by what the address looks like: a path is
|
||
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
|
||
address out of a file so a DSN carrying a password stays out of a configuration
|
||
that gets copied and pasted into issues. Every existing installation keeps
|
||
working untouched.
|
||
|
||
One set of queries rather than one per database, because the SQL here is almost
|
||
entirely portable and two copies would drift one query at a time. The rewrite
|
||
of placeholders is a wrapper around the handle rather than a change at sixty
|
||
call sites — which found six places reaching past it, including all three
|
||
manual transactions and two prepared statements, every one of which worked on
|
||
SQLite and would have failed on PostgreSQL.
|
||
|
||
Exactly one query is not the same sentence in both: the note search, MATCH
|
||
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
|
||
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
|
||
to keep in step and no trigger to forget. It uses the 'simple' configuration
|
||
rather than English — these are an agent's own notes, in whatever language it
|
||
chose, and stemming English rules over Turkish would quietly stop finding
|
||
things.
|
||
|
||
`llmgw keys move-store` copies an existing store rather than starting a fresh
|
||
one, because keys are stored hashed: a migration that drops them cannot
|
||
recreate them, and every person and every script would need a new one.
|
||
|
||
Testing it against a real database found three things nothing else would have.
|
||
`schema_migrations` is created in Go rather than in a migration file, so it
|
||
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
|
||
and a millisecond timestamp does not fit. The copy order followed the order the
|
||
migrations create tables in, and `users` is created after `api_keys` even
|
||
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
|
||
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
|
||
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
|
||
difference pgx cannot see, so it arrived as NULL.
|
||
|
||
The driver is pgx, which is pure Go: the whole binary is built with
|
||
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
|
||
and a cgo driver would take that away from every deployment, including the ones
|
||
staying on SQLite.
|
||
|
||
## v0.7.0 — 2026-09-25
|
||
|
||
What a cluster needs to be lived with rather than demonstrated: somewhere to
|
||
keep a conversation, somewhere to keep a note, and an honest answer about the
|
||
disk underneath both.
|
||
|
||
### Memory notes for agents, and threads without API keys
|
||
|
||
Notes an agent writes for itself and reads back in later conversations, offered
|
||
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
|
||
`forget`). A note is a key, text and tags, searched by words or tag, private to
|
||
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
|
||
tools into chat requests: an agent that is not given the server never sees
|
||
memory.
|
||
|
||
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
|
||
and the retention is required, as it is for threads.
|
||
|
||
On a gateway with no authentication, a `threads:` block (`store`,
|
||
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
|
||
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
|
||
nothing refuses it with an explanation instead. The shared pool is for one
|
||
trusted machine: anyone who can reach the port and knows an id can read it.
|
||
|
||
### Conversations that can move between models
|
||
|
||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||
llmgw replays the stored history to whichever model is serving, which is what
|
||
lets a conversation begun on a large model continue on a small one — including
|
||
when a spent budget moved it there and the caller never chose.
|
||
|
||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||
stores no message content anywhere, as it never has, and that default is worth
|
||
more than the feature. The retention is required rather than optional, because
|
||
a conversation store with no expiry is a transcript of everything a team ever
|
||
typed kept forever by omission, and it is measured from the last turn so a
|
||
conversation in progress does not expire underneath it.
|
||
|
||
History that does not fit the serving model is trimmed oldest-first and the
|
||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||
are never dropped — losing one silently changes what the model is for while it
|
||
carries on answering — and neither is the turn the caller just sent.
|
||
|
||
Threads are scoped to a project, and reading, writing or deleting one that
|
||
belongs to another answers exactly as one that never existed: two answers would
|
||
let a caller map which ids are real in projects they cannot read.
|
||
|
||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||
accumulates nothing on the hottest path in the system, and one with a thread
|
||
stops at a cap rather than growing to it.
|
||
|
||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||
operator: the person who said the thing should not have to ask someone to read
|
||
it before it can go.
|
||
|
||
### Disk, and taking back only what llmgw put there
|
||
|
||
Disk was the one resource nothing reported. `statfs` was already being called
|
||
on every model directory, but only to ask whether the filesystem was a network
|
||
one — the free space in the same struct was discarded, so the only way to find
|
||
out why a node was filling up was to ssh to it.
|
||
|
||
Nodes now report it, per filesystem rather than per configured directory: free
|
||
space belongs to the disk, and two model directories on one disk are one pool
|
||
of bytes. Shared filesystems are kept out of a node's own figure, because space
|
||
on a filer belongs to everyone mounting it and twenty nodes would otherwise
|
||
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
|
||
Prometheus, and a node that reported nothing reads as unknown everywhere — an
|
||
older agent must not look like a full disk.
|
||
|
||
A profile may declare `min_disk_gb`, and a node without the room is no longer a
|
||
candidate, with a reason `explain-placement` prints. Skipped on a node that
|
||
already holds the weights: it needs no room for them, and without that the node
|
||
best placed to run a model is the one most likely to be rejected.
|
||
|
||
`ctl weights` lists model files llmgw downloaded for models no longer in the
|
||
catalog, and `--collect` deletes them. The design is mostly about what it will
|
||
not do. Only files llmgw fetched are candidates, and that is a calculation
|
||
rather than a heuristic — the cache name is a hash of the source, so the wanted
|
||
set is computable exactly. Anything an operator put on a node is left alone
|
||
however unreferenced it looks. Each node checks every path again before
|
||
unlinking it, refusing anything outside its cache directory or without the
|
||
cache name, so a control plane that is wrong cannot talk a node into losing a
|
||
model that took a week to copy.
|
||
|
||
### Fixes
|
||
|
||
- **A capability was lost at an indirection.** The controller is handed a
|
||
forwarding dispatcher, and reaches past its interface by type assertion to
|
||
find what lies beyond scheduling. Deleting weights was implemented on the
|
||
agent service and not on the forwarder, so it was simply absent at runtime —
|
||
with the whole suite green, because the dispatchers the tests use do not
|
||
implement it either and its absence there is the expected case.
|
||
- **`ctl tool-check` blamed the engine for what the model did.** It had one
|
||
verdict for a failed attempt and told operators not to try another model.
|
||
Measured: the engine renders the tool prompt correctly, the coder models do
|
||
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
|
||
the same engine returns a populated `tool_calls` array. A failed attempt is
|
||
now two verdicts, told apart by whether the reply used the wrapper.
|
||
|
||
## v0.6.0 — 2026-09-18
|
||
|
||
Phase 4: a replica can be bigger than one machine, and the fabric it spans
|
||
decides where it is allowed to go. Around that, everything a cluster needs
|
||
before anyone other than its operator is let near it — keys, budgets, a way in
|
||
through a browser, and enough visibility to answer "why is nothing scheduling".
|
||
|
||
### Multi-node replicas
|
||
|
||
A replica is now a set of shards with ranks, a leader and a bootstrap address,
|
||
rather than one process. Two invariants are enforced rather than documented: a
|
||
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
|
||
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
|
||
several times slower, and nothing says which.
|
||
|
||
A replica missing a shard is not ready. It was previously derived from the
|
||
shards that had reported, so a two-node replica with one node up looked
|
||
servable and took traffic into a model that could not answer.
|
||
|
||
Adoption after a restart happens one agent at a time, because that is how
|
||
agents reconnect. Merging them as they arrive is what keeps a multi-node
|
||
replica whole instead of being torn into fragments that are then killed as
|
||
strays — and an adopted engine is health-checked before it is called ready,
|
||
rather than being assumed alive because its process exists.
|
||
|
||
### Fabrics are distribution domains too
|
||
|
||
An isolated fabric is not only a compute boundary: weights present in one are
|
||
not present in the other. Node weight inventories are reported and summarised
|
||
per fabric, so "the model is on the cluster" is never mistaken for "the model
|
||
is reachable from the nodes that would run it". A fabric name that matches
|
||
nothing a node actually has is refused at registration rather than becoming an
|
||
unschedulable replica hours later.
|
||
|
||
### Who may use it
|
||
|
||
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
|
||
still no cgo). Rate limits and budgets are enforced on the request path; usage
|
||
is written off it, in batches, so accounting never adds latency to inference.
|
||
Budgets survive a restart — counters are rebuilt from the store, because a
|
||
limit anyone can clear by bouncing the process is not a limit.
|
||
|
||
A spent budget is now served a cheaper model where the catalog offers one,
|
||
rather than refused. For a team trying models out, being dropped to something
|
||
smaller is a far better answer than a wall, and the response says which model
|
||
answered.
|
||
|
||
People can sign up, be approved, and issue and revoke their own keys. A key is
|
||
shown once; only its hash is kept.
|
||
|
||
### The browser surface
|
||
|
||
A landing page that says how to connect, a read-only cluster dashboard, and an
|
||
in-browser chat that never puts an API key in the browser. The dashboard shows
|
||
who is using the cluster and what they are running — per person and per model,
|
||
with requests, tokens, errors, spend and latency.
|
||
|
||
All of it is served from the client-facing port behind a session, deliberately
|
||
not from the admin port, which also serves pprof — and pprof can dump the heap
|
||
that holds every API key in plaintext.
|
||
|
||
### Models without a restart
|
||
|
||
The catalog is live. Models can be added and removed from the command line or
|
||
the API, take effect immediately, and outlive a restart; configuration still
|
||
wins over the store, and the whole catalog is validated before anything is
|
||
written.
|
||
|
||
A model is only listed once it can actually serve. It was listed as soon as it
|
||
was known, so the first thing a new user did was ask for a model that returned
|
||
503.
|
||
|
||
### Loading on demand, and letting go
|
||
|
||
A model whose scaling allows it is started when someone asks for it, and
|
||
stopped when nobody has for a while. An idle replica is evicted to make room
|
||
for one that was asked for, with guards that exist because each is a way to
|
||
lose work quietly: only on-demand models, only single-node replicas, never one
|
||
that has never been used, and never one younger than its minimum lifetime.
|
||
|
||
Unknown is not idle. A replica with no recorded use is left alone rather than
|
||
treated as idle since the epoch.
|
||
|
||
### Knowing whether it will run before it runs
|
||
|
||
- `ctl plan` works out whether a model fits the cluster and how it would be
|
||
split, in the precision the chosen engine can actually serve — recommending
|
||
a quantisation the engine cannot load would cause the failure the command
|
||
exists to prevent.
|
||
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
|
||
installed. Flags drift between releases, and the alternative is finding out
|
||
during a model load.
|
||
- Placement now knows a model's context length and memory footprint, and node
|
||
RAM, so a CPU profile is not placed on a machine that cannot hold it.
|
||
- Placement waits for the fleet to settle after a restart rather than deciding
|
||
against a half-reported cluster.
|
||
|
||
### Weights
|
||
|
||
Nodes report the model files they can see, without llmgw owning the storage:
|
||
a shared filer, an image exported over the fabric, rsync or a copy made by hand
|
||
all look the same to placement, which only cares whether the files are there.
|
||
Reporting nothing means "not asked to look", not "has nothing".
|
||
|
||
Where the simple arrangement is not enough, a node can fetch a model by URL —
|
||
single-flight, resumable, written to a temporary name and renamed atomically,
|
||
so a killed download never leaves something that looks like weights.
|
||
|
||
### Seeing the fleet
|
||
|
||
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
|
||
scrape time rather than from a background goroutine, and served on their own
|
||
endpoint rather than through the one that also exposes pprof. The same view is
|
||
in the dashboard, so nothing else needs installing to see it.
|
||
|
||
### Onboarding a node
|
||
|
||
Adding a node is one command. There are systemd units, a private CA and a
|
||
from-nothing install guide that starts with a machine that has no CUDA and no
|
||
binary.
|
||
|
||
### llama.cpp, and CPU profiles
|
||
|
||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||
can all be exercised on ordinary machines.
|
||
|
||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||
allocation to stop two of them sharing a machine's cores.
|
||
|
||
### Metric mappings are per engine
|
||
|
||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||
and a single global mapping left half of it unmeasurable.
|
||
|
||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||
violation. `Running` and `Waiting` can never be omitted.
|
||
|
||
### Mutual TLS on the agent stream
|
||
|
||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||
node id does not match its certificate's common name. Public authorities cannot
|
||
supply these: they stopped issuing the `clientAuth` usage.
|
||
|
||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||
agent turn it on.
|
||
|
||
### Fixes
|
||
|
||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||
context and then asked that context whether it was shutting down, so every
|
||
stream error read as a clean close: no message, and the backoff reset.
|
||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||
level, so a cluster could look healthy while every routing decision was made
|
||
blind. It is now a warning, once, with a matching line when it clears.
|
||
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
|
||
over about 2 KB was undercounted, so tokens-per-minute limits let through far
|
||
more than they were set to.
|
||
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
|
||
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
|
||
bfloat16 does not fit two 80 GB cards. It does.
|
||
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
|
||
already written the response header, which is the one point at which headers
|
||
stop being changeable. A client had no way to tell a fallback from the model
|
||
it asked for.
|
||
- **Fleet metrics published nothing.** They were registered after the registry
|
||
had been built, so the collector existed and was never scraped.
|
||
- **GPU discovery broke on older drivers.** A field added for utilisation was
|
||
required rather than optional, so a driver that does not report it produced no
|
||
inventory at all instead of an inventory without that number.
|
||
- **Replica ids and ports were reused** while the processes holding them were
|
||
still alive.
|
||
|
||
## v0.5.0 — 2026-09-16
|
||
|
||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||
control plane places replicas, and `ctl` makes all of it visible.
|
||
|
||
This release covers both phases because their exit criteria depended on the same
|
||
piece — the agent stream — and neither was demonstrable without it.
|
||
|
||
### ctl
|
||
|
||
```
|
||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||
```
|
||
|
||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||
the answer when a node looks idle but nothing schedules onto it.
|
||
- `explain-placement` prints every candidate considered and the reason it was
|
||
rejected. "Unschedulable" on its own says nothing about which of six
|
||
constraints bit.
|
||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||
is placed elsewhere, and the original is stopped only once the replacement is
|
||
serving. Bounded by a grace period, because draining the only node that can
|
||
host a model would otherwise never finish and leave it undrainable.
|
||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||
|
||
The management API is on the admin listener, not the client-facing one. The two
|
||
have different audiences: one serves inference to whoever holds an API key, the
|
||
other lets a caller cordon nodes.
|
||
|
||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||
not have. The failure path already ships an engine log tail with every status
|
||
report, so the case that matters most is covered; reading a healthy shard's
|
||
output on demand is not.
|
||
|
||
### Phases 2 and 3, in detail
|
||
|
||
Written as the work landed, which is why it reads as a log rather than as
|
||
notes. Everything here shipped in v0.5.0.
|
||
|
||
### Added
|
||
|
||
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
|
||
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
|
||
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
|
||
without IB is not an error — it is a single-node island.
|
||
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
|
||
would force the whole binary, server included, to be built with a C
|
||
toolchain. Health flags anything that should keep a card out of placement:
|
||
uncorrected ECC errors, and memory already held by another process.
|
||
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
|
||
always match the marketing name: an L40S is sold as 48GB and reports 44.99
|
||
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
|
||
box, because the number decides whether a model fits.
|
||
- **Engine adapters** for vLLM and SGLang: command and environment
|
||
construction, kept free of process handling so it is testable without GPUs.
|
||
vLLM supports both Ray and multiprocessing launch modes, because which one
|
||
works depends on the version.
|
||
- **NCCL environment generation**, the part plan §5.3 marks critical.
|
||
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
|
||
NCCL pick an adapter on another fabric where the peer is unreachable, and the
|
||
job hangs rather than failing. Bootstrap is pinned to the management
|
||
interface for the same reason.
|
||
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
|
||
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
|
||
compute capability 8.9 rather than silently falling back to a slower path.
|
||
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
|
||
process group and are stopped as a group, because engines fork workers that
|
||
hold GPU memory and outlive a signal sent only to the leader. Bounded log
|
||
tail, a state machine that separates Starting from Loading (a 70B model takes
|
||
minutes, and an operator needs to know that is normal), and orphan adoption
|
||
through pid files fingerprinted with the command line — pids get reused, and
|
||
adopting an unrelated process is worse than leaking the original.
|
||
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
|
||
says the weights loaded. Routing on `/health` alone sends traffic into a model
|
||
that is still loading.
|
||
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
|
||
as a table or JSON. Run it before adding a node: it is the fastest way to find
|
||
a card another process is using, or a port with no subnet manager.
|
||
|
||
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
|
||
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
|
||
a tensor-parallel group off the slow path inside a node. `Discover` falls back
|
||
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
|
||
topology. CI compiles the tagged build so it cannot rot unnoticed.
|
||
|
||
### Not verified
|
||
|
||
The NVML path compiles but has never run against a driver — there is no GPU
|
||
machine to try it on. Treat the NVLink domains it reports as unverified until
|
||
someone runs `llmgw agent inventory` on a real node.
|
||
|
||
- **Control plane wiring** (`internal/control`): Ready replicas become router
|
||
endpoints, with no static configuration involved. The router keeps live state
|
||
for endpoints that survive an update — inflight counts, breaker state, drain
|
||
rate — so a replica that was already serving does not start from cold because
|
||
another one moved. The sink is an interface, so when the router and control
|
||
plane split into separate processes this becomes a watch stream and nothing
|
||
above it changes.
|
||
|
||
### Phase 3 exit criterion, measured in `test/e2e`
|
||
|
||
A node hosting a live replica is killed while four clients stream continuously.
|
||
The control plane marks it unreachable, then lost, rebuilds the replica on a
|
||
survivor, and republishes. Across three runs: **~370 requests completed, zero
|
||
failures**, and no replica remained on the dead node.
|
||
|
||
This proves the control logic, not the transport: the agent fleet is simulated,
|
||
and the real path still needs the stream.
|
||
|
||
### A real bug the race detector caught
|
||
|
||
`startReplica` read the replica's fields after publishing it to the map, and
|
||
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
|
||
status reports, which arrive from another goroutine and mutate exactly those
|
||
fields. Under the simulated fleet the reports came back fast enough to hit the
|
||
window every time. Everything is snapshotted under the lock now and the
|
||
functions work from values.
|
||
|
||
### The agent stream
|
||
|
||
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
|
||
carrying registration, heartbeats and shard status upward, and start, stop and
|
||
drain commands downward. The agent dials out, because GPU nodes sit behind
|
||
firewalls far more often than a control plane does and an outbound connection
|
||
needs no inbound rule on every node in the fleet.
|
||
- **Registration carries the whole inventory**, so there is no separate
|
||
discovery call and a reconnecting agent refreshes what the scheduler knows.
|
||
The fabric labels travel with it — they cannot be inferred from the wire, so
|
||
losing them in transit would silently disable multi-node placement.
|
||
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
|
||
the reconciler's dispatcher: it holds the open streams, so it is the only
|
||
thing that can deliver a command to a node.
|
||
- **Agent** (`internal/agent`) validates a profile before launching anything —
|
||
an engine that refuses a profile does so after minutes of loading — then
|
||
supervises the process and reports every transition, including the engine's
|
||
output tail on failure.
|
||
- **Reconnects forever with jittered backoff.** A control plane restart must not
|
||
require touching every GPU node, and a fleet must not reconnect in lockstep
|
||
and knock the control plane over again. Engines keep serving throughout: the
|
||
router talks to them directly.
|
||
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
|
||
any model declares a profile. The static endpoint list remains for
|
||
router-only runs against sim-engines.
|
||
|
||
### Verified with real binaries
|
||
|
||
Server and agent as separate processes over a real stream: the agent registers,
|
||
a replica is placed, an engine is launched through the vLLM adapter's actual
|
||
argument construction, streaming traffic is served through the router, and
|
||
killing the agent moves the node through unreachable to lost with placement
|
||
explaining exactly why it can no longer run anything.
|
||
|
||
### Surviving a control plane restart
|
||
|
||
Not persisting replica state left one hole that had to be closed before it
|
||
mattered: restarting the control plane orphaned live engines. They kept holding
|
||
GPUs the inventory believed were free, so the next placement failed to allocate
|
||
with an error naming the new process rather than the old one.
|
||
|
||
The fix is on the agent, not in a database, and it cannot disagree with reality
|
||
the way a stale row can:
|
||
|
||
- The supervisor now writes the full shard spec, not just a pid and command
|
||
line. A pid identifies a process; the spec is what lets it be described back
|
||
to the control plane and matched to a replica.
|
||
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
|
||
waited on — it is not this process's child — so it is watched by polling, and
|
||
stopped by signalling its group directly. Its earlier output is gone, and the
|
||
log says so rather than presenting emptiness as silence.
|
||
- Every `Register`, not only the first, reports what the agent is running. A
|
||
reconnect may be a control plane restart and has to recover the same way.
|
||
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
|
||
Anything whose model or profile left the catalog is stopped: nothing will
|
||
route to it and it is holding cards.
|
||
- An engine the agent cannot identify — a corrupt state file — is killed. A
|
||
process holding GPUs that nothing can describe is worse than no process,
|
||
because placement will hand those GPUs out again.
|
||
|
||
One mechanism covers agent restarts and control plane restarts.
|
||
|
||
### A test that was passing for no reason
|
||
|
||
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
|
||
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
|
||
tracking hijacked connections. The agent never reconnected and the assertion —
|
||
on the connection map, which still held a stale entry — passed anyway. Tests now
|
||
sever connections through their own listener and count opened streams.
|
||
|
||
### Multi-node launch
|
||
|
||
Placement and environment generation had landed early, which made multi-node
|
||
look further along than it was: a multi-node replica could not come up at all.
|
||
Three things were missing, and all three came down to two assumptions that do
|
||
not hold once a replica spans nodes.
|
||
|
||
**Not every rank serves HTTP.** The agent health-checked every shard at its own
|
||
port, but only the leader answers. Workers timed out, reported Failed, and took
|
||
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
|
||
worker is judged by staying alive instead. That is weaker evidence on purpose —
|
||
the replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined.
|
||
|
||
**A shard is not always one process.** `Engine.Commands(spec)` returns a
|
||
sequence: every command but the last is a sidecar kept alive for the shard's
|
||
lifetime and torn down with it, and the last one's exit is the shard's exit.
|
||
vLLM under Ray is why — the leader runs a Ray head and then the server against
|
||
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
|
||
had been written and tested but never called, so `launch_mode: ray` would have
|
||
run `vllm serve` against a cluster that did not exist.
|
||
|
||
A command can also require the rendezvous point to be listening before the next
|
||
one starts, bounded by a timeout so a head that never appears fails the shard
|
||
rather than hanging it. Sidecars are killed with the shard: a Ray head left
|
||
behind holds the port the next replica needs.
|
||
|
||
**Start order was backwards.** Workers were started first, on the theory that
|
||
the head should wait for them. Both engines put the rendezvous point on rank 0,
|
||
so a worker started first has nothing to connect to and spends its startup
|
||
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
|
||
needs, so the leader can start immediately and waits on its own.
|
||
|
||
None of this is verified against real engines. It is tested with stub processes;
|
||
the Ray sequencing and which launch mode a given vLLM version wants can only be
|
||
confirmed on hardware.
|
||
|
||
### Still missing for phase 2
|
||
|
||
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
|
||
network.
|
||
|
||
### Phase 3: placement
|
||
|
||
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
|
||
their fabric attachments, and what is allocated where. A re-registering agent
|
||
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
|
||
without doing so either. A GPU with an ECC fault or another process on it is
|
||
not counted as free. Replicas are released whole, because they fail whole.
|
||
- **Placement** (`internal/control/placement`): picks the highest-priority
|
||
profile that fits and explains every candidate it rejected. Two constraints
|
||
are enforced structurally rather than checked afterwards — there is no code
|
||
path that can combine nodes from two fabrics, and only GPUs of the profile's
|
||
class are ever considered.
|
||
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
|
||
large models that cannot be split. Multi-node replicas get the opposite
|
||
treatment: they are the least movable thing in the cluster, so they take the
|
||
emptiest machines.
|
||
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
|
||
domain where the topology is known. Without the NVML build it falls back to
|
||
the lowest free indices, which is usually right but only by accident of how
|
||
vendors number slots.
|
||
- **Property-based invariant tests** (plan §13): 400 randomly generated
|
||
clusters, checking on every placement that a multi-node replica lives on one
|
||
fabric, that each shard receives only that fabric's adapters, and that no
|
||
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
|
||
The table-driven tests cover the cases someone thought of; this covers the
|
||
ones nobody did.
|
||
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
|
||
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
|
||
rather than at request time — a typo in a fallback should stop a deployment,
|
||
not surface during the outage when the fallback is finally needed. The
|
||
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
|
||
load instead of minutes into model loading.
|
||
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
|
||
loop. It places one replica per pass, because placement depends on what is
|
||
already allocated and starting several from one snapshot would hand them the
|
||
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
|
||
vLLM under Ray needs its workers attached before the head serves.
|
||
- **Replica failure is whole-replica failure.** One dead shard makes the others
|
||
useless — they wait on collectives that will never complete while holding
|
||
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
|
||
exponential backoff carried on the model rather than the replica it just
|
||
destroyed.
|
||
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
|
||
blip must not tear down replicas that are still serving, and rebuilding a
|
||
multi-node replica costs tens of minutes of model loading.
|
||
- **Scale-down drains before stopping**, so in-flight requests finish instead of
|
||
being cut off mid-stream.
|
||
|
||
## v0.3.0 — 2026-09-16
|
||
|
||
Flow control. A batch workload no longer starves interactive traffic.
|
||
|
||
### Added
|
||
|
||
- **Admission control** (`internal/router/admission`), inserted between quota
|
||
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||
still answers *where* it goes. Keeping them apart is the point: an engine
|
||
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||
no amount of endpoint selection rescues the request behind it.
|
||
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||
nearly full cache, or the reverse.
|
||
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||
"flow up to the healthy limit"; above 1 means never wait.
|
||
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||
one caller sending a thousand requests occupies one slot in the rotation
|
||
rather than a thousand.
|
||
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||
(504), because sending the GPU work the client has already abandoned helps
|
||
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||
`x-llmgw-queue-ms` when a request waited.
|
||
|
||
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||
|
||
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||
With admission switched off the same load pushes it to **55–60x**. Both
|
||
directions are asserted, so the test fails if admission regresses *and* if the
|
||
load generator ever stops being a real load.
|
||
|
||
### Two lags, two different fixes
|
||
|
||
Getting this right took longer than the queueing itself:
|
||
|
||
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||
count does not — but it includes *running* requests, and treating those as
|
||
queue depth makes a healthy engine look overloaded every interval. What is
|
||
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||
the metric contract, so it is learned as the high-water mark of observed
|
||
`running`.
|
||
- Between admission granting a request and the router dispatching it, the
|
||
request is invisible to every signal. A simultaneous burst therefore passed
|
||
the check en masse — exactly the batch arrival pattern this layer exists to
|
||
survive. The controller now counts its own outstanding leases against
|
||
dispatched inflight.
|
||
|
||
### Known limitations
|
||
|
||
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||
Anyone who can reach the port can promote themselves to platinum.
|
||
- The tenant key is the session id, for the same reason.
|
||
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||
|
||
## v0.2.0 — 2026-09-16
|
||
|
||
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||
engine replicas with an OpenAI-compatible API.
|
||
|
||
### Added
|
||
|
||
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||
propagates cancellation: when a client disconnects, the upstream request is
|
||
aborted so the GPU stops working. Separate deadlines for dial, response
|
||
headers, first token, inter-token idle and total duration — a wedged engine is
|
||
indistinguishable from a slow one without the idle watchdog.
|
||
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||
reached the client; once bytes are out, splicing two responses together would
|
||
be worse than the short stream. Models can name fallback models, and the
|
||
response says which one served it via `x-llmgw-served-model`.
|
||
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||
Only routing-relevant fields are parsed; the rest of the body passes through
|
||
untouched.
|
||
- **Usage accounting that does not depend on the client.** The router can ask
|
||
engines for usage the client never requested, then drop that chunk on the way
|
||
back, so token counts are recorded either way.
|
||
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||
duplicate endpoint ids and aliases that would shadow another model — both
|
||
would otherwise misroute silently at request time.
|
||
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||
without seeing what the scorers actually produced. Served on a separate admin
|
||
listener alongside health and pprof.
|
||
|
||
### Phase 1 exit criterion, measured in `test/e2e`
|
||
|
||
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||
- Every client disconnecting frees the KV blocks on every engine.
|
||
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||
|
||
### Known limitations
|
||
|
||
- No admission control, so a batch workload can still starve interactive
|
||
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||
|
||
## v0.1.0 — 2026-09-16
|
||
|
||
First tagged version. It is useful for exactly one thing: developing and testing
|
||
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
||
returns "not implemented".
|
||
|
||
### Added
|
||
|
||
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
||
that models the behaviours a router has to cope with:
|
||
|
||
- continuous batching with a bounded concurrent sequence count
|
||
- KV cache block accounting, queueing when blocks run out, including
|
||
head-of-line blocking
|
||
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
||
effect on time to first token
|
||
- per-token latency that grows with batch size
|
||
- dropping a request and freeing its blocks when the client disconnects
|
||
- `/metrics` under vLLM's real metric names
|
||
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
||
|
||
**Routing primitives** — libraries, not yet wired into a server:
|
||
|
||
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
||
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
||
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
||
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
||
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
||
detection and drain rate derived from engine counters.
|
||
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
||
endpoints missing any of the mandatory three. The metric name mapping is
|
||
overridable, because names drift between engine versions.
|
||
- `router/breaker`: cooldown and circuit breaking in one component, with
|
||
exponential backoff and half-open probes.
|
||
|
||
### Known limitations
|
||
|
||
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
||
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
||
- No admission control, so a batch workload can still starve interactive traffic
|
||
once the router exists. That is what v0.3.0 is for.
|
||
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
||
|
||
### Notes
|
||
|
||
- The project language is English throughout: code, comments, tests, commits and
|
||
the design document.
|
||
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
||
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|