kev-0.5b declared architecture "ggmlc", which llama.cpp does not know. It was fetched, catalogued and placed, and failed on every node — while the answer was in the first kilobytes of the file. internal/gguf reads a GGUF header (v1-v3) up to general.architecture, over HTTP with a Range request, and judges it against the architectures llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at that tag); a vision projector is not a model on its own. An unknown build is not refused, only not vouched for, and a header that cannot be fetched does not refuse anything. Checked in three places: the registry listing (from the architecture Hugging Face reports with expand[]=gguf, so one request per repository) shows it and withholds the fetch button; a fetch reads the header and refuses unless "fetch anyway"; and the model service refuses to add such a model, which covers the serve form and ctl model add. Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3 (bert), and Qwen2.5-0.5B on Hugging Face through its redirect. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1183 lines
60 KiB
Markdown
1183 lines
60 KiB
Markdown
# Changelog
|
||
|
||
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
||
is only released once its **exit criterion** is met, not when its code lands.
|
||
|
||
| Version | Phase | Scope | State |
|
||
|---|---|---|---|
|
||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
|
||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
|
||
| v0.7.0 | — | Conversations, agent memory, disk management | released |
|
||
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
|
||
| v0.9.0 | — | Managing people and the tenancy tree | released |
|
||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||
|
||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||
APIs change between minors without a deprecation period.
|
||
|
||
## Unreleased
|
||
|
||
### An organization's own page
|
||
|
||
A member of an organization can be made its **manager** (`users join --org …
|
||
--role manager`, or the star on the people page). A manager gets `/org`: the
|
||
organization's usage over time by team, project or person, its teams and
|
||
projects with limits and spend, and its members — read-only, and only what was
|
||
spent in its projects. Administrators can open every organization's page.
|
||
|
||
### How a model runs, from the mirror page
|
||
|
||
Each model a mirrored file is serving now shows how it runs — always on, on
|
||
demand, or stopped — and it can be changed there, without the CLI. The serve
|
||
form asks the same: on demand loads the model when somebody asks for it and
|
||
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
||
mode and says to change it there, because the file is read on every start.
|
||
|
||
### Weights the nodes cannot load are not fetched
|
||
|
||
`kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
||
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
||
The architecture was in the first kilobytes of the file all along. Now:
|
||
|
||
- The mirror page's registry listing shows each file's architecture and, for
|
||
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
||
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
||
- A fetch reads the file's header first — a range request, not a download —
|
||
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
||
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
||
|
||
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
||
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
||
not refused, only not checked.
|
||
|
||
### Fixes
|
||
|
||
- Usage by team, project or organization, and the spend on the quotas page,
|
||
left out every request made on the chat page: they were joined through the
|
||
key, and the chat page has none. They now count where a request was spent.
|
||
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
||
a test now drives every session page through the authentication middleware.
|
||
|
||
### People belong to many places, or to none
|
||
|
||
A person was in exactly one project, chosen at approval, and could not use the
|
||
gateway before that. Now:
|
||
|
||
- **Memberships.** A person can be a member of any number of organizations,
|
||
teams and projects, or of none. A member of a team or organization may use
|
||
every project in it. `llmgw users join/leave`, and the people page adds and
|
||
removes them from either side. Approving no longer requires a project.
|
||
- **Personal keys.** A key can belong to a person with no project. On
|
||
`/profile` a person chooses what a new key spends against: one of their
|
||
projects, or Personal. `llmgw keys create --for <person>` does the same.
|
||
`auth.personal` sets default limits for personal use, or turns it off.
|
||
- **Limits for people.** `--scope user` limits a person across every key,
|
||
project and the chat page. `--per-member` on an organization, team or
|
||
project gives each member their own share there. Both are on the quotas
|
||
page, and are enforced alongside the tree, each in its own pool.
|
||
- **The chat page** asks what to spend against when a person has more than one
|
||
choice, and checks it on every message.
|
||
- **Spend is recorded with who spent it**, so a person's budget survives a
|
||
restart, personal use and the chat page included.
|
||
|
||
Leaving a project revokes the keys the person made there, unless another
|
||
membership still reaches it. A personal key keeps no conversations or notes:
|
||
those are kept per project.
|
||
|
||
**Upgrading.** Migration 0010 turns everyone's project into a membership and
|
||
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
||
the keys table; it was run against a copy of a production PostgreSQL store
|
||
before release. `llmgw users move` is gone: use `join` and `leave`.
|
||
|
||
### One menu on every page
|
||
|
||
Each page drew its own menu, and no two agreed: different links, in a different
|
||
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
||
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
||
left for an administrator, Chat, Your keys and Sign out on the right for
|
||
everyone — with the current page marked. The chat and key pages name the
|
||
person's project by its path rather than its id.
|
||
|
||
### The quotas page says what actually applies
|
||
|
||
A branch that sets no limit of its own used to read "—" in every column while
|
||
its organization held it to sixty requests a minute. Each limit now shows the
|
||
value it is held to from above, muted and marked ↑, with where it comes from on
|
||
hover; a larger value set below a stricter one says what caps it, for rate and
|
||
token limits as well as budgets. Spend and budget share a column with a meter
|
||
that turns red past 90%. The tree is indented like the people page, opening a
|
||
form or saving one no longer reloads the page, and the explanations moved under
|
||
"How limits combine". Projects are placed under their team by id rather than by
|
||
the start of their path, which misplaced them when two organizations shared a
|
||
name.
|
||
|
||
## v0.9.2 — 2026-09-28
|
||
|
||
### A new control plane serves a model by itself
|
||
|
||
`llmgw server init` now also runs a model mirror on the same machine, makes the
|
||
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
|
||
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
|
||
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
|
||
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
|
||
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
|
||
and every step is skipped on a rerun when it is already done.
|
||
|
||
Not yet run end to end on a real machine.
|
||
|
||
## v0.9.1 — 2026-09-28
|
||
|
||
### The mirror page is one line per model
|
||
|
||
What the mirror holds used to take two table rows per file — seven columns and
|
||
the address under them — so a sharded model filled the screen. Now each version
|
||
is one line: model and version, the file (or "4 files" for a set), total size,
|
||
and which catalog models use it. The address moved into a "copy address" button;
|
||
the checksum is in the file's tooltip. A version already in use is not offered
|
||
for serving again, and removing one warns that the catalog uses it. The header
|
||
counts versions rather than files.
|
||
|
||
## v0.9.0 — 2026-09-28
|
||
|
||
### Managing people and the tenancy tree
|
||
|
||
Organizations, teams and projects can now be renamed, moved and removed, and an
|
||
approved person can be moved to another project or removed — none of which
|
||
could be done at all before, from the web or the CLI.
|
||
|
||
- **`/admin/users`** shows the whole tree, each project opening onto its people
|
||
and keys, with approve, move, suspend, reinstate and remove for people,
|
||
revoke for keys, and add, rename, move and remove at every level. A search
|
||
over every account sits below it. Changes update the page in place (htmx,
|
||
embedded in the binary) and it still works with scripts off.
|
||
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
|
||
- **`llmgw users move`** and **`llmgw users delete`**.
|
||
- **`llmgw keys list --all`** lists every key with its project and owner.
|
||
- Every command that takes a project, team or organization accepts an id, a
|
||
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
|
||
be omitted when there is only one. Users are named by id or email.
|
||
|
||
Moving a person takes the keys they made with them; keys an operator issued stay
|
||
with the project. Only empty things are removed, and a project that ever had a
|
||
key is never removed, because usage history refers to it.
|
||
|
||
### Changes
|
||
|
||
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
|
||
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
|
||
built from the web has no project of that name. With one project it is
|
||
chosen; with several the command lists them.
|
||
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
|
||
rather than micro-units. `--budget 50000000` now means fifty million units;
|
||
divide existing scripts' figures by 1 000 000.
|
||
|
||
### Fixes
|
||
|
||
- `keys init` could silently move an existing team or project to another
|
||
organization or team by redeclaring it. It now refuses.
|
||
- An account rejected while pending and then reinstated became active with no
|
||
project, and failed at its first key. Reinstating one is refused; approving
|
||
gives it a project. Rejecting a signup now removes it.
|
||
|
||
## v0.8.8 — 2026-09-28
|
||
|
||
### Fixes
|
||
|
||
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
|
||
executable file not found". The PostgreSQL project's own packages (PGDG)
|
||
provide `postgresql-server` too, so dnf can answer that name with
|
||
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
|
||
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
|
||
reads the data directory from it, and runs `initdb` itself when the directory
|
||
is empty. A PostgreSQL that was already installed is used as it is.
|
||
|
||
### Known gaps
|
||
|
||
Not yet run on a real machine. Three arrangements are not handled: a
|
||
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
|
||
and says so), one listening on a port other than 5432, and several PGDG
|
||
versions where the highest is not the one on 5432.
|
||
|
||
## v0.8.7 — 2026-09-28
|
||
|
||
### PostgreSQL in one command
|
||
|
||
```
|
||
sudo llmgw server init --postgres
|
||
```
|
||
|
||
Installs PostgreSQL with the machine's own package manager when there is none,
|
||
creates the `llmgw` role and database with a generated password kept in
|
||
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
|
||
already running on SQLite it stops the gateway, copies the store across so
|
||
every account and key survives, points `auth.store` at PostgreSQL and starts
|
||
the gateway again; if the move fails, the gateway is started again on SQLite.
|
||
Given on the first run, there is never a SQLite store.
|
||
|
||
`server.yaml` is edited in place with its comments intact, and the previous
|
||
file is kept beside it. A PostgreSQL that init did not install is left sized as
|
||
it was, since something else may depend on it.
|
||
|
||
The PostgreSQL installation itself has not yet run on a real machine; the move,
|
||
the configuration edit and the rerun behaviour are tested.
|
||
|
||
### Fixes
|
||
|
||
- **A new control plane with no models did not start.** A configuration with
|
||
no static endpoints and no models was refused as having nothing to serve, so
|
||
under systemd the unit restarted every two seconds and nothing listened on
|
||
8080. It now runs as a control plane with an empty catalog.
|
||
|
||
## v0.8.6 — 2026-09-28
|
||
|
||
### Reverse proxy configurations
|
||
|
||
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
|
||
port, and both are attached to every release beside the binaries. They do not
|
||
buffer streamed replies, their read timeout outlasts a model's wait before its
|
||
first token, their body limit matches the router's, and they set
|
||
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
|
||
cookies `Secure` from the first and shows the second as the base URL on its
|
||
landing page.
|
||
|
||
## v0.8.5 — 2026-09-28
|
||
|
||
### A control plane in one command
|
||
|
||
```
|
||
sudo llmgw server init --admin you@example.com
|
||
```
|
||
|
||
Creates the llmgw user and its directories, the certificate authority, the
|
||
control plane's certificate for the machine's hostname and every address it
|
||
has, `server.yaml`, the default project, and an administrator whose password is
|
||
generated and printed once — then installs the systemd unit, starts it and
|
||
waits until it answers. What used to be two steps of install.md is now this.
|
||
|
||
Each step is skipped when its result exists, so running it again finishes an
|
||
install that stopped and changes nothing on one that did not. `--host` adds an
|
||
address the machine cannot see of itself (a DNS alias, a NAT or floating
|
||
address) and reissues the certificate keeping the old ones. It never overwrites
|
||
`server.yaml` or an authority, and certificates issued by an authority kept on
|
||
another machine are left as they are.
|
||
|
||
### Fixes
|
||
|
||
- **The configuration install.md showed could not start.** An empty catalog
|
||
without `control.enabled: true` is refused as having nothing to serve. The
|
||
documentation and the file `init` writes both set it.
|
||
|
||
## v0.8.4 — 2026-09-28
|
||
|
||
### Node logs on the admin page
|
||
|
||
Each node on the cluster page links to its logs — the agent's own, and the
|
||
output of every engine running there — with an option to follow them. `llmgw
|
||
ctl logs NODE` reads the same. They come over the agent's existing stream, so
|
||
no ssh, jump host or root is needed to see what a node did. The agent keeps its
|
||
last 2000 lines in memory; each engine its last few hundred.
|
||
|
||
## v0.8.3 — 2026-09-28
|
||
|
||
Found by building a node from nothing with v0.8.2.
|
||
|
||
### Fixes
|
||
|
||
- **A new node's first engine install failed on apt's lock.** cloud-init and
|
||
unattended-upgrades run apt in a machine's first minutes, and
|
||
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
|
||
package command held up by a lock now waits for up to ten minutes, and says
|
||
so; a failed install is retried after a minute rather than ten.
|
||
- **A new node reported no disk** until its first download created the weights
|
||
directory. It is created when the agent starts.
|
||
- **Onboarding took one control plane address.** It takes several, comma
|
||
separated, so a new node fails over like the rest.
|
||
|
||
## v0.8.2 — 2026-09-28
|
||
|
||
A node no longer runs whatever engine somebody once put on it. It installs its
|
||
own.
|
||
|
||
### Engines are installed by the node
|
||
|
||
`engines:` in server.yaml names the engines every node has and their versions,
|
||
defaulting to the llama.cpp build this release was checked against (b10991).
|
||
Each agent converges on it: it picks the build for its hardware — CUDA with its
|
||
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
|
||
packages that build needs with the machine's own package manager, downloads the
|
||
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
|
||
anything that would land outside its directory, and runs it. `ldd` names a
|
||
missing library or a system too old for the build ("libstdc++.so.6 is too old
|
||
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
|
||
again every ten minutes.
|
||
|
||
An engine is reported only once installed and seen to run, or when the agent's
|
||
configuration names it on purpose. A node that has installed nothing yet is not
|
||
a candidate for any model, where before it was a candidate for all of them.
|
||
`ctl nodes`, the cluster page and placement's refusals say where each
|
||
installation stands, and warn about a CPU without AVX2.
|
||
|
||
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
|
||
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
|
||
installed this way and must still be named in the agent's configuration.
|
||
|
||
### Onboarding
|
||
|
||
Onboarding installs llmgw from the CI release for the node's architecture,
|
||
checked against the release's checksums, and writes an agent configuration of
|
||
only the control plane's addresses and the TLS files.
|
||
|
||
### Fixes
|
||
|
||
- A race on the weights fingerprint, which two heartbeat loops touched during a
|
||
failover.
|
||
|
||
## v0.8.1 — 2026-09-28
|
||
|
||
Nothing here changes a configuration format: every file that worked with
|
||
v0.8.0 still works. What changes is how much of it needs writing, and how much
|
||
of running the cluster can be done from a page rather than a shell.
|
||
|
||
### A node describes itself
|
||
|
||
An agent file used to name its node, its address, the path to llama-server and
|
||
its weight directories — different on every node and kept true by hand. The
|
||
agent now finds these out: its id is the hostname, its address is the one on
|
||
the route to the control plane, llama-server is looked for on PATH and in
|
||
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
|
||
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
|
||
Memory is re-read on every heartbeat and installed engines once a minute, so a
|
||
VM given more RAM or a node given an engine becomes a candidate without an
|
||
agent restart; the control plane logs both. The agent file shrinks to the
|
||
control plane's addresses and the TLS files, the same on every node.
|
||
|
||
### Models from the mirror, without writing YAML
|
||
|
||
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
|
||
and serve a mirrored version from its row — name, source, engine and an
|
||
estimated memory requirement filled in. A model can say what it is for, and
|
||
`/v1/models` shows it. Placement no longer hands the same memory to two models
|
||
or puts an engine on a node that lacks it, and says when a replica runs an
|
||
edited profile.
|
||
|
||
### Tenancy, quotas and usage on pages
|
||
|
||
The tenancy tree, quotas and budgets are managed from the admin pages; an
|
||
organization's budget bounds its teams, a group's quota counts against the
|
||
group, and a budget that cannot be spent against says so. `/admin/usage`
|
||
charts requests, tokens and spend over time by organization, team, project or
|
||
person, and a request is priced by the model that served it.
|
||
|
||
### Watching the cluster
|
||
|
||
The cluster page has its own charts, backed by a year of Prometheus history,
|
||
with alerts that were made to fire before being trusted — including for the
|
||
disk the metrics share with the models.
|
||
|
||
### Chat
|
||
|
||
The picker offers each model once rather than once per alias, never asks for
|
||
more tokens than the model allows, and offers a model that loads on demand —
|
||
sending it a message is what loads it.
|
||
|
||
## v0.8.0 — 2026-09-27
|
||
|
||
The store stops being a file on one machine's disk. PostgreSQL, a way to move
|
||
an existing installation onto it without reissuing every key, and the two
|
||
things a second control plane needs so that running one is safe rather than
|
||
clever.
|
||
|
||
### A second control plane, and a way to fail over to it
|
||
|
||
Leadership is a lease in the store, so two control planes against one database
|
||
cannot both place work — which they otherwise would, both seeing a model
|
||
wanting replicas and none running, leaving twice the replicas each holding GPUs
|
||
the other believes are free.
|
||
|
||
A standby also refuses agent streams. Commands travel down the stream the agent
|
||
opened, so a standby holding one is a node the leader cannot command, and the
|
||
leader's placements would fail with "no agent connected" for a node that is
|
||
plainly connected. Agents therefore take a list of addresses rather than one,
|
||
trying each in turn: a standby refusing, a stopped control plane refusing the
|
||
connection and a crashed one never answering are the same thing from a node.
|
||
|
||
Measured on the cluster. Leadership moved three seconds after the leader was
|
||
stopped — it released the lease on the way out rather than letting it lapse —
|
||
and all five agents had registered with the standby six seconds later. The
|
||
standby adopted every replica rather than restarting any: six replicas, all
|
||
ready, engines that never stopped serving.
|
||
|
||
Everything is unset-means-leader, so a single control plane and every
|
||
deployment that existed before this behave exactly as they did.
|
||
|
||
What this does not do is move client traffic. The standby served inference
|
||
correctly on its own port while the public address returned 502, because
|
||
nothing in front of them moved — that needs a load balancer or a shared
|
||
address, and it is the remaining piece.
|
||
|
||
### PostgreSQL
|
||
|
||
The store runs on PostgreSQL, chosen by what the address looks like: a path is
|
||
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
|
||
address out of a file so a DSN carrying a password stays out of a configuration
|
||
that gets copied and pasted into issues. Every existing installation keeps
|
||
working untouched.
|
||
|
||
One set of queries rather than one per database, because the SQL here is almost
|
||
entirely portable and two copies would drift one query at a time. The rewrite
|
||
of placeholders is a wrapper around the handle rather than a change at sixty
|
||
call sites — which found six places reaching past it, including all three
|
||
manual transactions and two prepared statements, every one of which worked on
|
||
SQLite and would have failed on PostgreSQL.
|
||
|
||
Exactly one query is not the same sentence in both: the note search, MATCH
|
||
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
|
||
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
|
||
to keep in step and no trigger to forget. It uses the 'simple' configuration
|
||
rather than English — these are an agent's own notes, in whatever language it
|
||
chose, and stemming English rules over Turkish would quietly stop finding
|
||
things.
|
||
|
||
`llmgw keys move-store` copies an existing store rather than starting a fresh
|
||
one, because keys are stored hashed: a migration that drops them cannot
|
||
recreate them, and every person and every script would need a new one.
|
||
|
||
Testing it against a real database found three things nothing else would have.
|
||
`schema_migrations` is created in Go rather than in a migration file, so it
|
||
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
|
||
and a millisecond timestamp does not fit. The copy order followed the order the
|
||
migrations create tables in, and `users` is created after `api_keys` even
|
||
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
|
||
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
|
||
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
|
||
difference pgx cannot see, so it arrived as NULL.
|
||
|
||
The driver is pgx, which is pure Go: the whole binary is built with
|
||
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
|
||
and a cgo driver would take that away from every deployment, including the ones
|
||
staying on SQLite.
|
||
|
||
## v0.7.0 — 2026-09-25
|
||
|
||
What a cluster needs to be lived with rather than demonstrated: somewhere to
|
||
keep a conversation, somewhere to keep a note, and an honest answer about the
|
||
disk underneath both.
|
||
|
||
### Memory notes for agents, and threads without API keys
|
||
|
||
Notes an agent writes for itself and reads back in later conversations, offered
|
||
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
|
||
`forget`). A note is a key, text and tags, searched by words or tag, private to
|
||
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
|
||
tools into chat requests: an agent that is not given the server never sees
|
||
memory.
|
||
|
||
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
|
||
and the retention is required, as it is for threads.
|
||
|
||
On a gateway with no authentication, a `threads:` block (`store`,
|
||
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
|
||
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
|
||
nothing refuses it with an explanation instead. The shared pool is for one
|
||
trusted machine: anyone who can reach the port and knows an id can read it.
|
||
|
||
### Conversations that can move between models
|
||
|
||
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
|
||
llmgw replays the stored history to whichever model is serving, which is what
|
||
lets a conversation begun on a large model continue on a small one — including
|
||
when a spent budget moved it there and the caller never chose.
|
||
|
||
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
|
||
stores no message content anywhere, as it never has, and that default is worth
|
||
more than the feature. The retention is required rather than optional, because
|
||
a conversation store with no expiry is a transcript of everything a team ever
|
||
typed kept forever by omission, and it is measured from the last turn so a
|
||
conversation in progress does not expire underneath it.
|
||
|
||
History that does not fit the serving model is trimmed oldest-first and the
|
||
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
|
||
are never dropped — losing one silently changes what the model is for while it
|
||
carries on answering — and neither is the turn the caller just sent.
|
||
|
||
Threads are scoped to a project, and reading, writing or deleting one that
|
||
belongs to another answers exactly as one that never existed: two answers would
|
||
let a caller map which ids are real in projects they cannot read.
|
||
|
||
Capturing the reply is opt-in and bounded, so a request without a thread
|
||
accumulates nothing on the hottest path in the system, and one with a thread
|
||
stops at a cap rather than growing to it.
|
||
|
||
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
|
||
operator: the person who said the thing should not have to ask someone to read
|
||
it before it can go.
|
||
|
||
### Disk, and taking back only what llmgw put there
|
||
|
||
Disk was the one resource nothing reported. `statfs` was already being called
|
||
on every model directory, but only to ask whether the filesystem was a network
|
||
one — the free space in the same struct was discarded, so the only way to find
|
||
out why a node was filling up was to ssh to it.
|
||
|
||
Nodes now report it, per filesystem rather than per configured directory: free
|
||
space belongs to the disk, and two model directories on one disk are one pool
|
||
of bytes. Shared filesystems are kept out of a node's own figure, because space
|
||
on a filer belongs to everyone mounting it and twenty nodes would otherwise
|
||
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
|
||
Prometheus, and a node that reported nothing reads as unknown everywhere — an
|
||
older agent must not look like a full disk.
|
||
|
||
A profile may declare `min_disk_gb`, and a node without the room is no longer a
|
||
candidate, with a reason `explain-placement` prints. Skipped on a node that
|
||
already holds the weights: it needs no room for them, and without that the node
|
||
best placed to run a model is the one most likely to be rejected.
|
||
|
||
`ctl weights` lists model files llmgw downloaded for models no longer in the
|
||
catalog, and `--collect` deletes them. The design is mostly about what it will
|
||
not do. Only files llmgw fetched are candidates, and that is a calculation
|
||
rather than a heuristic — the cache name is a hash of the source, so the wanted
|
||
set is computable exactly. Anything an operator put on a node is left alone
|
||
however unreferenced it looks. Each node checks every path again before
|
||
unlinking it, refusing anything outside its cache directory or without the
|
||
cache name, so a control plane that is wrong cannot talk a node into losing a
|
||
model that took a week to copy.
|
||
|
||
### Fixes
|
||
|
||
- **A capability was lost at an indirection.** The controller is handed a
|
||
forwarding dispatcher, and reaches past its interface by type assertion to
|
||
find what lies beyond scheduling. Deleting weights was implemented on the
|
||
agent service and not on the forwarder, so it was simply absent at runtime —
|
||
with the whole suite green, because the dispatchers the tests use do not
|
||
implement it either and its absence there is the expected case.
|
||
- **`ctl tool-check` blamed the engine for what the model did.** It had one
|
||
verdict for a failed attempt and told operators not to try another model.
|
||
Measured: the engine renders the tool prompt correctly, the coder models do
|
||
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
|
||
the same engine returns a populated `tool_calls` array. A failed attempt is
|
||
now two verdicts, told apart by whether the reply used the wrapper.
|
||
|
||
## v0.6.0 — 2026-09-18
|
||
|
||
Phase 4: a replica can be bigger than one machine, and the fabric it spans
|
||
decides where it is allowed to go. Around that, everything a cluster needs
|
||
before anyone other than its operator is let near it — keys, budgets, a way in
|
||
through a browser, and enough visibility to answer "why is nothing scheduling".
|
||
|
||
### Multi-node replicas
|
||
|
||
A replica is now a set of shards with ranks, a leader and a bootstrap address,
|
||
rather than one process. Two invariants are enforced rather than documented: a
|
||
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
|
||
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
|
||
several times slower, and nothing says which.
|
||
|
||
A replica missing a shard is not ready. It was previously derived from the
|
||
shards that had reported, so a two-node replica with one node up looked
|
||
servable and took traffic into a model that could not answer.
|
||
|
||
Adoption after a restart happens one agent at a time, because that is how
|
||
agents reconnect. Merging them as they arrive is what keeps a multi-node
|
||
replica whole instead of being torn into fragments that are then killed as
|
||
strays — and an adopted engine is health-checked before it is called ready,
|
||
rather than being assumed alive because its process exists.
|
||
|
||
### Fabrics are distribution domains too
|
||
|
||
An isolated fabric is not only a compute boundary: weights present in one are
|
||
not present in the other. Node weight inventories are reported and summarised
|
||
per fabric, so "the model is on the cluster" is never mistaken for "the model
|
||
is reachable from the nodes that would run it". A fabric name that matches
|
||
nothing a node actually has is refused at registration rather than becoming an
|
||
unschedulable replica hours later.
|
||
|
||
### Who may use it
|
||
|
||
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
|
||
still no cgo). Rate limits and budgets are enforced on the request path; usage
|
||
is written off it, in batches, so accounting never adds latency to inference.
|
||
Budgets survive a restart — counters are rebuilt from the store, because a
|
||
limit anyone can clear by bouncing the process is not a limit.
|
||
|
||
A spent budget is now served a cheaper model where the catalog offers one,
|
||
rather than refused. For a team trying models out, being dropped to something
|
||
smaller is a far better answer than a wall, and the response says which model
|
||
answered.
|
||
|
||
People can sign up, be approved, and issue and revoke their own keys. A key is
|
||
shown once; only its hash is kept.
|
||
|
||
### The browser surface
|
||
|
||
A landing page that says how to connect, a read-only cluster dashboard, and an
|
||
in-browser chat that never puts an API key in the browser. The dashboard shows
|
||
who is using the cluster and what they are running — per person and per model,
|
||
with requests, tokens, errors, spend and latency.
|
||
|
||
All of it is served from the client-facing port behind a session, deliberately
|
||
not from the admin port, which also serves pprof — and pprof can dump the heap
|
||
that holds every API key in plaintext.
|
||
|
||
### Models without a restart
|
||
|
||
The catalog is live. Models can be added and removed from the command line or
|
||
the API, take effect immediately, and outlive a restart; configuration still
|
||
wins over the store, and the whole catalog is validated before anything is
|
||
written.
|
||
|
||
A model is only listed once it can actually serve. It was listed as soon as it
|
||
was known, so the first thing a new user did was ask for a model that returned
|
||
503.
|
||
|
||
### Loading on demand, and letting go
|
||
|
||
A model whose scaling allows it is started when someone asks for it, and
|
||
stopped when nobody has for a while. An idle replica is evicted to make room
|
||
for one that was asked for, with guards that exist because each is a way to
|
||
lose work quietly: only on-demand models, only single-node replicas, never one
|
||
that has never been used, and never one younger than its minimum lifetime.
|
||
|
||
Unknown is not idle. A replica with no recorded use is left alone rather than
|
||
treated as idle since the epoch.
|
||
|
||
### Knowing whether it will run before it runs
|
||
|
||
- `ctl plan` works out whether a model fits the cluster and how it would be
|
||
split, in the precision the chosen engine can actually serve — recommending
|
||
a quantisation the engine cannot load would cause the failure the command
|
||
exists to prevent.
|
||
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
|
||
installed. Flags drift between releases, and the alternative is finding out
|
||
during a model load.
|
||
- Placement now knows a model's context length and memory footprint, and node
|
||
RAM, so a CPU profile is not placed on a machine that cannot hold it.
|
||
- Placement waits for the fleet to settle after a restart rather than deciding
|
||
against a half-reported cluster.
|
||
|
||
### Weights
|
||
|
||
Nodes report the model files they can see, without llmgw owning the storage:
|
||
a shared filer, an image exported over the fabric, rsync or a copy made by hand
|
||
all look the same to placement, which only cares whether the files are there.
|
||
Reporting nothing means "not asked to look", not "has nothing".
|
||
|
||
Where the simple arrangement is not enough, a node can fetch a model by URL —
|
||
single-flight, resumable, written to a temporary name and renamed atomically,
|
||
so a killed download never leaves something that looks like weights.
|
||
|
||
### Seeing the fleet
|
||
|
||
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
|
||
scrape time rather than from a background goroutine, and served on their own
|
||
endpoint rather than through the one that also exposes pprof. The same view is
|
||
in the dashboard, so nothing else needs installing to see it.
|
||
|
||
### Onboarding a node
|
||
|
||
Adding a node is one command. There are systemd units, a private CA and a
|
||
from-nothing install guide that starts with a machine that has no CUDA and no
|
||
binary.
|
||
|
||
### llama.cpp, and CPU profiles
|
||
|
||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||
can all be exercised on ordinary machines.
|
||
|
||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||
allocation to stop two of them sharing a machine's cores.
|
||
|
||
### Metric mappings are per engine
|
||
|
||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||
and a single global mapping left half of it unmeasurable.
|
||
|
||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||
violation. `Running` and `Waiting` can never be omitted.
|
||
|
||
### Mutual TLS on the agent stream
|
||
|
||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||
node id does not match its certificate's common name. Public authorities cannot
|
||
supply these: they stopped issuing the `clientAuth` usage.
|
||
|
||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||
agent turn it on.
|
||
|
||
### Fixes
|
||
|
||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||
context and then asked that context whether it was shutting down, so every
|
||
stream error read as a clean close: no message, and the backoff reset.
|
||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||
level, so a cluster could look healthy while every routing decision was made
|
||
blind. It is now a warning, once, with a matching line when it clears.
|
||
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
|
||
over about 2 KB was undercounted, so tokens-per-minute limits let through far
|
||
more than they were set to.
|
||
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
|
||
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
|
||
bfloat16 does not fit two 80 GB cards. It does.
|
||
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
|
||
already written the response header, which is the one point at which headers
|
||
stop being changeable. A client had no way to tell a fallback from the model
|
||
it asked for.
|
||
- **Fleet metrics published nothing.** They were registered after the registry
|
||
had been built, so the collector existed and was never scraped.
|
||
- **GPU discovery broke on older drivers.** A field added for utilisation was
|
||
required rather than optional, so a driver that does not report it produced no
|
||
inventory at all instead of an inventory without that number.
|
||
- **Replica ids and ports were reused** while the processes holding them were
|
||
still alive.
|
||
|
||
## v0.5.0 — 2026-09-16
|
||
|
||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||
control plane places replicas, and `ctl` makes all of it visible.
|
||
|
||
This release covers both phases because their exit criteria depended on the same
|
||
piece — the agent stream — and neither was demonstrable without it.
|
||
|
||
### ctl
|
||
|
||
```
|
||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||
```
|
||
|
||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||
the answer when a node looks idle but nothing schedules onto it.
|
||
- `explain-placement` prints every candidate considered and the reason it was
|
||
rejected. "Unschedulable" on its own says nothing about which of six
|
||
constraints bit.
|
||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||
is placed elsewhere, and the original is stopped only once the replacement is
|
||
serving. Bounded by a grace period, because draining the only node that can
|
||
host a model would otherwise never finish and leave it undrainable.
|
||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||
|
||
The management API is on the admin listener, not the client-facing one. The two
|
||
have different audiences: one serves inference to whoever holds an API key, the
|
||
other lets a caller cordon nodes.
|
||
|
||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||
not have. The failure path already ships an engine log tail with every status
|
||
report, so the case that matters most is covered; reading a healthy shard's
|
||
output on demand is not.
|
||
|
||
### Phases 2 and 3, in detail
|
||
|
||
Written as the work landed, which is why it reads as a log rather than as
|
||
notes. Everything here shipped in v0.5.0.
|
||
|
||
### Added
|
||
|
||
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
|
||
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
|
||
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
|
||
without IB is not an error — it is a single-node island.
|
||
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
|
||
would force the whole binary, server included, to be built with a C
|
||
toolchain. Health flags anything that should keep a card out of placement:
|
||
uncorrected ECC errors, and memory already held by another process.
|
||
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
|
||
always match the marketing name: an L40S is sold as 48GB and reports 44.99
|
||
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
|
||
box, because the number decides whether a model fits.
|
||
- **Engine adapters** for vLLM and SGLang: command and environment
|
||
construction, kept free of process handling so it is testable without GPUs.
|
||
vLLM supports both Ray and multiprocessing launch modes, because which one
|
||
works depends on the version.
|
||
- **NCCL environment generation**, the part plan §5.3 marks critical.
|
||
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
|
||
NCCL pick an adapter on another fabric where the peer is unreachable, and the
|
||
job hangs rather than failing. Bootstrap is pinned to the management
|
||
interface for the same reason.
|
||
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
|
||
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
|
||
compute capability 8.9 rather than silently falling back to a slower path.
|
||
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
|
||
process group and are stopped as a group, because engines fork workers that
|
||
hold GPU memory and outlive a signal sent only to the leader. Bounded log
|
||
tail, a state machine that separates Starting from Loading (a 70B model takes
|
||
minutes, and an operator needs to know that is normal), and orphan adoption
|
||
through pid files fingerprinted with the command line — pids get reused, and
|
||
adopting an unrelated process is worse than leaking the original.
|
||
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
|
||
says the weights loaded. Routing on `/health` alone sends traffic into a model
|
||
that is still loading.
|
||
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
|
||
as a table or JSON. Run it before adding a node: it is the fastest way to find
|
||
a card another process is using, or a port with no subnet manager.
|
||
|
||
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
|
||
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
|
||
a tensor-parallel group off the slow path inside a node. `Discover` falls back
|
||
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
|
||
topology. CI compiles the tagged build so it cannot rot unnoticed.
|
||
|
||
### Not verified
|
||
|
||
The NVML path compiles but has never run against a driver — there is no GPU
|
||
machine to try it on. Treat the NVLink domains it reports as unverified until
|
||
someone runs `llmgw agent inventory` on a real node.
|
||
|
||
- **Control plane wiring** (`internal/control`): Ready replicas become router
|
||
endpoints, with no static configuration involved. The router keeps live state
|
||
for endpoints that survive an update — inflight counts, breaker state, drain
|
||
rate — so a replica that was already serving does not start from cold because
|
||
another one moved. The sink is an interface, so when the router and control
|
||
plane split into separate processes this becomes a watch stream and nothing
|
||
above it changes.
|
||
|
||
### Phase 3 exit criterion, measured in `test/e2e`
|
||
|
||
A node hosting a live replica is killed while four clients stream continuously.
|
||
The control plane marks it unreachable, then lost, rebuilds the replica on a
|
||
survivor, and republishes. Across three runs: **~370 requests completed, zero
|
||
failures**, and no replica remained on the dead node.
|
||
|
||
This proves the control logic, not the transport: the agent fleet is simulated,
|
||
and the real path still needs the stream.
|
||
|
||
### A real bug the race detector caught
|
||
|
||
`startReplica` read the replica's fields after publishing it to the map, and
|
||
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
|
||
status reports, which arrive from another goroutine and mutate exactly those
|
||
fields. Under the simulated fleet the reports came back fast enough to hit the
|
||
window every time. Everything is snapshotted under the lock now and the
|
||
functions work from values.
|
||
|
||
### The agent stream
|
||
|
||
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
|
||
carrying registration, heartbeats and shard status upward, and start, stop and
|
||
drain commands downward. The agent dials out, because GPU nodes sit behind
|
||
firewalls far more often than a control plane does and an outbound connection
|
||
needs no inbound rule on every node in the fleet.
|
||
- **Registration carries the whole inventory**, so there is no separate
|
||
discovery call and a reconnecting agent refreshes what the scheduler knows.
|
||
The fabric labels travel with it — they cannot be inferred from the wire, so
|
||
losing them in transit would silently disable multi-node placement.
|
||
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
|
||
the reconciler's dispatcher: it holds the open streams, so it is the only
|
||
thing that can deliver a command to a node.
|
||
- **Agent** (`internal/agent`) validates a profile before launching anything —
|
||
an engine that refuses a profile does so after minutes of loading — then
|
||
supervises the process and reports every transition, including the engine's
|
||
output tail on failure.
|
||
- **Reconnects forever with jittered backoff.** A control plane restart must not
|
||
require touching every GPU node, and a fleet must not reconnect in lockstep
|
||
and knock the control plane over again. Engines keep serving throughout: the
|
||
router talks to them directly.
|
||
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
|
||
any model declares a profile. The static endpoint list remains for
|
||
router-only runs against sim-engines.
|
||
|
||
### Verified with real binaries
|
||
|
||
Server and agent as separate processes over a real stream: the agent registers,
|
||
a replica is placed, an engine is launched through the vLLM adapter's actual
|
||
argument construction, streaming traffic is served through the router, and
|
||
killing the agent moves the node through unreachable to lost with placement
|
||
explaining exactly why it can no longer run anything.
|
||
|
||
### Surviving a control plane restart
|
||
|
||
Not persisting replica state left one hole that had to be closed before it
|
||
mattered: restarting the control plane orphaned live engines. They kept holding
|
||
GPUs the inventory believed were free, so the next placement failed to allocate
|
||
with an error naming the new process rather than the old one.
|
||
|
||
The fix is on the agent, not in a database, and it cannot disagree with reality
|
||
the way a stale row can:
|
||
|
||
- The supervisor now writes the full shard spec, not just a pid and command
|
||
line. A pid identifies a process; the spec is what lets it be described back
|
||
to the control plane and matched to a replica.
|
||
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
|
||
waited on — it is not this process's child — so it is watched by polling, and
|
||
stopped by signalling its group directly. Its earlier output is gone, and the
|
||
log says so rather than presenting emptiness as silence.
|
||
- Every `Register`, not only the first, reports what the agent is running. A
|
||
reconnect may be a control plane restart and has to recover the same way.
|
||
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
|
||
Anything whose model or profile left the catalog is stopped: nothing will
|
||
route to it and it is holding cards.
|
||
- An engine the agent cannot identify — a corrupt state file — is killed. A
|
||
process holding GPUs that nothing can describe is worse than no process,
|
||
because placement will hand those GPUs out again.
|
||
|
||
One mechanism covers agent restarts and control plane restarts.
|
||
|
||
### A test that was passing for no reason
|
||
|
||
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
|
||
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
|
||
tracking hijacked connections. The agent never reconnected and the assertion —
|
||
on the connection map, which still held a stale entry — passed anyway. Tests now
|
||
sever connections through their own listener and count opened streams.
|
||
|
||
### Multi-node launch
|
||
|
||
Placement and environment generation had landed early, which made multi-node
|
||
look further along than it was: a multi-node replica could not come up at all.
|
||
Three things were missing, and all three came down to two assumptions that do
|
||
not hold once a replica spans nodes.
|
||
|
||
**Not every rank serves HTTP.** The agent health-checked every shard at its own
|
||
port, but only the leader answers. Workers timed out, reported Failed, and took
|
||
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
|
||
worker is judged by staying alive instead. That is weaker evidence on purpose —
|
||
the replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined.
|
||
|
||
**A shard is not always one process.** `Engine.Commands(spec)` returns a
|
||
sequence: every command but the last is a sidecar kept alive for the shard's
|
||
lifetime and torn down with it, and the last one's exit is the shard's exit.
|
||
vLLM under Ray is why — the leader runs a Ray head and then the server against
|
||
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
|
||
had been written and tested but never called, so `launch_mode: ray` would have
|
||
run `vllm serve` against a cluster that did not exist.
|
||
|
||
A command can also require the rendezvous point to be listening before the next
|
||
one starts, bounded by a timeout so a head that never appears fails the shard
|
||
rather than hanging it. Sidecars are killed with the shard: a Ray head left
|
||
behind holds the port the next replica needs.
|
||
|
||
**Start order was backwards.** Workers were started first, on the theory that
|
||
the head should wait for them. Both engines put the rendezvous point on rank 0,
|
||
so a worker started first has nothing to connect to and spends its startup
|
||
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
|
||
needs, so the leader can start immediately and waits on its own.
|
||
|
||
None of this is verified against real engines. It is tested with stub processes;
|
||
the Ray sequencing and which launch mode a given vLLM version wants can only be
|
||
confirmed on hardware.
|
||
|
||
### Still missing for phase 2
|
||
|
||
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
|
||
network.
|
||
|
||
### Phase 3: placement
|
||
|
||
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
|
||
their fabric attachments, and what is allocated where. A re-registering agent
|
||
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
|
||
without doing so either. A GPU with an ECC fault or another process on it is
|
||
not counted as free. Replicas are released whole, because they fail whole.
|
||
- **Placement** (`internal/control/placement`): picks the highest-priority
|
||
profile that fits and explains every candidate it rejected. Two constraints
|
||
are enforced structurally rather than checked afterwards — there is no code
|
||
path that can combine nodes from two fabrics, and only GPUs of the profile's
|
||
class are ever considered.
|
||
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
|
||
large models that cannot be split. Multi-node replicas get the opposite
|
||
treatment: they are the least movable thing in the cluster, so they take the
|
||
emptiest machines.
|
||
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
|
||
domain where the topology is known. Without the NVML build it falls back to
|
||
the lowest free indices, which is usually right but only by accident of how
|
||
vendors number slots.
|
||
- **Property-based invariant tests** (plan §13): 400 randomly generated
|
||
clusters, checking on every placement that a multi-node replica lives on one
|
||
fabric, that each shard receives only that fabric's adapters, and that no
|
||
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
|
||
The table-driven tests cover the cases someone thought of; this covers the
|
||
ones nobody did.
|
||
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
|
||
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
|
||
rather than at request time — a typo in a fallback should stop a deployment,
|
||
not surface during the outage when the fallback is finally needed. The
|
||
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
|
||
load instead of minutes into model loading.
|
||
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
|
||
loop. It places one replica per pass, because placement depends on what is
|
||
already allocated and starting several from one snapshot would hand them the
|
||
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
|
||
vLLM under Ray needs its workers attached before the head serves.
|
||
- **Replica failure is whole-replica failure.** One dead shard makes the others
|
||
useless — they wait on collectives that will never complete while holding
|
||
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
|
||
exponential backoff carried on the model rather than the replica it just
|
||
destroyed.
|
||
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
|
||
blip must not tear down replicas that are still serving, and rebuilding a
|
||
multi-node replica costs tens of minutes of model loading.
|
||
- **Scale-down drains before stopping**, so in-flight requests finish instead of
|
||
being cut off mid-stream.
|
||
|
||
## v0.3.0 — 2026-09-16
|
||
|
||
Flow control. A batch workload no longer starves interactive traffic.
|
||
|
||
### Added
|
||
|
||
- **Admission control** (`internal/router/admission`), inserted between quota
|
||
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||
still answers *where* it goes. Keeping them apart is the point: an engine
|
||
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||
no amount of endpoint selection rescues the request behind it.
|
||
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||
nearly full cache, or the reverse.
|
||
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||
"flow up to the healthy limit"; above 1 means never wait.
|
||
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||
one caller sending a thousand requests occupies one slot in the rotation
|
||
rather than a thousand.
|
||
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||
(504), because sending the GPU work the client has already abandoned helps
|
||
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||
`x-llmgw-queue-ms` when a request waited.
|
||
|
||
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||
|
||
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||
With admission switched off the same load pushes it to **55–60x**. Both
|
||
directions are asserted, so the test fails if admission regresses *and* if the
|
||
load generator ever stops being a real load.
|
||
|
||
### Two lags, two different fixes
|
||
|
||
Getting this right took longer than the queueing itself:
|
||
|
||
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||
count does not — but it includes *running* requests, and treating those as
|
||
queue depth makes a healthy engine look overloaded every interval. What is
|
||
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||
the metric contract, so it is learned as the high-water mark of observed
|
||
`running`.
|
||
- Between admission granting a request and the router dispatching it, the
|
||
request is invisible to every signal. A simultaneous burst therefore passed
|
||
the check en masse — exactly the batch arrival pattern this layer exists to
|
||
survive. The controller now counts its own outstanding leases against
|
||
dispatched inflight.
|
||
|
||
### Known limitations
|
||
|
||
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||
Anyone who can reach the port can promote themselves to platinum.
|
||
- The tenant key is the session id, for the same reason.
|
||
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||
|
||
## v0.2.0 — 2026-09-16
|
||
|
||
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||
engine replicas with an OpenAI-compatible API.
|
||
|
||
### Added
|
||
|
||
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||
propagates cancellation: when a client disconnects, the upstream request is
|
||
aborted so the GPU stops working. Separate deadlines for dial, response
|
||
headers, first token, inter-token idle and total duration — a wedged engine is
|
||
indistinguishable from a slow one without the idle watchdog.
|
||
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||
reached the client; once bytes are out, splicing two responses together would
|
||
be worse than the short stream. Models can name fallback models, and the
|
||
response says which one served it via `x-llmgw-served-model`.
|
||
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||
Only routing-relevant fields are parsed; the rest of the body passes through
|
||
untouched.
|
||
- **Usage accounting that does not depend on the client.** The router can ask
|
||
engines for usage the client never requested, then drop that chunk on the way
|
||
back, so token counts are recorded either way.
|
||
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||
duplicate endpoint ids and aliases that would shadow another model — both
|
||
would otherwise misroute silently at request time.
|
||
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||
without seeing what the scorers actually produced. Served on a separate admin
|
||
listener alongside health and pprof.
|
||
|
||
### Phase 1 exit criterion, measured in `test/e2e`
|
||
|
||
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||
- Every client disconnecting frees the KV blocks on every engine.
|
||
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||
|
||
### Known limitations
|
||
|
||
- No admission control, so a batch workload can still starve interactive
|
||
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||
|
||
## v0.1.0 — 2026-09-16
|
||
|
||
First tagged version. It is useful for exactly one thing: developing and testing
|
||
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
||
returns "not implemented".
|
||
|
||
### Added
|
||
|
||
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
||
that models the behaviours a router has to cope with:
|
||
|
||
- continuous batching with a bounded concurrent sequence count
|
||
- KV cache block accounting, queueing when blocks run out, including
|
||
head-of-line blocking
|
||
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
||
effect on time to first token
|
||
- per-token latency that grows with batch size
|
||
- dropping a request and freeing its blocks when the client disconnects
|
||
- `/metrics` under vLLM's real metric names
|
||
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
||
|
||
**Routing primitives** — libraries, not yet wired into a server:
|
||
|
||
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
||
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
||
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
||
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
||
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
||
detection and drain rate derived from engine counters.
|
||
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
||
endpoints missing any of the mandatory three. The metric name mapping is
|
||
overridable, because names drift between engine versions.
|
||
- `router/breaker`: cooldown and circuit breaking in one component, with
|
||
exponential backoff and half-open probes.
|
||
|
||
### Known limitations
|
||
|
||
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
||
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
||
- No admission control, so a batch workload can still starve interactive traffic
|
||
once the router exists. That is what v0.3.0 is for.
|
||
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
||
|
||
### Notes
|
||
|
||
- The project language is English throughout: code, comments, tests, commits and
|
||
the design document.
|
||
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
||
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|