Files
llmgw/CHANGELOG.md
T
Ilker ManapandClaude Opus 5.5 f13b18149a Refuse weights the nodes' llama.cpp cannot load, before fetching them
kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.

internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.

Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.

Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:44:36 +02:00

1183 lines
60 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
### An organization's own page
A member of an organization can be made its **manager** (`users join --org …
--role manager`, or the star on the people page). A manager gets `/org`: the
organization's usage over time by team, project or person, its teams and
projects with limits and spend, and its members — read-only, and only what was
spent in its projects. Administrators can open every organization's page.
### How a model runs, from the mirror page
Each model a mirrored file is serving now shows how it runs — always on, on
demand, or stopped — and it can be changed there, without the CLI. The serve
form asks the same: on demand loads the model when somebody asks for it and
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
mode and says to change it there, because the file is read on every start.
### Weights the nodes cannot load are not fetched
`kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
was downloaded, catalogued and placed, and failed on every node it was tried on.
The architecture was in the first kilobytes of the file all along. Now:
- The mirror page's registry listing shows each file's architecture and, for
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
it. A vision projector (`mmproj`) is marked as not a model on its own.
- A fetch reads the file's header first — a range request, not a download —
and refuses one that cannot be served, unless "fetch anyway" is ticked.
- Serving from the mirror and `llmgw ctl model add` refuse the same.
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
not refused, only not checked.
### Fixes
- Usage by team, project or organization, and the spend on the quotas page,
left out every request made on the chat page: they were joined through the
key, and the chat page has none. They now count where a request was spent.
- `/org` was answered with "no api key was provided" for a signed-in browser;
a test now drives every session page through the authentication middleware.
### People belong to many places, or to none
A person was in exactly one project, chosen at approval, and could not use the
gateway before that. Now:
- **Memberships.** A person can be a member of any number of organizations,
teams and projects, or of none. A member of a team or organization may use
every project in it. `llmgw users join/leave`, and the people page adds and
removes them from either side. Approving no longer requires a project.
- **Personal keys.** A key can belong to a person with no project. On
`/profile` a person chooses what a new key spends against: one of their
projects, or Personal. `llmgw keys create --for <person>` does the same.
`auth.personal` sets default limits for personal use, or turns it off.
- **Limits for people.** `--scope user` limits a person across every key,
project and the chat page. `--per-member` on an organization, team or
project gives each member their own share there. Both are on the quotas
page, and are enforced alongside the tree, each in its own pool.
- **The chat page** asks what to spend against when a person has more than one
choice, and checks it on every message.
- **Spend is recorded with who spent it**, so a person's budget survives a
restart, personal use and the chat page included.
Leaving a project revokes the keys the person made there, unless another
membership still reaches it. A personal key keeps no conversations or notes:
those are kept per project.
**Upgrading.** Migration 0010 turns everyone's project into a membership and
attributes past usage to people where a key had an owner. On SQLite it rebuilds
the keys table; it was run against a copy of a production PostgreSQL store
before release. `llmgw users move` is gone: use `join` and `leave`.
### One menu on every page
Each page drew its own menu, and no two agreed: different links, in a different
order, the page you were on sometimes listed and sometimes not. Every signed-in
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
left for an administrator, Chat, Your keys and Sign out on the right for
everyone — with the current page marked. The chat and key pages name the
person's project by its path rather than its id.
### The quotas page says what actually applies
A branch that sets no limit of its own used to read "—" in every column while
its organization held it to sixty requests a minute. Each limit now shows the
value it is held to from above, muted and marked ↑, with where it comes from on
hover; a larger value set below a stricter one says what caps it, for rate and
token limits as well as budgets. Spend and budget share a column with a meter
that turns red past 90%. The tree is indented like the people page, opening a
form or saving one no longer reloads the page, and the explanations moved under
"How limits combine". Projects are placed under their team by id rather than by
the start of their path, which misplaced them when two organizations shared a
name.
## v0.9.2 — 2026-09-28
### A new control plane serves a model by itself
`llmgw server init` now also runs a model mirror on the same machine, makes the
machine a node of its own control plane, and fetches Qwen2.5 0.5B into the
mirror and serves it there — on the machine's NVIDIA GPU when `nvidia-smi` finds
one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a
CPU profile as the fallback. An NVIDIA card on the PCI bus with no working
driver is reported. `--no-mirror`, `--no-node` and `--no-model` leave these out,
and every step is skipped on a rerun when it is already done.
Not yet run end to end on a real machine.
## v0.9.1 — 2026-09-28
### The mirror page is one line per model
What the mirror holds used to take two table rows per file — seven columns and
the address under them — so a sharded model filled the screen. Now each version
is one line: model and version, the file (or "4 files" for a set), total size,
and which catalog models use it. The address moved into a "copy address" button;
the checksum is in the file's tooltip. A version already in use is not offered
for serving again, and removing one warns that the catalog uses it. The header
counts versions rather than files.
## v0.9.0 — 2026-09-28
### Managing people and the tenancy tree
Organizations, teams and projects can now be renamed, moved and removed, and an
approved person can be moved to another project or removed — none of which
could be done at all before, from the web or the CLI.
- **`/admin/users`** shows the whole tree, each project opening onto its people
and keys, with approve, move, suspend, reinstate and remove for people,
revoke for keys, and add, rename, move and remove at every level. A search
over every account sits below it. Changes update the page in place (htmx,
embedded in the binary) and it still works with scripts off.
- **`llmgw org`** — `tree [--people]`, `add`, `rename`, `move`, `remove`.
- **`llmgw users move`** and **`llmgw users delete`**.
- **`llmgw keys list --all`** lists every key with its project and owner.
- Every command that takes a project, team or organization accepts an id, a
path such as `Acme/platform/gateway`, or a unique name, and `--project` may
be omitted when there is only one. Users are named by id or email.
Moving a person takes the keys they made with them; keys an operator issued stay
with the project. Only empty things are removed, and a project that ever had a
key is never removed, because usage history refers to it.
### Changes
- **`--project` no longer defaults to `default`** in `keys create`, `keys list`,
`users approve`, `users create`, `keys threads` and `keys memory`: a tree
built from the web has no project of that name. With one project it is
chosen; with several the command lists them.
- **`keys limits --budget` is in whole units**, as the quotas page takes it,
rather than micro-units. `--budget 50000000` now means fifty million units;
divide existing scripts' figures by 1 000 000.
### Fixes
- `keys init` could silently move an existing team or project to another
organization or team by redeclaring it. It now refuses.
- An account rejected while pending and then reinstated became active with no
project, and failed at its first key. Reinstating one is refused; approving
gives it a project. Rejecting a signup now removes it.
## v0.8.8 — 2026-09-28
### Fixes
- **`server init --postgres` failed on Rocky Linux** with "postgresql-setup:
executable file not found". The PostgreSQL project's own packages (PGDG)
provide `postgresql-server` too, so dnf can answer that name with
`postgresql16-server`, which names its unit `postgresql-16.service` and keeps
its tools in `/usr/pgsql-16/bin`. init now finds the server unit systemd has,
reads the data directory from it, and runs `initdb` itself when the directory
is empty. A PostgreSQL that was already installed is used as it is.
### Known gaps
Not yet run on a real machine. Three arrangements are not handled: a
PostgreSQL initialised with `ident` for TCP logins (the connection check fails
and says so), one listening on a port other than 5432, and several PGDG
versions where the highest is not the one on 5432.
## v0.8.7 — 2026-09-28
### PostgreSQL in one command
```
sudo llmgw server init --postgres
```
Installs PostgreSQL with the machine's own package manager when there is none,
creates the `llmgw` role and database with a generated password kept in
`/etc/llmgw/secrets/store-dsn`, and checks that it connects. On a control plane
already running on SQLite it stops the gateway, copies the store across so
every account and key survives, points `auth.store` at PostgreSQL and starts
the gateway again; if the move fails, the gateway is started again on SQLite.
Given on the first run, there is never a SQLite store.
`server.yaml` is edited in place with its comments intact, and the previous
file is kept beside it. A PostgreSQL that init did not install is left sized as
it was, since something else may depend on it.
The PostgreSQL installation itself has not yet run on a real machine; the move,
the configuration edit and the rerun behaviour are tested.
### Fixes
- **A new control plane with no models did not start.** A configuration with
no static endpoints and no models was refused as having nothing to serve, so
under systemd the unit restarted every two seconds and nothing listened on
8080. It now runs as a control plane with an empty catalog.
## v0.8.6 — 2026-09-28
### Reverse proxy configurations
`deploy/proxy/` has a Caddyfile and an nginx configuration for the client
port, and both are attached to every release beside the binaries. They do not
buffer streamed replies, their read timeout outlasts a model's wait before its
first token, their body limit matches the router's, and they set
`X-Forwarded-Proto` and `X-Forwarded-Host` themselves — llmgw marks session
cookies `Secure` from the first and shows the second as the base URL on its
landing page.
## v0.8.5 — 2026-09-28
### A control plane in one command
```
sudo llmgw server init --admin you@example.com
```
Creates the llmgw user and its directories, the certificate authority, the
control plane's certificate for the machine's hostname and every address it
has, `server.yaml`, the default project, and an administrator whose password is
generated and printed once — then installs the systemd unit, starts it and
waits until it answers. What used to be two steps of install.md is now this.
Each step is skipped when its result exists, so running it again finishes an
install that stopped and changes nothing on one that did not. `--host` adds an
address the machine cannot see of itself (a DNS alias, a NAT or floating
address) and reissues the certificate keeping the old ones. It never overwrites
`server.yaml` or an authority, and certificates issued by an authority kept on
another machine are left as they are.
### Fixes
- **The configuration install.md showed could not start.** An empty catalog
without `control.enabled: true` is refused as having nothing to serve. The
documentation and the file `init` writes both set it.
## v0.8.4 — 2026-09-28
### Node logs on the admin page
Each node on the cluster page links to its logs — the agent's own, and the
output of every engine running there — with an option to follow them. `llmgw
ctl logs NODE` reads the same. They come over the agent's existing stream, so
no ssh, jump host or root is needed to see what a node did. The agent keeps its
last 2000 lines in memory; each engine its last few hundred.
## v0.8.3 — 2026-09-28
Found by building a node from nothing with v0.8.2.
### Fixes
- **A new node's first engine install failed on apt's lock.** cloud-init and
unattended-upgrades run apt in a machine's first minutes, and
`DPkg::Lock::Timeout` does not cover the lock `apt-get update` takes. A
package command held up by a lock now waits for up to ten minutes, and says
so; a failed install is retried after a minute rather than ten.
- **A new node reported no disk** until its first download created the weights
directory. It is created when the agent starts.
- **Onboarding took one control plane address.** It takes several, comma
separated, so a new node fails over like the rest.
## v0.8.2 — 2026-09-28
A node no longer runs whatever engine somebody once put on it. It installs its
own.
### Engines are installed by the node
`engines:` in server.yaml names the engines every node has and their versions,
defaulting to the llama.cpp build this release was checked against (b10991).
Each agent converges on it: it picks the build for its hardware — CUDA with its
runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS
packages that build needs with the machine's own package manager, downloads the
archive and checks it against the sha256 GitHub publishes, unpacks it refusing
anything that would land outside its directory, and runs it. `ldd` names a
missing library or a system too old for the build ("libstdc++.so.6 is too old
(needs GLIBCXX_3.4.30)"); `--version` proves the rest. Installs are checked
again every ten minutes.
An engine is reported only once installed and seen to run, or when the agent's
configuration names it on purpose. A node that has installed nothing yet is not
a candidate for any model, where before it was a candidate for all of them.
`ctl nodes`, the cluster page and placement's refusals say where each
installation stands, and warn about a CPU without AVX2.
Not every system is refused for its name: the build runs on Ubuntu 22.04+,
Debian 12+ and RHEL-family 10, and `ldd` decides. vLLM and SGLang are not yet
installed this way and must still be named in the agent's configuration.
### Onboarding
Onboarding installs llmgw from the CI release for the node's architecture,
checked against the release's checksums, and writes an agent configuration of
only the control plane's addresses and the TLS files.
### Fixes
- A race on the weights fingerprint, which two heartbeat loops touched during a
failover.
## v0.8.1 — 2026-09-28
Nothing here changes a configuration format: every file that worked with
v0.8.0 still works. What changes is how much of it needs writing, and how much
of running the cluster can be done from a page rather than a shell.
### A node describes itself
An agent file used to name its node, its address, the path to llama-server and
its weight directories — different on every node and kept true by hand. The
agent now finds these out: its id is the hostname, its address is the one on
the route to the control plane, llama-server is looked for on PATH and in
`/opt/llama.cpp`, a node without nvidia-smi simply has no NVIDIA cards, and
fetched weights land in `/var/lib/llmgw-weights`, which is always scanned.
Memory is re-read on every heartbeat and installed engines once a minute, so a
VM given more RAM or a node given an engine becomes a candidate without an
agent restart; the control plane logs both. The agent file shrinks to the
control plane's addresses and the TLS files, the same on every node.
### Models from the mirror, without writing YAML
Search the registry from `/admin/mirror`, fetch a sharded model in one action,
and serve a mirrored version from its row — name, source, engine and an
estimated memory requirement filled in. A model can say what it is for, and
`/v1/models` shows it. Placement no longer hands the same memory to two models
or puts an engine on a node that lacks it, and says when a replica runs an
edited profile.
### Tenancy, quotas and usage on pages
The tenancy tree, quotas and budgets are managed from the admin pages; an
organization's budget bounds its teams, a group's quota counts against the
group, and a budget that cannot be spent against says so. `/admin/usage`
charts requests, tokens and spend over time by organization, team, project or
person, and a request is priced by the model that served it.
### Watching the cluster
The cluster page has its own charts, backed by a year of Prometheus history,
with alerts that were made to fire before being trusted — including for the
disk the metrics share with the models.
### Chat
The picker offers each model once rather than once per alias, never asks for
more tokens than the model allows, and offers a model that loads on demand —
sending it a message is what loads it.
## v0.8.0 — 2026-09-27
The store stops being a file on one machine's disk. PostgreSQL, a way to move
an existing installation onto it without reissuing every key, and the two
things a second control plane needs so that running one is safe rather than
clever.
### A second control plane, and a way to fail over to it
Leadership is a lease in the store, so two control planes against one database
cannot both place work — which they otherwise would, both seeing a model
wanting replicas and none running, leaving twice the replicas each holding GPUs
the other believes are free.
A standby also refuses agent streams. Commands travel down the stream the agent
opened, so a standby holding one is a node the leader cannot command, and the
leader's placements would fail with "no agent connected" for a node that is
plainly connected. Agents therefore take a list of addresses rather than one,
trying each in turn: a standby refusing, a stopped control plane refusing the
connection and a crashed one never answering are the same thing from a node.
Measured on the cluster. Leadership moved three seconds after the leader was
stopped — it released the lease on the way out rather than letting it lapse —
and all five agents had registered with the standby six seconds later. The
standby adopted every replica rather than restarting any: six replicas, all
ready, engines that never stopped serving.
Everything is unset-means-leader, so a single control plane and every
deployment that existed before this behave exactly as they did.
What this does not do is move client traffic. The standby served inference
correctly on its own port while the public address returned 502, because
nothing in front of them moved — that needs a load balancer or a shared
address, and it is the remaining piece.
### PostgreSQL
The store runs on PostgreSQL, chosen by what the address looks like: a path is
still SQLite, a `postgres://` URL is not, and `@` followed by a path reads the
address out of a file so a DSN carrying a password stays out of a configuration
that gets copied and pasted into issues. Every existing installation keeps
working untouched.
One set of queries rather than one per database, because the SQL here is almost
entirely portable and two copies would drift one query at a time. The rewrite
of placeholders is a wrapper around the handle rather than a change at sixty
call sites — which found six places reaching past it, including all three
manual transactions and two prepared statements, every one of which worked on
SQLite and would have failed on PostgreSQL.
Exactly one query is not the same sentence in both: the note search, MATCH
against `@@`. SQLite gets full-text search from an FTS5 table kept in step by
three triggers; PostgreSQL derives a tsvector from the row, so there is nothing
to keep in step and no trigger to forget. It uses the 'simple' configuration
rather than English — these are an agent's own notes, in whatever language it
chose, and stemming English rules over Turkish would quietly stop finding
things.
`llmgw keys move-store` copies an existing store rather than starting a fresh
one, because keys are stored hashed: a migration that drops them cannot
recreate them, and every person and every script would need a new one.
Testing it against a real database found three things nothing else would have.
`schema_migrations` is created in Go rather than in a migration file, so it
escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL,
and a millisecond timestamp does not fit. The copy order followed the order the
migrations create tables in, and `users` is created after `api_keys` even
though `api_keys` references it. And `key_claims.sealed` is emptied the moment
a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an
empty blob scans as a typed nil slice and a real NULL as an untyped one, a
difference pgx cannot see, so it arrived as NULL.
The driver is pgx, which is pure Go: the whole binary is built with
CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's,
and a cgo driver would take that away from every deployment, including the ones
staying on SQLite.
## v0.7.0 — 2026-09-25
What a cluster needs to be lived with rather than demonstrated: somewhere to
keep a conversation, somewhere to keep a note, and an honest answer about the
disk underneath both.
### Memory notes for agents, and threads without API keys
Notes an agent writes for itself and reads back in later conversations, offered
as REST (`/v1/memory`) and as MCP tools at `/mcp` (`remember`, `recall`,
`forget`). A note is a key, text and tags, searched by words or tag, private to
a project and to the agent id in `x-llmgw-agent`. llmgw does not inject the
tools into chat requests: an agent that is not given the server never sees
memory.
Off for every project until someone runs `llmgw keys memory --retain-hours N`,
and the retention is required, as it is for threads.
On a gateway with no authentication, a `threads:` block (`store`,
`retain_hours`) now keeps conversations and notes in one shared pool. Before,
`x-llmgw-thread` was silently ignored there; now a gateway configured to keep
nothing refuses it with an explanation instead. The shared pool is for one
trusted machine: anyone who can reach the port and knows an id can read it.
### Conversations that can move between models
A thread a client names with `x-llmgw-thread`, carrying only its new turn.
llmgw replays the stored history to whichever model is serving, which is what
lets a conversation begun on a large model continue on a small one — including
when a spent budget moved it there and the caller never chose.
Off for every project until someone runs `llmgw keys threads`. Until then llmgw
stores no message content anywhere, as it never has, and that default is worth
more than the feature. The retention is required rather than optional, because
a conversation store with no expiry is a transcript of everything a team ever
typed kept forever by omission, and it is measured from the last turn so a
conversation in progress does not expire underneath it.
History that does not fit the serving model is trimmed oldest-first and the
reply says how many turns went, in `x-llmgw-thread-trimmed`. System messages
are never dropped — losing one silently changes what the model is for while it
carries on answering — and neither is the turn the caller just sent.
Threads are scoped to a project, and reading, writing or deleting one that
belongs to another answers exactly as one that never existed: two answers would
let a caller map which ids are real in projects they cannot read.
Capturing the reply is opt-in and bounded, so a request without a thread
accumulates nothing on the hottest path in the system, and one with a thread
stops at a cap rather than growing to it.
`DELETE /v1/threads/{id}` forgets a conversation, as a client rather than as an
operator: the person who said the thing should not have to ask someone to read
it before it can go.
### Disk, and taking back only what llmgw put there
Disk was the one resource nothing reported. `statfs` was already being called
on every model directory, but only to ask whether the filesystem was a network
one — the free space in the same struct was discarded, so the only way to find
out why a node was filling up was to ssh to it.
Nodes now report it, per filesystem rather than per configured directory: free
space belongs to the disk, and two model directories on one disk are one pool
of bytes. Shared filesystems are kept out of a node's own figure, because space
on a filer belongs to everyone mounting it and twenty nodes would otherwise
each report the same free terabyte. It shows in `ctl nodes`, the dashboard and
Prometheus, and a node that reported nothing reads as unknown everywhere — an
older agent must not look like a full disk.
A profile may declare `min_disk_gb`, and a node without the room is no longer a
candidate, with a reason `explain-placement` prints. Skipped on a node that
already holds the weights: it needs no room for them, and without that the node
best placed to run a model is the one most likely to be rejected.
`ctl weights` lists model files llmgw downloaded for models no longer in the
catalog, and `--collect` deletes them. The design is mostly about what it will
not do. Only files llmgw fetched are candidates, and that is a calculation
rather than a heuristic — the cache name is a hash of the source, so the wanted
set is computable exactly. Anything an operator put on a node is left alone
however unreferenced it looks. Each node checks every path again before
unlinking it, refusing anything outside its cache directory or without the
cache name, so a control plane that is wrong cannot talk a node into losing a
model that took a week to copy.
### Fixes
- **A capability was lost at an indirection.** The controller is handed a
forwarding dispatcher, and reaches past its interface by type assertion to
find what lies beyond scheduling. Deleting weights was implemented on the
agent service and not on the forwarder, so it was simply absent at runtime —
with the whole suite green, because the dispatchers the tests use do not
implement it either and its absence there is the expected case.
- **`ctl tool-check` blamed the engine for what the model did.** It had one
verdict for a failed attempt and told operators not to try another model.
Measured: the engine renders the tool prompt correctly, the coder models do
not use the `<tool_call>` wrapper it asks for, and `qwen2.5-3b-instruct` on
the same engine returns a populated `tool_calls` array. A failed attempt is
now two verdicts, told apart by whether the reply used the wrapper.
## v0.6.0 — 2026-09-18
Phase 4: a replica can be bigger than one machine, and the fabric it spans
decides where it is allowed to go. Around that, everything a cluster needs
before anyone other than its operator is let near it — keys, budgets, a way in
through a browser, and enough visibility to answer "why is nothing scheduling".
### Multi-node replicas
A replica is now a set of shards with ranks, a leader and a bootstrap address,
rather than one process. Two invariants are enforced rather than documented: a
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
several times slower, and nothing says which.
A replica missing a shard is not ready. It was previously derived from the
shards that had reported, so a two-node replica with one node up looked
servable and took traffic into a model that could not answer.
Adoption after a restart happens one agent at a time, because that is how
agents reconnect. Merging them as they arrive is what keeps a multi-node
replica whole instead of being torn into fragments that are then killed as
strays — and an adopted engine is health-checked before it is called ready,
rather than being assumed alive because its process exists.
### Fabrics are distribution domains too
An isolated fabric is not only a compute boundary: weights present in one are
not present in the other. Node weight inventories are reported and summarised
per fabric, so "the model is on the cluster" is never mistaken for "the model
is reachable from the nodes that would run it". A fabric name that matches
nothing a node actually has is refused at registration rather than becoming an
unschedulable replica hours later.
### Who may use it
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
still no cgo). Rate limits and budgets are enforced on the request path; usage
is written off it, in batches, so accounting never adds latency to inference.
Budgets survive a restart — counters are rebuilt from the store, because a
limit anyone can clear by bouncing the process is not a limit.
A spent budget is now served a cheaper model where the catalog offers one,
rather than refused. For a team trying models out, being dropped to something
smaller is a far better answer than a wall, and the response says which model
answered.
People can sign up, be approved, and issue and revoke their own keys. A key is
shown once; only its hash is kept.
### The browser surface
A landing page that says how to connect, a read-only cluster dashboard, and an
in-browser chat that never puts an API key in the browser. The dashboard shows
who is using the cluster and what they are running — per person and per model,
with requests, tokens, errors, spend and latency.
All of it is served from the client-facing port behind a session, deliberately
not from the admin port, which also serves pprof — and pprof can dump the heap
that holds every API key in plaintext.
### Models without a restart
The catalog is live. Models can be added and removed from the command line or
the API, take effect immediately, and outlive a restart; configuration still
wins over the store, and the whole catalog is validated before anything is
written.
A model is only listed once it can actually serve. It was listed as soon as it
was known, so the first thing a new user did was ask for a model that returned
503.
### Loading on demand, and letting go
A model whose scaling allows it is started when someone asks for it, and
stopped when nobody has for a while. An idle replica is evicted to make room
for one that was asked for, with guards that exist because each is a way to
lose work quietly: only on-demand models, only single-node replicas, never one
that has never been used, and never one younger than its minimum lifetime.
Unknown is not idle. A replica with no recorded use is left alone rather than
treated as idle since the epoch.
### Knowing whether it will run before it runs
- `ctl plan` works out whether a model fits the cluster and how it would be
split, in the precision the chosen engine can actually serve — recommending
a quantisation the engine cannot load would cause the failure the command
exists to prevent.
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
installed. Flags drift between releases, and the alternative is finding out
during a model load.
- Placement now knows a model's context length and memory footprint, and node
RAM, so a CPU profile is not placed on a machine that cannot hold it.
- Placement waits for the fleet to settle after a restart rather than deciding
against a half-reported cluster.
### Weights
Nodes report the model files they can see, without llmgw owning the storage:
a shared filer, an image exported over the fabric, rsync or a copy made by hand
all look the same to placement, which only cares whether the files are there.
Reporting nothing means "not asked to look", not "has nothing".
Where the simple arrangement is not enough, a node can fetch a model by URL —
single-flight, resumable, written to a temporary name and renamed atomically,
so a killed download never leaves something that looks like weights.
### Seeing the fleet
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
scrape time rather than from a background goroutine, and served on their own
endpoint rather than through the one that also exposes pprof. The same view is
in the dashboard, so nothing else needs installing to see it.
### Onboarding a node
Adding a node is one command. There are systemd units, a private CA and a
from-nothing install guide that starts with a machine that has no CUDA and no
binary.
### llama.cpp, and CPU profiles
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
API without a GPU, so placement, supervision, scraping, scheduling and admission
can all be exercised on ordinary machines.
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
engines that can actually serve without a card may omit it — for vLLM an omitted
class stays an error, because otherwise a typo becomes a replica placed on a
machine with no hardware and diagnosed minutes later during model loading. CPU
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
allocation to stop two of them sharing a machine's cores.
### Metric mappings are per engine
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
than one for the cluster. A mixed fleet has two names for the same three numbers,
and a single global mapping left half of it unmeasurable.
A mapping that leaves a metric unnamed now means the engine does not publish it,
and the field stays zero. Naming one the engine does not serve is still a contract
violation. `Running` and `Waiting` can never be omitted.
### Mutual TLS on the agent stream
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
certificates on both ends, and the control plane refusing any agent whose claimed
node id does not match its certificate's common name. Public authorities cannot
supply these: they stopped issuing the `clientAuth` usage.
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
agent turn it on.
### Fixes
- **An agent told to stop did not stop.** Cancelling the context is not enough —
HTTP/2 watches it while the request body is being written, and a bidirectional
stream spends its life blocked on the read side. Measured on a deployed node:
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
- **An agent could not explain why it was refused.** The session cancelled its own
context and then asked that context whether it was shutting down, so every
stream error read as a clean close: no message, and the backoff reset.
- **A broken metric contract was invisible.** It was logged per scrape at debug
level, so a cluster could look healthy while every routing decision was made
blind. It is now a warning, once, with a matching line when it clears.
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
over about 2 KB was undercounted, so tokens-per-minute limits let through far
more than they were set to.
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
bfloat16 does not fit two 80 GB cards. It does.
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
already written the response header, which is the one point at which headers
stop being changeable. A client had no way to tell a fallback from the model
it asked for.
- **Fleet metrics published nothing.** They were registered after the registry
had been built, so the collector existed and was never scraped.
- **GPU discovery broke on older drivers.** A field added for utilisation was
required rather than optional, so a driver that does not report it produced no
inventory at all instead of an inventory without that number.
- **Replica ids and ports were reused** while the processes holding them were
still alive.
## v0.5.0 — 2026-09-16
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
control plane places replicas, and `ctl` makes all of it visible.
This release covers both phases because their exit criteria depended on the same
piece — the agent stream — and neither was demonstrable without it.
### ctl
```
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
```
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
the answer when a node looks idle but nothing schedules onto it.
- `explain-placement` prints every candidate considered and the reason it was
rejected. "Unschedulable" on its own says nothing about which of six
constraints bit.
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
is placed elsewhere, and the original is stopped only once the replacement is
serving. Bounded by a grace period, because draining the only node that can
host a model would otherwise never finish and leave it undrainable.
- `cordon` is the weaker operation: no new placement, nothing disturbed.
The management API is on the admin listener, not the client-facing one. The two
have different audiences: one serves inference to whoever holds an API key, the
other lets a caller cordon nodes.
`ctl logs` is not here. It needs a request/response pair the agent protocol does
not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
### Phases 2 and 3, in detail
Written as the work landed, which is why it reads as a log rather than as
notes. Everything here shipped in v0.5.0.
### Added
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
without IB is not an error — it is a single-node island.
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
would force the whole binary, server included, to be built with a C
toolchain. Health flags anything that should keep a card out of placement:
uncorrected ECC errors, and memory already held by another process.
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
always match the marketing name: an L40S is sold as 48GB and reports 44.99
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
box, because the number decides whether a model fits.
- **Engine adapters** for vLLM and SGLang: command and environment
construction, kept free of process handling so it is testable without GPUs.
vLLM supports both Ray and multiprocessing launch modes, because which one
works depends on the version.
- **NCCL environment generation**, the part plan §5.3 marks critical.
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
NCCL pick an adapter on another fabric where the peer is unreachable, and the
job hangs rather than failing. Bootstrap is pinned to the management
interface for the same reason.
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
compute capability 8.9 rather than silently falling back to a slower path.
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
process group and are stopped as a group, because engines fork workers that
hold GPU memory and outlive a signal sent only to the leader. Bounded log
tail, a state machine that separates Starting from Loading (a 70B model takes
minutes, and an operator needs to know that is normal), and orphan adoption
through pid files fingerprinted with the command line — pids get reused, and
adopting an unrelated process is worse than leaking the original.
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
says the weights loaded. Routing on `/health` alone sends traffic into a model
that is still loading.
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
as a table or JSON. Run it before adding a node: it is the fastest way to find
a card another process is using, or a port with no subnet manager.
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
a tensor-parallel group off the slow path inside a node. `Discover` falls back
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
topology. CI compiles the tagged build so it cannot rot unnoticed.
### Not verified
The NVML path compiles but has never run against a driver — there is no GPU
machine to try it on. Treat the NVLink domains it reports as unverified until
someone runs `llmgw agent inventory` on a real node.
- **Control plane wiring** (`internal/control`): Ready replicas become router
endpoints, with no static configuration involved. The router keeps live state
for endpoints that survive an update — inflight counts, breaker state, drain
rate — so a replica that was already serving does not start from cold because
another one moved. The sink is an interface, so when the router and control
plane split into separate processes this becomes a watch stream and nothing
above it changes.
### Phase 3 exit criterion, measured in `test/e2e`
A node hosting a live replica is killed while four clients stream continuously.
The control plane marks it unreachable, then lost, rebuilds the replica on a
survivor, and republishes. Across three runs: **~370 requests completed, zero
failures**, and no replica remained on the dead node.
This proves the control logic, not the transport: the agent fleet is simulated,
and the real path still needs the stream.
### A real bug the race detector caught
`startReplica` read the replica's fields after publishing it to the map, and
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
status reports, which arrive from another goroutine and mutate exactly those
fields. Under the simulated fleet the reports came back fast enough to hit the
window every time. Everything is snapshotted under the lock now and the
functions work from values.
### The agent stream
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
carrying registration, heartbeats and shard status upward, and start, stop and
drain commands downward. The agent dials out, because GPU nodes sit behind
firewalls far more often than a control plane does and an outbound connection
needs no inbound rule on every node in the fleet.
- **Registration carries the whole inventory**, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it — they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
the reconciler's dispatcher: it holds the open streams, so it is the only
thing that can deliver a command to a node.
- **Agent** (`internal/agent`) validates a profile before launching anything —
an engine that refuses a profile does so after minutes of loading — then
supervises the process and reports every transition, including the engine's
output tail on failure.
- **Reconnects forever with jittered backoff.** A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock the control plane over again. Engines keep serving throughout: the
router talks to them directly.
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
any model declares a profile. The static endpoint list remains for
router-only runs against sim-engines.
### Verified with real binaries
Server and agent as separate processes over a real stream: the agent registers,
a replica is placed, an engine is launched through the vLLM adapter's actual
argument construction, streaming traffic is served through the router, and
killing the agent moves the node through unreachable to lost with placement
explaining exactly why it can no longer run anything.
### Surviving a control plane restart
Not persisting replica state left one hole that had to be closed before it
mattered: restarting the control plane orphaned live engines. They kept holding
GPUs the inventory believed were free, so the next placement failed to allocate
with an error naming the new process rather than the old one.
The fix is on the agent, not in a database, and it cannot disagree with reality
the way a stale row can:
- The supervisor now writes the full shard spec, not just a pid and command
line. A pid identifies a process; the spec is what lets it be described back
to the control plane and matched to a replica.
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
waited on — it is not this process's child — so it is watched by polling, and
stopped by signalling its group directly. Its earlier output is gone, and the
log says so rather than presenting emptiness as silence.
- Every `Register`, not only the first, reports what the agent is running. A
reconnect may be a control plane restart and has to recover the same way.
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
Anything whose model or profile left the catalog is stopped: nothing will
route to it and it is holding cards.
- An engine the agent cannot identify — a corrupt state file — is killed. A
process holding GPUs that nothing can describe is worse than no process,
because placement will hand those GPUs out again.
One mechanism covers agent restarts and control plane restarts.
### A test that was passing for no reason
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
tracking hijacked connections. The agent never reconnected and the assertion —
on the connection map, which still held a stale entry — passed anyway. Tests now
sever connections through their own listener and count opened streams.
### Multi-node launch
Placement and environment generation had landed early, which made multi-node
look further along than it was: a multi-node replica could not come up at all.
Three things were missing, and all three came down to two assumptions that do
not hold once a replica spans nodes.
**Not every rank serves HTTP.** The agent health-checked every shard at its own
port, but only the leader answers. Workers timed out, reported Failed, and took
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
worker is judged by staying alive instead. That is weaker evidence on purpose —
the replica's real readiness is the leader answering, which only happens once
every worker has joined.
**A shard is not always one process.** `Engine.Commands(spec)` returns a
sequence: every command but the last is a sidecar kept alive for the shard's
lifetime and torn down with it, and the last one's exit is the shard's exit.
vLLM under Ray is why — the leader runs a Ray head and then the server against
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
had been written and tested but never called, so `launch_mode: ray` would have
run `vllm serve` against a cluster that did not exist.
A command can also require the rendezvous point to be listening before the next
one starts, bounded by a timeout so a head that never appears fails the shard
rather than hanging it. Sidecars are killed with the shard: a Ray head left
behind holds the port the next replica needs.
**Start order was backwards.** Workers were started first, on the theory that
the head should wait for them. Both engines put the rendezvous point on rank 0,
so a worker started first has nothing to connect to and spends its startup
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
needs, so the leader can start immediately and waits on its own.
None of this is verified against real engines. It is tested with stub processes;
the Ray sequencing and which launch mode a given vLLM version wants can only be
confirmed on hardware.
### Still missing for phase 2
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
network.
### Phase 3: placement
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
their fabric attachments, and what is allocated where. A re-registering agent
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
without doing so either. A GPU with an ECC fault or another process on it is
not counted as free. Replicas are released whole, because they fail whole.
- **Placement** (`internal/control/placement`): picks the highest-priority
profile that fits and explains every candidate it rejected. Two constraints
are enforced structurally rather than checked afterwards — there is no code
path that can combine nodes from two fabrics, and only GPUs of the profile's
class are ever considered.
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
large models that cannot be split. Multi-node replicas get the opposite
treatment: they are the least movable thing in the cluster, so they take the
emptiest machines.
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
domain where the topology is known. Without the NVML build it falls back to
the lowest free indices, which is usually right but only by accident of how
vendors number slots.
- **Property-based invariant tests** (plan §13): 400 randomly generated
clusters, checking on every placement that a multi-node replica lives on one
fabric, that each shard receives only that fabric's adapters, and that no
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
The table-driven tests cover the cases someone thought of; this covers the
ones nobody did.
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
rather than at request time — a typo in a fallback should stop a deployment,
not surface during the outage when the fallback is finally needed. The
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
load instead of minutes into model loading.
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
loop. It places one replica per pass, because placement depends on what is
already allocated and starting several from one snapshot would hand them the
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
vLLM under Ray needs its workers attached before the head serves.
- **Replica failure is whole-replica failure.** One dead shard makes the others
useless — they wait on collectives that will never complete while holding
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
exponential backoff carried on the model rather than the replica it just
destroyed.
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
blip must not tear down replicas that are still serving, and rebuilding a
multi-node replica costs tens of minutes of model loading.
- **Scale-down drains before stopping**, so in-flight requests finish instead of
being cut off mid-stream.
## v0.3.0 — 2026-09-16
Flow control. A batch workload no longer starves interactive traffic.
### Added
- **Admission control** (`internal/router/admission`), inserted between quota
checks and scheduling. It answers *when* a request may proceed; the scheduler
still answers *where* it goes. Keeping them apart is the point: an engine
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
no amount of endpoint selection rescues the request behind it.
- **Saturation detector.** One number per pool, the worse of queue depth and KV
occupancy. Either signal alone misleads — a pool can have an empty queue and a
nearly full cache, or the reverse.
- **Priority bands.** Each band has a saturation ceiling; above it, that band
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
"flow up to the healthy limit"; above 1 means never wait.
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
one caller sending a thousand requests occupies one slot in the rotation
rather than a thousand.
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
(504), because sending the GPU work the client has already abandoned helps
nobody. Cancelled requests leave the queue instead of holding capacity.
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
`x-llmgw-queue-ms` when a request waited.
### Phase 1.5 exit criterion, measured in `test/e2e`
While 500 concurrent bronze requests run continuously, interactive p99 in the
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
With admission switched off the same load pushes it to **55–60x**. Both
directions are asserted, so the test fails if admission regresses *and* if the
load generator ever stops being a real load.
### Two lags, two different fixes
Getting this right took longer than the queueing itself:
- Engine metrics are scraped, so they lag by an interval. The router's inflight
count does not — but it includes *running* requests, and treating those as
queue depth makes a healthy engine look overloaded every interval. What is
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
the metric contract, so it is learned as the high-water mark of observed
`running`.
- Between admission granting a request and the router dispatching it, the
request is invisible to every signal. A simultaneous burst therefore passed
the check en masse — exactly the batch arrival pattern this layer exists to
survive. The controller now counts its own outstanding leases against
dispatched inflight.
### Known limitations
- The band comes from a trusted header until authentication arrives in v1.0.0.
Anyone who can reach the port can promote themselves to platinum.
- The tenant key is the session id, for the same reason.
- Displacement (dropping queued low-band work to make room) is not implemented;
the plan defers it, and bounded queues plus TTL cover most of the need.
## v0.2.0 — 2026-09-16
The router serves traffic. `llmgw server` is real: it fronts a static set of
engine replicas with an OpenAI-compatible API.
### Added
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog.
- **Retry and fallback.** Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it via `x-llmgw-served-model`.
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched.
- **Usage accounting that does not depend on the client.** The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way.
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time.
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
### Phase 1 exit criterion, measured in `test/e2e`
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
### Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing.
- The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.