llmgw/CHANGELOG.md

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version	Phase	Scope	State
v0.1.0	0	Skeleton and sim-engine	released
v0.2.0	1	Static router: streaming proxy, filter/score/pick	released
v0.3.0	1.5	Admission control: priority bands, saturation detector	released
v0.4.0	2	Agent: GPU/IB discovery, single-node engines	released as part of v0.5.0
v0.5.0	3	Control plane: catalog, placement, reconciler	released
v0.6.0	4	Multi-node replicas and fabric awareness	released
v0.7.0	—	Conversations, agent memory, disk management	released
v0.8.0	—	PostgreSQL, leader election, agent failover	released
v0.9.0	—	Managing people and the tenancy tree	released
v1.0.0	5	Tenancy, prefix affinity, benchmarking	planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

Unreleased

An organization's own page

A member of an organization can be made its manager (users join --org … --role manager, or the star on the people page). A manager gets /org: the organization's usage over time by team, project or person, its teams and projects with limits and spend, and its members — read-only, and only what was spent in its projects. Administrators can open every organization's page.

How a model runs, from the mirror page

Each model a mirrored file is serving now shows how it runs — always on, on demand, or stopped — and it can be changed there, without the CLI. The serve form asks the same: on demand loads the model when somebody asks for it and stops it after 15 minutes unused. A model declared in server.yaml shows its mode and says to change it there, because the file is read on every start.

Weights the nodes cannot load are not fetched

kev-0.5b was a GGUF whose architecture, ggmlc, llama.cpp does not know. It was downloaded, catalogued and placed, and failed on every node it was tried on. The architecture was in the first kilobytes of the file all along. Now:

• The mirror page's registry listing shows each file's architecture and, for one the nodes' llama.cpp cannot load, says so instead of offering to fetch it. A vision projector (mmproj) is marked as not a model on its own.
• A fetch reads the file's header first — a range request, not a download — and refuses one that cannot be served, unless "fetch anyway" is ticked.
• Serving from the mirror and llmgw ctl model add refuse the same.

The list of architectures is llama.cpp's own (LLM_ARCH_NAMES) at the build llmgw pins, b10991: 152 of them. A different build configured in engines: is not refused, only not checked.

Fixes

• Usage by team, project or organization, and the spend on the quotas page, left out every request made on the chat page: they were joined through the key, and the chat page has none. They now count where a request was spent.
• /org was answered with "no api key was provided" for a signed-in browser; a test now drives every session page through the authentication middleware.

People belong to many places, or to none

A person was in exactly one project, chosen at approval, and could not use the gateway before that. Now:

• Memberships. A person can be a member of any number of organizations, teams and projects, or of none. A member of a team or organization may use every project in it. llmgw users join/leave, and the people page adds and removes them from either side. Approving no longer requires a project.
• Personal keys. A key can belong to a person with no project. On /profile a person chooses what a new key spends against: one of their projects, or Personal. llmgw keys create --for <person> does the same. auth.personal sets default limits for personal use, or turns it off.
• Limits for people. --scope user limits a person across every key, project and the chat page. --per-member on an organization, team or project gives each member their own share there. Both are on the quotas page, and are enforced alongside the tree, each in its own pool.
• The chat page asks what to spend against when a person has more than one choice, and checks it on every message.
• Spend is recorded with who spent it, so a person's budget survives a restart, personal use and the chat page included.

Leaving a project revokes the keys the person made there, unless another membership still reaches it. A personal key keeps no conversations or notes: those are kept per project.

Upgrading. Migration 0010 turns everyone's project into a membership and attributes past usage to people where a key had an owner. On SQLite it rebuilds the keys table; it was run against a copy of a production PostgreSQL store before release. llmgw users move is gone: use join and leave.

One menu on every page

Each page drew its own menu, and no two agreed: different links, in a different order, the page you were on sometimes listed and sometimes not. Every signed-in page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the left for an administrator, Chat, Your keys and Sign out on the right for everyone — with the current page marked. The chat and key pages name the person's project by its path rather than its id.

The quotas page says what actually applies

A branch that sets no limit of its own used to read "—" in every column while its organization held it to sixty requests a minute. Each limit now shows the value it is held to from above, muted and marked ↑, with where it comes from on hover; a larger value set below a stricter one says what caps it, for rate and token limits as well as budgets. Spend and budget share a column with a meter that turns red past 90%. The tree is indented like the people page, opening a form or saving one no longer reloads the page, and the explanations moved under "How limits combine". Projects are placed under their team by id rather than by the start of their path, which misplaced them when two organizations shared a name.

v0.9.2 — 2026-09-28

A new control plane serves a model by itself

llmgw server init now also runs a model mirror on the same machine, makes the machine a node of its own control plane, and fetches Qwen2.5 0.5B into the mirror and serves it there — on the machine's NVIDIA GPU when nvidia-smi finds one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a CPU profile as the fallback. An NVIDIA card on the PCI bus with no working driver is reported. --no-mirror, --no-node and --no-model leave these out, and every step is skipped on a rerun when it is already done.

Not yet run end to end on a real machine.

v0.9.1 — 2026-09-28

The mirror page is one line per model

What the mirror holds used to take two table rows per file — seven columns and the address under them — so a sharded model filled the screen. Now each version is one line: model and version, the file (or "4 files" for a set), total size, and which catalog models use it. The address moved into a "copy address" button; the checksum is in the file's tooltip. A version already in use is not offered for serving again, and removing one warns that the catalog uses it. The header counts versions rather than files.

v0.9.0 — 2026-09-28

Managing people and the tenancy tree

Organizations, teams and projects can now be renamed, moved and removed, and an approved person can be moved to another project or removed — none of which could be done at all before, from the web or the CLI.

• /admin/users shows the whole tree, each project opening onto its people and keys, with approve, move, suspend, reinstate and remove for people, revoke for keys, and add, rename, move and remove at every level. A search over every account sits below it. Changes update the page in place (htmx, embedded in the binary) and it still works with scripts off.
• llmgw org — tree [--people], add, rename, move, remove.
• llmgw users move and llmgw users delete.
• llmgw keys list --all lists every key with its project and owner.
• Every command that takes a project, team or organization accepts an id, a path such as Acme/platform/gateway, or a unique name, and --project may be omitted when there is only one. Users are named by id or email.

Moving a person takes the keys they made with them; keys an operator issued stay with the project. Only empty things are removed, and a project that ever had a key is never removed, because usage history refers to it.

Changes

• --project no longer defaults to default in keys create, keys list, users approve, users create, keys threads and keys memory: a tree built from the web has no project of that name. With one project it is chosen; with several the command lists them.
• keys limits --budget is in whole units, as the quotas page takes it, rather than micro-units. --budget 50000000 now means fifty million units; divide existing scripts' figures by 1 000 000.

Fixes

• keys init could silently move an existing team or project to another organization or team by redeclaring it. It now refuses.
• An account rejected while pending and then reinstated became active with no project, and failed at its first key. Reinstating one is refused; approving gives it a project. Rejecting a signup now removes it.

v0.8.8 — 2026-09-28

Fixes

• server init --postgres failed on Rocky Linux with "postgresql-setup: executable file not found". The PostgreSQL project's own packages (PGDG) provide postgresql-server too, so dnf can answer that name with postgresql16-server, which names its unit postgresql-16.service and keeps its tools in /usr/pgsql-16/bin. init now finds the server unit systemd has, reads the data directory from it, and runs initdb itself when the directory is empty. A PostgreSQL that was already installed is used as it is.

Known gaps

Not yet run on a real machine. Three arrangements are not handled: a PostgreSQL initialised with ident for TCP logins (the connection check fails and says so), one listening on a port other than 5432, and several PGDG versions where the highest is not the one on 5432.

v0.8.7 — 2026-09-28

PostgreSQL in one command

sudo llmgw server init --postgres

Installs PostgreSQL with the machine's own package manager when there is none, creates the llmgw role and database with a generated password kept in /etc/llmgw/secrets/store-dsn, and checks that it connects. On a control plane already running on SQLite it stops the gateway, copies the store across so every account and key survives, points auth.store at PostgreSQL and starts the gateway again; if the move fails, the gateway is started again on SQLite. Given on the first run, there is never a SQLite store.

server.yaml is edited in place with its comments intact, and the previous file is kept beside it. A PostgreSQL that init did not install is left sized as it was, since something else may depend on it.

The PostgreSQL installation itself has not yet run on a real machine; the move, the configuration edit and the rerun behaviour are tested.

Fixes

• A new control plane with no models did not start. A configuration with no static endpoints and no models was refused as having nothing to serve, so under systemd the unit restarted every two seconds and nothing listened on 8080. It now runs as a control plane with an empty catalog.

v0.8.6 — 2026-09-28

Reverse proxy configurations

deploy/proxy/ has a Caddyfile and an nginx configuration for the client port, and both are attached to every release beside the binaries. They do not buffer streamed replies, their read timeout outlasts a model's wait before its first token, their body limit matches the router's, and they set X-Forwarded-Proto and X-Forwarded-Host themselves — llmgw marks session cookies Secure from the first and shows the second as the base URL on its landing page.

v0.8.5 — 2026-09-28

A control plane in one command

sudo llmgw server init --admin you@example.com

Creates the llmgw user and its directories, the certificate authority, the control plane's certificate for the machine's hostname and every address it has, server.yaml, the default project, and an administrator whose password is generated and printed once — then installs the systemd unit, starts it and waits until it answers. What used to be two steps of install.md is now this.

Each step is skipped when its result exists, so running it again finishes an install that stopped and changes nothing on one that did not. --host adds an address the machine cannot see of itself (a DNS alias, a NAT or floating address) and reissues the certificate keeping the old ones. It never overwrites server.yaml or an authority, and certificates issued by an authority kept on another machine are left as they are.

Fixes

• The configuration install.md showed could not start. An empty catalog without control.enabled: true is refused as having nothing to serve. The documentation and the file init writes both set it.

v0.8.4 — 2026-09-28

Node logs on the admin page

Each node on the cluster page links to its logs — the agent's own, and the output of every engine running there — with an option to follow them. llmgw ctl logs NODE reads the same. They come over the agent's existing stream, so no ssh, jump host or root is needed to see what a node did. The agent keeps its last 2000 lines in memory; each engine its last few hundred.

v0.8.3 — 2026-09-28

Found by building a node from nothing with v0.8.2.

Fixes

• A new node's first engine install failed on apt's lock. cloud-init and unattended-upgrades run apt in a machine's first minutes, and DPkg::Lock::Timeout does not cover the lock apt-get update takes. A package command held up by a lock now waits for up to ten minutes, and says so; a failed install is retried after a minute rather than ten.
• A new node reported no disk until its first download created the weights directory. It is created when the agent starts.
• Onboarding took one control plane address. It takes several, comma separated, so a new node fails over like the rest.

v0.8.2 — 2026-09-28

A node no longer runs whatever engine somebody once put on it. It installs its own.

Engines are installed by the node

engines: in server.yaml names the engines every node has and their versions, defaulting to the llama.cpp build this release was checked against (b10991). Each agent converges on it: it picks the build for its hardware — CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS packages that build needs with the machine's own package manager, downloads the archive and checks it against the sha256 GitHub publishes, unpacks it refusing anything that would land outside its directory, and runs it. ldd names a missing library or a system too old for the build ("libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)"); --version proves the rest. Installs are checked again every ten minutes.

An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose. A node that has installed nothing yet is not a candidate for any model, where before it was a candidate for all of them. ctl nodes, the cluster page and placement's refusals say where each installation stands, and warn about a CPU without AVX2.

Not every system is refused for its name: the build runs on Ubuntu 22.04+, Debian 12+ and RHEL-family 10, and ldd decides. vLLM and SGLang are not yet installed this way and must still be named in the agent's configuration.

Onboarding

Onboarding installs llmgw from the CI release for the node's architecture, checked against the release's checksums, and writes an agent configuration of only the control plane's addresses and the TLS files.

Fixes

• A race on the weights fingerprint, which two heartbeat loops touched during a failover.

v0.8.1 — 2026-09-28

Nothing here changes a configuration format: every file that worked with v0.8.0 still works. What changes is how much of it needs writing, and how much of running the cluster can be done from a page rather than a shell.

A node describes itself

An agent file used to name its node, its address, the path to llama-server and its weight directories — different on every node and kept true by hand. The agent now finds these out: its id is the hostname, its address is the one on the route to the control plane, llama-server is looked for on PATH and in /opt/llama.cpp, a node without nvidia-smi simply has no NVIDIA cards, and fetched weights land in /var/lib/llmgw-weights, which is always scanned. Memory is re-read on every heartbeat and installed engines once a minute, so a VM given more RAM or a node given an engine becomes a candidate without an agent restart; the control plane logs both. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node.

Models from the mirror, without writing YAML

Search the registry from /admin/mirror, fetch a sharded model in one action, and serve a mirrored version from its row — name, source, engine and an estimated memory requirement filled in. A model can say what it is for, and /v1/models shows it. Placement no longer hands the same memory to two models or puts an engine on a node that lacks it, and says when a replica runs an edited profile.

Tenancy, quotas and usage on pages

The tenancy tree, quotas and budgets are managed from the admin pages; an organization's budget bounds its teams, a group's quota counts against the group, and a budget that cannot be spent against says so. /admin/usage charts requests, tokens and spend over time by organization, team, project or person, and a request is priced by the model that served it.

Watching the cluster

The cluster page has its own charts, backed by a year of Prometheus history, with alerts that were made to fire before being trusted — including for the disk the metrics share with the models.

Chat

The picker offers each model once rather than once per alias, never asks for more tokens than the model allows, and offers a model that loads on demand — sending it a message is what loads it.

v0.8.0 — 2026-09-27

The store stops being a file on one machine's disk. PostgreSQL, a way to move an existing installation onto it without reissuing every key, and the two things a second control plane needs so that running one is safe rather than clever.

A second control plane, and a way to fail over to it

Leadership is a lease in the store, so two control planes against one database cannot both place work — which they otherwise would, both seeing a model wanting replicas and none running, leaving twice the replicas each holding GPUs the other believes are free.

A standby also refuses agent streams. Commands travel down the stream the agent opened, so a standby holding one is a node the leader cannot command, and the leader's placements would fail with "no agent connected" for a node that is plainly connected. Agents therefore take a list of addresses rather than one, trying each in turn: a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node.

Measured on the cluster. Leadership moved three seconds after the leader was stopped — it released the lease on the way out rather than letting it lapse — and all five agents had registered with the standby six seconds later. The standby adopted every replica rather than restarting any: six replicas, all ready, engines that never stopped serving.

Everything is unset-means-leader, so a single control plane and every deployment that existed before this behave exactly as they did.

What this does not do is move client traffic. The standby served inference correctly on its own port while the public address returned 502, because nothing in front of them moved — that needs a load balancer or a shared address, and it is the remaining piece.

PostgreSQL

The store runs on PostgreSQL, chosen by what the address looks like: a path is still SQLite, a postgres:// URL is not, and @ followed by a path reads the address out of a file so a DSN carrying a password stays out of a configuration that gets copied and pasted into issues. Every existing installation keeps working untouched.

One set of queries rather than one per database, because the SQL here is almost entirely portable and two copies would drift one query at a time. The rewrite of placeholders is a wrapper around the handle rather than a change at sixty call sites — which found six places reaching past it, including all three manual transactions and two prepared statements, every one of which worked on SQLite and would have failed on PostgreSQL.

Exactly one query is not the same sentence in both: the note search, MATCH against @@. SQLite gets full-text search from an FTS5 table kept in step by three triggers; PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. It uses the 'simple' configuration rather than English — these are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things.

llmgw keys move-store copies an existing store rather than starting a fresh one, because keys are stored hashed: a migration that drops them cannot recreate them, and every person and every script would need a new one.

Testing it against a real database found three things nothing else would have. schema_migrations is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit. The copy order followed the order the migrations create tables in, and users is created after api_keys even though api_keys references it. And key_claims.sealed is emptied the moment a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an empty blob scans as a typed nil slice and a real NULL as an untyped one, a difference pgx cannot see, so it arrived as NULL.

The driver is pgx, which is pure Go: the whole binary is built with CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones staying on SQLite.

v0.7.0 — 2026-09-25

What a cluster needs to be lived with rather than demonstrated: somewhere to keep a conversation, somewhere to keep a note, and an honest answer about the disk underneath both.

Memory notes for agents, and threads without API keys

Notes an agent writes for itself and reads back in later conversations, offered as REST (/v1/memory) and as MCP tools at /mcp (remember, recall, forget). A note is a key, text and tags, searched by words or tag, private to a project and to the agent id in x-llmgw-agent. llmgw does not inject the tools into chat requests: an agent that is not given the server never sees memory.

Off for every project until someone runs llmgw keys memory --retain-hours N, and the retention is required, as it is for threads.

On a gateway with no authentication, a threads: block (store, retain_hours) now keeps conversations and notes in one shared pool. Before, x-llmgw-thread was silently ignored there; now a gateway configured to keep nothing refuses it with an explanation instead. The shared pool is for one trusted machine: anyone who can reach the port and knows an id can read it.

Conversations that can move between models

A thread a client names with x-llmgw-thread, carrying only its new turn. llmgw replays the stored history to whichever model is serving, which is what lets a conversation begun on a large model continue on a small one — including when a spent budget moved it there and the caller never chose.

Off for every project until someone runs llmgw keys threads. Until then llmgw stores no message content anywhere, as it never has, and that default is worth more than the feature. The retention is required rather than optional, because a conversation store with no expiry is a transcript of everything a team ever typed kept forever by omission, and it is measured from the last turn so a conversation in progress does not expire underneath it.

History that does not fit the serving model is trimmed oldest-first and the reply says how many turns went, in x-llmgw-thread-trimmed. System messages are never dropped — losing one silently changes what the model is for while it carries on answering — and neither is the turn the caller just sent.

Threads are scoped to a project, and reading, writing or deleting one that belongs to another answers exactly as one that never existed: two answers would let a caller map which ids are real in projects they cannot read.

Capturing the reply is opt-in and bounded, so a request without a thread accumulates nothing on the hottest path in the system, and one with a thread stops at a cap rather than growing to it.

DELETE /v1/threads/{id} forgets a conversation, as a client rather than as an operator: the person who said the thing should not have to ask someone to read it before it can go.

Disk, and taking back only what llmgw put there

Disk was the one resource nothing reported. statfs was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space in the same struct was discarded, so the only way to find out why a node was filling up was to ssh to it.

Nodes now report it, per filesystem rather than per configured directory: free space belongs to the disk, and two model directories on one disk are one pool of bytes. Shared filesystems are kept out of a node's own figure, because space on a filer belongs to everyone mounting it and twenty nodes would otherwise each report the same free terabyte. It shows in ctl nodes, the dashboard and Prometheus, and a node that reported nothing reads as unknown everywhere — an older agent must not look like a full disk.

A profile may declare min_disk_gb, and a node without the room is no longer a candidate, with a reason explain-placement prints. Skipped on a node that already holds the weights: it needs no room for them, and without that the node best placed to run a model is the one most likely to be rejected.

ctl weights lists model files llmgw downloaded for models no longer in the catalog, and --collect deletes them. The design is mostly about what it will not do. Only files llmgw fetched are candidates, and that is a calculation rather than a heuristic — the cache name is a hash of the source, so the wanted set is computable exactly. Anything an operator put on a node is left alone however unreferenced it looks. Each node checks every path again before unlinking it, refusing anything outside its cache directory or without the cache name, so a control plane that is wrong cannot talk a node into losing a model that took a week to copy.

Fixes

• A capability was lost at an indirection. The controller is handed a forwarding dispatcher, and reaches past its interface by type assertion to find what lies beyond scheduling. Deleting weights was implemented on the agent service and not on the forwarder, so it was simply absent at runtime — with the whole suite green, because the dispatchers the tests use do not implement it either and its absence there is the expected case.
• ctl tool-check blamed the engine for what the model did. It had one verdict for a failed attempt and told operators not to try another model. Measured: the engine renders the tool prompt correctly, the coder models do not use the <tool_call> wrapper it asks for, and qwen2.5-3b-instruct on the same engine returns a populated tool_calls array. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper.

v0.6.0 — 2026-09-18

Phase 4: a replica can be bigger than one machine, and the fabric it spans decides where it is allowed to go. Around that, everything a cluster needs before anyone other than its operator is let near it — keys, budgets, a way in through a browser, and enough visibility to answer "why is nothing scheduling".

Multi-node replicas

A replica is now a set of shards with ranks, a leader and a bootstrap address, rather than one process. Two invariants are enforced rather than documented: a replica never spans two fabrics, and a tensor-parallel group never mixes GPU classes. Both are the kind of mistake that does not fail — it hangs, or it runs several times slower, and nothing says which.

A replica missing a shard is not ready. It was previously derived from the shards that had reported, so a two-node replica with one node up looked servable and took traffic into a model that could not answer.

Adoption after a restart happens one agent at a time, because that is how agents reconnect. Merging them as they arrive is what keeps a multi-node replica whole instead of being torn into fragments that are then killed as strays — and an adopted engine is health-checked before it is called ready, rather than being assumed alive because its process exists.

Fabrics are distribution domains too

An isolated fabric is not only a compute boundary: weights present in one are not present in the other. Node weight inventories are reported and summarised per fabric, so "the model is on the cluster" is never mistaken for "the model is reachable from the nodes that would run it". A fabric name that matches nothing a node actually has is refused at registration rather than becoming an unschedulable replica hours later.

Who may use it

API keys, tenants and per-model allowlists, on SQLite (modernc.org/sqlite, so still no cgo). Rate limits and budgets are enforced on the request path; usage is written off it, in batches, so accounting never adds latency to inference. Budgets survive a restart — counters are rebuilt from the store, because a limit anyone can clear by bouncing the process is not a limit.

A spent budget is now served a cheaper model where the catalog offers one, rather than refused. For a team trying models out, being dropped to something smaller is a far better answer than a wall, and the response says which model answered.

People can sign up, be approved, and issue and revoke their own keys. A key is shown once; only its hash is kept.

The browser surface

A landing page that says how to connect, a read-only cluster dashboard, and an in-browser chat that never puts an API key in the browser. The dashboard shows who is using the cluster and what they are running — per person and per model, with requests, tokens, errors, spend and latency.

All of it is served from the client-facing port behind a session, deliberately not from the admin port, which also serves pprof — and pprof can dump the heap that holds every API key in plaintext.

Models without a restart

The catalog is live. Models can be added and removed from the command line or the API, take effect immediately, and outlive a restart; configuration still wins over the store, and the whole catalog is validated before anything is written.

A model is only listed once it can actually serve. It was listed as soon as it was known, so the first thing a new user did was ask for a model that returned 503.

Loading on demand, and letting go

A model whose scaling allows it is started when someone asks for it, and stopped when nobody has for a while. An idle replica is evicted to make room for one that was asked for, with guards that exist because each is a way to lose work quietly: only on-demand models, only single-node replicas, never one that has never been used, and never one younger than its minimum lifetime.

Unknown is not idle. A replica with no recorded use is left alone rather than treated as idle since the epoch.

Knowing whether it will run before it runs

• ctl plan works out whether a model fits the cluster and how it would be split, in the precision the chosen engine can actually serve — recommending a quantisation the engine cannot load would cause the failure the command exists to prevent.
• ctl tool-check verifies llmgw's engine flags against the engines actually installed. Flags drift between releases, and the alternative is finding out during a model load.
• Placement now knows a model's context length and memory footprint, and node RAM, so a CPU profile is not placed on a machine that cannot hold it.
• Placement waits for the fleet to settle after a restart rather than deciding against a half-reported cluster.

Weights

Nodes report the model files they can see, without llmgw owning the storage: a shared filer, an image exported over the fabric, rsync or a copy made by hand all look the same to placement, which only cares whether the files are there. Reporting nothing means "not asked to look", not "has nothing".

Where the simple arrangement is not enough, a node can fetch a model by URL — single-flight, resumable, written to a temporary name and renamed atomically, so a killed download never leaves something that looks like weights.

Seeing the fleet

Prometheus metrics for nodes, GPUs and replicas, collected from live state at scrape time rather than from a background goroutine, and served on their own endpoint rather than through the one that also exposes pprof. The same view is in the dashboard, so nothing else needs installing to see it.

Onboarding a node

Adding a node is one command. There are systemd units, a private CA and a from-nothing install guide that starts with a machine that has no CUDA and no binary.

llama.cpp, and CPU profiles

A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI API without a GPU, so placement, supervision, scraping, scheduling and admission can all be exercised on ordinary machines.

A profile may now omit gpu_class, which is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it — for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. CPU replicas are spread across nodes rather than bin packed: unlike GPUs there is no allocation to stop two of them sharing a machine's cores.

Metric mappings are per engine

Running, Waiting and KVCacheUsage now have a vocabulary per engine rather than one for the cluster. A mixed fleet has two names for the same three numbers, and a single global mapping left half of it unmeasurable.

A mapping that leaves a metric unnamed now means the engine does not publish it, and the field stays zero. Naming one the engine does not serve is still a contract violation. Running and Waiting can never be omitted.

Mutual TLS on the agent stream

A private CA (llmgw ctl cert init-ca | issue-server | issue-client | inspect), certificates on both ends, and the control plane refusing any agent whose claimed node id does not match its certificate's common name. Public authorities cannot supply these: they stopped issuing the clientAuth usage.

Plaintext HTTP/2 remains the default. agent_tls: on the server and tls: on the agent turn it on.

Fixes

• An agent told to stop did not stop. Cancelling the context is not enough — HTTP/2 watches it while the request body is being written, and a bidirectional stream spends its life blocked on the read side. Measured on a deployed node: systemctl stop took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
• An agent could not explain why it was refused. The session cancelled its own context and then asked that context whether it was shutting down, so every stream error read as a clean close: no message, and the backoff reset.
• A broken metric contract was invisible. It was logged per scrape at debug level, so a cluster could look healthy while every routing decision was made blind. It is now a warning, once, with a matching line when it clears.
• Token estimates came from the routing prefix, not the prompt. Every prompt over about 2 KB was undercounted, so tokens-per-minute limits let through far more than they were set to.
• Model-fitting double-counted overhead. Whole-model overhead was charged per rank and a 10% reserve taken twice, so plan reported that a 70B model in bfloat16 does not fit two 80 GB cards. It does.
• x-llmgw-served-model was never sent. It was set after the proxy had already written the response header, which is the one point at which headers stop being changeable. A client had no way to tell a fallback from the model it asked for.
• Fleet metrics published nothing. They were registered after the registry had been built, so the collector existed and was never scraped.
• GPU discovery broke on older drivers. A field added for utilisation was required rather than optional, so a driver that does not report it produced no inventory at all instead of an inventory without that number.
• Replica ids and ports were reused while the processes holding them were still alive.

v0.5.0 — 2026-09-16

Phases 2 and 3. The cluster runs itself: agents report their hardware, the control plane places replicas, and ctl makes all of it visible.

This release covers both phases because their exit criteria depended on the same piece — the agent stream — and neither was demonstrable without it.

ctl

llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain

• nodes surfaces unhealthy GPUs and why they are excluded, which is usually the answer when a node looks idle but nothing schedules onto it.
• explain-placement prints every candidate considered and the reason it was rejected. "Unschedulable" on its own says nothing about which of six constraints bit.
• drain is a rolling move, not a stop: the replica is drained, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. Bounded by a grace period, because draining the only node that can host a model would otherwise never finish and leave it undrainable.
• cordon is the weaker operation: no new placement, nothing disturbed.

The management API is on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes.

ctl logs is not here. It needs a request/response pair the agent protocol does not have. The failure path already ships an engine log tail with every status report, so the case that matters most is covered; reading a healthy shard's output on demand is not.

Phases 2 and 3, in detail

Written as the work landed, which is why it reads as a log rather than as notes. Everything here shipped in v0.5.0.

Added

• IB discovery (internal/agent/discovery): reads /sys/class/infiniband, parses the formats sysfs actually uses ("4: ACTIVE", "200 Gb/sec (4X HDR)", hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node without IB is not an error — it is a single-node island.

• GPU discovery via nvidia-smi, the cgo-free path. NVML needs cgo, which would force the whole binary, server included, to be built with a C toolchain. Health flags anything that should keep a card out of placement: uncorrected ECC errors, and memory already held by another process.

• GPU class derivation. Sizes come from the driver in GiB, which does not always match the marketing name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g. Consistency matters more than matching the box, because the number decides whether a model fits.

• Engine adapters for vLLM and SGLang: command and environment construction, kept free of process handling so it is testable without GPUs. vLLM supports both Ray and multiprocessing launch modes, because which one works depends on the version.

• NCCL environment generation, the part plan §5.3 marks critical. NCCL_IB_HCA lists only the chosen fabric's adapters — leaving it unset lets NCCL pick an adapter on another fabric where the peer is unreachable, and the job hangs rather than failing. Bootstrap is pinned to the management interface for the same reason.

• Validation before launch: a tensor-parallel group may not mix GPU types, the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below compute capability 8.9 rather than silently falling back to a slower path.

• Supervisor (internal/agent/supervisor): processes run in their own process group and are stopped as a group, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader. Bounded log tail, a state machine that separates Starting from Loading (a 70B model takes minutes, and an operator needs to know that is normal), and orphan adoption through pid files fingerprinted with the command line — pids get reused, and adopting an unrelated process is worse than leaking the original.

• Readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a model that is still loading.

• llmgw agent inventory reports what the scheduler would see on a node, as a table or JSON. Run it before adding a node: it is the fastest way to find a card another process is using, or a port with no subnet manager.

• NVML backend behind the nvml build tag, which adds NVLink topology. Nothing else exposes which GPUs share an NVLink domain, and that is what keeps a tensor-parallel group off the slow path inside a node. Discover falls back to nvidia-smi if NVML fails, and says so loudly rather than silently losing the topology. CI compiles the tagged build so it cannot rot unnoticed.

Not verified

The NVML path compiles but has never run against a driver — there is no GPU machine to try it on. Treat the NVLink domains it reports as unverified until someone runs llmgw agent inventory on a real node.

• Control plane wiring (internal/control): Ready replicas become router endpoints, with no static configuration involved. The router keeps live state for endpoints that survive an update — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes this becomes a watch stream and nothing above it changes.

Phase 3 exit criterion, measured in test/e2e

A node hosting a live replica is killed while four clients stream continuously. The control plane marks it unreachable, then lost, rebuilds the replica on a survivor, and republishes. Across three runs: ~370 requests completed, zero failures, and no replica remained on the dead node.

This proves the control logic, not the transport: the agent fleet is simulated, and the real path still needs the stream.

A real bug the race detector caught

startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock. Both were racing with agent status reports, which arrive from another goroutine and mutate exactly those fields. Under the simulated fleet the reports came back fast enough to hit the window every time. Everything is snapshotted under the lock now and the functions work from values.

The agent stream

• Protocol (api/proto/llmgw/v1): one bidirectional connect-go stream carrying registration, heartbeats and shard status upward, and start, stop and drain commands downward. The agent dials out, because GPU nodes sit behind firewalls far more often than a control plane does and an outbound connection needs no inbound rule on every node in the fleet.
• Registration carries the whole inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it — they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement.
• Agent service (internal/control/agentsvc) is both the stream handler and the reconciler's dispatcher: it holds the open streams, so it is the only thing that can deliver a command to a node.
• Agent (internal/agent) validates a profile before launching anything — an engine that refuses a profile does so after minutes of loading — then supervises the process and reports every transition, including the engine's output tail on failure.
• Reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock the control plane over again. Engines keep serving throughout: the router talks to them directly.
• llmgw agent is real, and llmgw server runs the control plane whenever any model declares a profile. The static endpoint list remains for router-only runs against sim-engines.

Verified with real binaries

Server and agent as separate processes over a real stream: the agent registers, a replica is placed, an engine is launched through the vLLM adapter's actual argument construction, streaming traffic is served through the router, and killing the agent moves the node through unreachable to lost with placement explaining exactly why it can no longer run anything.

Surviving a control plane restart

Not persisting replica state left one hole that had to be closed before it mattered: restarting the control plane orphaned live engines. They kept holding GPUs the inventory believed were free, so the next placement failed to allocate with an error naming the new process rather than the old one.

The fix is on the agent, not in a database, and it cannot disagree with reality the way a stale row can:

• The supervisor now writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica.
• Adopt takes over a still-running engine from a previous agent. It cannot be waited on — it is not this process's child — so it is watched by polling, and stopped by signalling its group directly. Its earlier output is gone, and the log says so rather than presenting emptiness as silence.
• Every Register, not only the first, reports what the agent is running. A reconnect may be a control plane restart and has to recover the same way.
• The reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped: nothing will route to it and it is holding cards.
• An engine the agent cannot identify — a corrupt state file — is killed. A process holding GPUs that nothing can describe is worse than no process, because placement will hand those GPUs out again.

One mechanism covers agent restarts and control plane restarts.

A test that was passing for no reason

TestAgentReconnects used httptest.CloseClientConnections, which does nothing to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops tracking hijacked connections. The agent never reconnected and the assertion — on the connection map, which still held a stale entry — passed anyway. Tests now sever connections through their own listener and count opened streams.

Multi-node launch

Placement and environment generation had landed early, which made multi-node look further along than it was: a multi-node replica could not come up at all. Three things were missing, and all three came down to two assumptions that do not hold once a replica spans nodes.

Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers. Workers timed out, reported Failed, and took the replica with them. Engine.ServesHTTP(spec) now says which ranks serve; a worker is judged by staying alive instead. That is weaker evidence on purpose — the replica's real readiness is the leader answering, which only happens once every worker has joined.

A shard is not always one process. Engine.Commands(spec) returns a sequence: every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's exit. vLLM under Ray is why — the leader runs a Ray head and then the server against it, and a worker runs only the join, with no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist.

A command can also require the rendezvous point to be listening before the next one starts, bounded by a timeout so a head that never appears fails the shard rather than hanging it. Sidecars are killed with the shard: a Ray head left behind holds the port the next replica needs.

Start order was backwards. Workers were started first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to and spends its startup budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own.

None of this is verified against real engines. It is tested with stub processes; the Ray sequencing and which launch mode a given vLLM version wants can only be confirmed on hardware.

Still missing for phase 2

• mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted network.

Phase 3: placement

• Cluster inventory (internal/control/inventory): nodes, their hardware, their fabric attachments, and what is allocated where. A re-registering agent cannot clear an operator's cordon, and a heartbeat revives an unreachable node without doing so either. A GPU with an ECC fault or another process on it is not counted as free. Replicas are released whole, because they fail whole.
• Placement (internal/control/placement): picks the highest-priority profile that fits and explains every candidate it rejected. Two constraints are enforced structurally rather than checked afterwards — there is no code path that can combine nodes from two fabrics, and only GPUs of the profile's class are ever considered.
• Bin packing fills partly used nodes first, so whole nodes stay free for large models that cannot be split. Multi-node replicas get the opposite treatment: they are the least movable thing in the cluster, so they take the emptiest machines.
• NVLink-aware GPU selection keeps a tensor-parallel group inside one NVLink domain where the topology is known. Without the NVML build it falls back to the lowest free indices, which is usually right but only by accident of how vendors number slots.
• Property-based invariant tests (plan §13): 400 randomly generated clusters, checking on every placement that a multi-node replica lives on one fabric, that each shard receives only that fabric's adapters, and that no placement mixes GPU classes or uses an unhealthy or already-allocated GPU. The table-driven tests cover the cases someone thought of; this covers the ones nobody did.
• Catalog (internal/control/catalog): models, profiles, aliases, fallbacks and scaling bounds. Aliases and fallbacks are resolved at load rather than at request time — a typo in a fallback should stop a deployment, not surface during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load instead of minutes into model loading.
• Reconciler (internal/control/reconciler): the desired-versus-actual loop. It places one replica per pass, because placement depends on what is already allocated and starting several from one snapshot would hand them the same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves.
• Replica failure is whole-replica failure. One dead shard makes the others useless — they wait on collectives that will never complete while holding GPUs — so the reconciler stops every shard and rebuilds, with a jittered exponential backoff carried on the model rather than the replica it just destroyed.
• Node expiry in two stages. Unreachable first, Lost later. A brief network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading.
• Scale-down drains before stopping, so in-flight requests finish instead of being cut off mid-stream.

v0.3.0 — 2026-09-16

Flow control. A batch workload no longer starves interactive traffic.

Added

• Admission control (internal/router/admission), inserted between quota checks and scheduling. It answers when a request may proceed; the scheduler still answers where it goes. Keeping them apart is the point: an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no amount of endpoint selection rescues the request behind it.
• Saturation detector. One number per pool, the worse of queue depth and KV occupancy. Either signal alone misleads — a pool can have an empty queue and a nearly full cache, or the reverse.
• Priority bands. Each band has a saturation ceiling; above it, that band waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means "flow up to the healthy limit"; above 1 means never wait.
• Tenant fairness. Within a band, dispatch turns rotate between tenants, so one caller sending a thousand requests occupies one slot in the rotation rather than a thousand.
• Bounded queues. Capacity per band (immediate 503 when full) and a TTL (504), because sending the GPU work the client has already abandoned helps nobody. Cancelled requests leave the queue instead of holding capacity.
• Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total, llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry x-llmgw-queue-ms when a request waited.

Phase 1.5 exit criterion, measured in test/e2e

While 500 concurrent bronze requests run continuously, interactive p99 in the platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget. With admission switched off the same load pushes it to 55–60x. Both directions are asserted, so the test fails if admission regresses and if the load generator ever stops being a real load.

Two lags, two different fixes

Getting this right took longer than the queueing itself:

• Engine metrics are scraped, so they lag by an interval. The router's inflight count does not — but it includes running requests, and treating those as queue depth makes a healthy engine look overloaded every interval. What is genuinely queued is inflight − batch capacity, and batch capacity is not in the metric contract, so it is learned as the high-water mark of observed running.
• Between admission granting a request and the router dispatching it, the request is invisible to every signal. A simultaneous burst therefore passed the check en masse — exactly the batch arrival pattern this layer exists to survive. The controller now counts its own outstanding leases against dispatched inflight.

Known limitations

• The band comes from a trusted header until authentication arrives in v1.0.0. Anyone who can reach the port can promote themselves to platinum.
• The tenant key is the session id, for the same reason.
• Displacement (dropping queued low-band work to make room) is not implemented; the plan defers it, and bounded queues plus TTL cover most of the need.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

• Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
• Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
• OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
• Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
• Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
• Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

• 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
• Every client disconnecting frees the KV blocks on every engine.
• Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

• No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
• The endpoint list is static; the control plane arrives in v0.5.0.
• No authentication, rate limiting or budgets yet (v1.0.0).
• Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

• continuous batching with a bounded concurrent sequence count
• KV cache block accounting, queueing when blocks run out, including head-of-line blocking
• automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
• per-token latency that grows with batch size
• dropping a request and freeing its blocks when the client disconnects
• /metrics under vLLM's real metric names
• failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

• router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
• router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
• router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
• router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

• llmgw server, llmgw agent and llmgw ctl are stubs.
• The scheduler has no HTTP layer yet; nothing routes real requests.
• No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
• Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

• The project language is English throughout: code, comments, tests, commits and the design document.
• Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.