llmgw/CHANGELOG.md

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version	Phase	Scope	State
v0.1.0	0	Skeleton and sim-engine	released
v0.2.0	1	Static router: streaming proxy, filter/score/pick	released
v0.3.0	1.5	Admission control: priority bands, saturation detector	released
v0.4.0	2	Agent: GPU/IB discovery, single-node engines	released as part of v0.5.0
v0.5.0	3	Control plane: catalog, placement, reconciler	released
v0.6.0	4	Multi-node replicas and fabric awareness	released
v0.7.0	—	Conversations, agent memory, disk management	released
v0.8.0	—	PostgreSQL, leader election, agent failover	released
v0.9.0	—	Managing people and the tenancy tree	released
v0.10.0	—	InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes	released
v0.11.0	—	High availability from setup, the setup and console screens, usage sums and per-tenant alerts	released
v0.12.0	—	The cluster's settings in the store, in force when changed; the firewall switch; wipe	released
v0.13.0	—	The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together	released
v0.14.0	—	Knowledge bases: documents from people's own machines, kept apart per tenant, answered from	released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine)
v0.15.0	—	The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions	released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files)
v0.16.0	—	Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are	released
v1.0.0	5	What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation	planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

v0.16.0 — 2026-10-30

Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256 Hugging Face published; the documents llmgw-kb sends embedded by the gateway; Strata offered by the plan, ctl model serve, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they are, and do not fail a node that came back on the version.

Upgrading.

• No migrations. Upgrade the control planes, then the nodes (llmgw ctl upgrade v0.16.0), then each mirror's host (sudo llmgw upgrade v0.16.0 --service llmgw-mirror).
• A node checks what it fetches against the catalog's checksums only once it runs this version; until then it fetches as before.
• A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as they are.
• A Strata profile must say --size as well as --family, and both must be ones Strata runs.
• llmgw-kb sends text alone to a gateway of this version; an older llmgw-kb keeps working, sending vectors.

Added

• Strata offered where models are chosen. For a repository of a family Strata runs, ctl model plan offers "one card with Strata, experts in RAM" with the RAM it takes, ctl model serve --engine strata (or the plan's choice) builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the RAM a size keeps its experts in.
• The gateway embeds the documents llmgw-kb sends. A document may be sent as its passages' text alone; the gateway embeds them through its own /v1/embeddings as the caller, 32 at a time and four batches at once, waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead of a vector down and up again, some thirty times more. llmgw-kb does so with a gateway that says it can (server_embeds in a collection's answer); --embed-here keeps the old way.
• Weights pinned to a commit, and checked against what Hugging Face published. A model added from Hugging Face (hf:, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at then — the file's URL at that commit, or --revision — and each file's published sha256 goes in its origin. A node refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for what ctl model serve fetches, and llmgw mirror add for a Hugging Face URL (--sha256 for others). Nodes need this version to check; older ones fetch as before.

Fixed

• A cluster upgrade that sat silent, or stopped on a node that had upgraded.

  • The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30 seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
  • The control plane takes the node registering again on the version as done: the agent's answer, sent a moment before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
  • The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the control plane waits longer than the download may take; a second request while one runs is refused.
  • A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
  • The console's node upgrade waits as long as the control plane, and offers no development version.
  • systemd never gives up restarting the agent (StartLimitIntervalSec=0), in units written from now on.

• Strata. One named by hand in agent.yaml was never reported, so never placed on; its engine, which leaves the server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.

• A name with a slash in it, asked through a standby control plane. llmgw ctl kb delete Demo/hr on a standby was answered 405: the standby passed the request on to the leader with Demo%2Fhr unescaped, two segments instead of one. The path now reaches the leader as it was sent.

v0.15.3 — 2026-10-29

A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.

Upgrading.

• No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
• The mirrors carry the fixes: on each mirror's machine, sudo llmgw upgrade v0.15.3 --service llmgw-mirror. A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.

Added

• A model stopped and kept, and a model removed with its files. llmgw ctl model stop|start MODEL: none of its replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an administrator stopped it. llmgw ctl model remove MODEL --purge (DELETE /api/v1/models/{name}?purge=true) also deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have stopped; what another model uses stays. Both are in the console's model menu.

Fixed

• llmgw mirror add-peer on a mirror with a machine of its own. It set the second mirror up and then failed for want of the control plane's server.yaml; now it prints the llmgw ctl config set mirror.peers to run on a control plane.
• Models fetched before checksums were kept reach the second mirror. A mirror kept in step copies only what it can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once.
• A file removed with llmgw mirror remove is gone from the mirror's answers too. The running mirror kept reporting a file it had fetched as present for a week after it was removed, so a control plane could place a model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.

v0.15.2 — 2026-10-29

A served model moved to more cards, or fewer, while it is served.

Upgrading.

• No migrations, and nothing changes until the command is used. The control planes carry it; the nodes need nothing new.

Added

• A served model moved to more cards, or fewer. llmgw ctl model gpus MODEL N [--profile P] [--in-place], POST /api/v1/models/{name}/gpus, and GPUs… in the console's model menu. How many cards of one node each replica uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against the model's KV heads, and the layers split for llama.cpp. Refused for a count no node has, or whose cards would not hold the model and its context, with the figures; its replicas are then started again on it, one by one, beside the old — or, with --in-place when there is no room beside them, each stopped first.

v0.15.1 — 2026-10-29

A 125B model on one card, a second mirror, and weights that run code refused. Strata runs as an engine: a mixture-of-experts model with its experts in RAM, served from one 24 GB card. A second model mirror keeps in step with the first, so an HA install has no single machine left. Pickled weights are refused unless a model says otherwise. A served model's context can be changed in place. The explorer says how fast a model would answer here before it is fetched, and a card in the wrong PCIe slot is said so.

Upgrading.

• No migrations. Upgrade the control planes before the nodes, for the pickle refusal below.
• A model whose weights are in a pickle format (.bin, .pt, .pth, .ckpt, .pkl) stops being placed: its node refuses the source. Give it its safetensors or GGUF, or add it again with allow_pickle: true (llmgw ctl model add -f) if its files were checked. Look at the models' sources before upgrading the nodes: one ending in .ckpt or .bin, or a mirror version holding pytorch_model.bin. Upgrade the control planes first: a node on this version refuses those weights until its control plane can say allow_pickle.
• Strata is opt-in: nothing installs it until engines or an engine_overrides entry names a version (0.2.7).

Added

• Strata as an engine: a 125B model of experts on one card. Strata (github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client; more clients share about that, its slots buying fairness rather than throughput on one card. It runs a few families only — Qwen3.8-Flash-Next and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a profile's args with --family and --size (--family qwen --size IQ2_XS --context 32768). The source is the directory of the model's GGUF files, or the repository, which Strata then downloads itself, pinned. The agent installs it on an NVIDIA node of x86-64: its source pinned by digest, in a Python environment, and its release's engine checked against the digest GitHub publishes — the CUDA 13 build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal to Ada, a V100 included, on driver 575. A replica prepares its model the first time (minutes; its draft layer is a 5 GB download) and loads in about a minute and a half after. One card of one node per replica, for now; not in an offline bundle, its packages coming from PyPI.

• A served model's context, changed. llmgw ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context, and Context length… in the console's model menu. The catalog, the engine's arguments and the memory change together, checked against what the model was trained for and what its cards hold; its replicas are then started again one by one. The models API shows context_length and max_output.

• A second model mirror, kept in step. In a three-machine install the setup screens put one on a database machine (--second-mirror 1|2). An install with one mirror gets one with sudo llmgw mirror add-peer user@host.

  • In step: each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and removes what the other removed, even if it was down at the time (llmgw mirror serve --peer).
  • Control planes: they name both (mirror.peers) and use whichever answers.
  • Nodes: while one mirror is down, a node is given a model's file on the other.

  This takes away the last single point of an HA install: with the first machine off, models are still added and fetched.

Fixed

• An engine that computes in a process of its own group was not its replica's. A GPU replica read as running on the CPU, and the memory it held on its card as somebody else's, which placement then kept free of other work. A process the replica started, at any remove, is now the replica's. And what a replica computes on, once sent, was never sent again: a wrong reading kept from before an agent's restart stayed wrong. It is sent again when it changes, and read again when a GPU replica was kept as the CPU.

• VS Code's sign-in lost its place through the sign-in page, so the prompt to confirm came every time; a link to the same gateway by plain http could take an https setting down to http; and a new build of the extension was not installed over an old one (the command now says --force).

• The install guide's recordings were not checked for secrets, only the page; every frame is now.

• MCP server fields could not be emptied, an address could carry a password, the cluster's own server could not be changed from the console, and a grant to a key that does not exist was said done.

• A guardrail service two policies named was asked as the first said, not the stricter; a null answer read as safe; and with restore the request log kept the answer with the redacted values back in it.

• What an engine downloads by a Hugging Face id was not held to the pickle refusal — an adapter by its id, a TensorRT-LLM model's; a model with either now needs allow_pickle.

• server init run again on a machine already set up was refused its memory, as on an 8 GB install fixing its certificate with --host, and judges that would not be installed (no mirror) were counted.

• Removing an MCP server left its sealed credential in the store, and replacing a credential under another header kept the old one sent. A server's secrets now go when it no longer reads them.

• A cloned repository could point the VS Code extension at another gateway. llmgw.gatewayUrl is a machine setting now, never taken from a workspace.

• A node's memory bandwidth could be measured in its cache on a busy node, and was 0 after a reconnect until the next heartbeat. The buffer is at least eight times the processor's last cache.

• A project kept in the cluster could have its questions embedded outside it. The similar-question cache now asks the embedding model as the question's project would, external fallbacks refused, and skips a model that is itself outside. Its comparing no longer holds the cache's lock, which every project's lookups waited on.

• Restored values were left as placeholders in streamed tool calls, in completions and in reasoning, and a value with a quote in a whole answer's tool-call arguments broke their JSON.

• An outside guardrail service read what redact had taken out, and a streamed answer was sent to it twice. The docs now say a service stops a streamed answer only under stream: buffer_all.

• A replica found slow once could stay unused. Its time to first token is forgotten after five minutes without an answer, so it is tried again.

• Weights that run code got past the refusal in four ways. A mirror version already on a node was used without being looked at again; a ?query or #fragment on an hf: source hid the file's name; .pt2 was not on the list; and a profile could write --load_format, --trust_remote_code or an abbreviation such as --trust-remote, which vLLM and argparse read as the flag, or give SGLang --load-format. All refused now. TripoSR's preset says allow_pickle, without which it could not be added, and a shared ComfyUI environment takes its first member's.

• A temperature written as a string was refused. Since the GenAI traces read it, "temperature": "0.7" got a 400 before reaching the engine; it is read loosely again, for the trace only.

• A reused answer could carry another caller's personal data. With restore: true and the answer cache on, two callers' requests that redacted to the same text shared one answer — with the first caller's address put back in it. Such requests are now never answered from the cache nor kept in it.

• Granting a server's tools could take others away; revoking could close a level to every tool. A level with no tool list sees what the levels above grant; giving it its first list made it see only that list. And a level left with an empty list was closed to every tool, which the screens did not show. Now a first list starts from the nearest list above, taking a server away leaves the rest or goes back to no list, the screens say when a level above keeps granted tools out of reach and which levels are closed, and the console reads and writes the level at once rather than from what it loaded.

• Similar questions about different pictures got each other's answer. A question with a picture, a sound or a file in it is reused only when it is the same request.

Added

• A card in the wrong PCIe slot is said so. Each agent reads how every card is linked — the fastest generation card and slot allow together, and the lanes in use — against the card's own most, from the kernel (the driver's figure is what card and slot allow together); a card on fewer lanes or an older generation is a node warning, with what it carries and what it could, on the dashboard, the console and in llmgw doctor.
• And on a web page, /admin/mcp. For administrators: the same list, adding a server with its credential sealed, testing it and listing its tools, granting them in the tree beside that level's other grants and limits, revoking, changing, replacing the credential, turning it off, removing; and the line that points an agent at this gateway's /mcp. In English and Turkish; every change in the audit log, the secret not.
• The company's MCP servers in the console. A screen of its own: each server with its state, who has its tools, its rate, the names of the credentials it is sent and its address; add one with a bearer token or a header kept sealed, test it and list its tools, grant them to a team, project, organization, person or key beside that level's other grants, revoke, change, replace the credential, turn off, remove.
• And on the cards, as this cluster would lay it out. The explorer's GPU tok/s and a repository's files give one reply's speed for each layout the planner offers here — one card, tensor parallel across a node's cards, llama.cpp's layer split, a pipeline across nodes on a fabric — the fastest of them, with the layout in the tooltip. From each card's memory bandwidth by its model name, NVLink or PCIe between a node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed classes is judged class by class. A model of experts reads 1.6 times its active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144 tok/s; qwen3-vl-8b Q8: 62).
• How fast a model would answer on a CPU, before it is fetched. Each agent measures how fast its node's memory reads when it starts (Host gets mem_bandwidth_gbs, the node's view mem_bandwidth_gbs); the explorer's CPU tok/s column and a repository's GGUF files say about how many tokens a second one reply gets on the fastest node that holds the model — the bandwidth over the bytes a token reads, its active part for a model of experts. Calibrated on the PoC against llama.cpp's own timings: within 15% from 1.5B to 14B.
• The explorer lists what this cluster can run. Judged by the nodes as they are now — every card's memory and compute capability, every node's RAM — through the same planner as serving a model, rather than a size picked from a list. A model too big for any node, in FP8 without a card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and counted; the page says what it judged by. With cards, safetensors repositories are listed beside GGUF. On a repository's page a file too big for the cluster is shown with what it needs and no Fetch button.
• Weights that run code are refused. Python pickles — .bin, .pt, .pth, .ckpt — run whatever they hold when loaded, which is how the malicious models on Hugging Face work. The node refuses such a source, or a mirror version with one in it, before fetching it; adding such a model is refused; vLLM and SGLang downloading a model by its id read its safetensors only; the mirror page marks the file. allow_pickle: true on a model lets it through on purpose. The mirror's explorer no longer lists a repository with nothing a node loads (no GGUF, no safetensors) while "runs here" is ticked, and a repository's pickle files are not offered, only counted. BACKLOG M lists what is next: checksums against Hugging Face everywhere, its own scan, trusted publishers, engines without the internet.

v0.15.0 — 2026-10-25

The company's tools and editors through the gateway, and guardrails that can lean on others. Its own MCP servers sit behind llmgw's /mcp under the guardrails, a rate and a breaker. VS Code gets the gateway's models from an extension the gateway serves. A guardrail policy can ask an outside service, and can put redacted values back in the answer. Chat goes to the replica answering soonest. A similar question can get the answer already given. A cached prompt can be priced apart. Traces follow the OpenTelemetry GenAI conventions, and Kong is written down from trying it. Also an install guide with every screen, at /install.

Upgrading.

• Migrations 0049 (mcp_servers), 0050 (their rate and breaker) and 0051 (projects.cache_similar), applied when the control plane starts.
• llmgw server init now refuses a machine that would serve models without the memory for them: the starter model and the judges need 12 GB on one machine, where 8 GB was said before. --no-model, --no-guard-model or --no-node make it fit. A running cluster is not checked again.
• Traces carry the GenAI conventions' gen_ai.* attributes beside the llmgw.* ones, which stay; docs/install.md, "Tracing", lists them.
• Nothing else changes until it is turned on: services and restore in a guardrail policy, keys cache --similar with router.cache.embedding_model, cached_per_1k in a price.

Added

• Answers to similar questions. llmgw keys cache --similar 0.95 also gives a chat question the answer already given to one like it in the same conversation, compared by router.cache.embedding_model, as x-llmgw-cache: similar with the similarity. Held in memory beside the exact answers; nothing written to the store but the setting.

• A cached prompt costs the cached rate. cached_per_1k in a price charges the prompt tokens an engine or provider read from its cache (the answer's cached_tokens) at their own rate; without it they cost what any prompt token does, as before.

• Chat goes to the replica answering soonest. The router keeps each replica's time to first token, smoothed over its last answers, and the chat profile scores by it beside the queue and the KV cache: of two replicas of one model, the one on the slower card or the busier host gets less, and a replica not yet measured is tried.

• llmgw and Kong, written down (docs/kong.md): Kong 3.9 in front of llmgw by a plain route or its ai-proxy, each team's llmgw key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts no completion tokens of a streamed answer, so bill from llmgw's usage.

• A guardrail that is someone else's service. A policy's services asks Azure Content Safety, Lakera, NeMo or a company's own classifier over HTTP: a body template, keys from files, dotted paths to where the answer says unsafe and why, on input, output or both, block or flag, and the policy's fail when it does not answer.

• Redacted values put back in the answer. restore: true on a guardrail policy's input: the model reads [EMAIL_1], the client reads its own address in the reply, streamed or not. Only the request's own values, held in memory for its life; the output rules read the answer as the client will.

• Traces in the OpenTelemetry GenAI conventions. An inference request and each attempt at an engine carry gen_ai.operation.name, the requested and the answering model, max_tokens, temperature and top_p when set, the session as gen_ai.conversation.id, and the input, output and cached tokens, and an attempt the engine that answered as gen_ai.provider.name. Tools that read the conventions show llmgw's calls as model calls with no dashboard of their own. No prompt or answer text is ever in a span.

• llmgw for VS Code. An extension of llmgw's own, served by every gateway at /setup/vscode/llmgw.vsix with that gateway's address in it, made from the binary itself: plain JavaScript, no dependencies, no build, and so no marketplace needed at a site with no internet. It offers the gateway's chat models to VS Code's language-model API — the chat's model picker, agent mode, and every extension that asks VS Code for a model — with the sizes and the tool and vision capabilities the catalog says, streamed, with tool calls and their results both ways. VS Code is a tool on the account page's Connect your editor: a key, the one-line install, and an Open in VS Code link that hands the key to the extension, which keeps it in VS Code's secret storage. llmgw: Sign In opens that page with a state of its own; a link without it is confirmed first, the gateway named.

• An install guide with every screen, at /install. From three bare Ubuntu machines to a first answer: what each machine needs and how to check it, the release fetched and checked, the setup screens one by one, the system's state, and a first request with a key from the account page. The screens are a real three-machine install, recorded and kept as terminal frames — sharp at any size and a few kilobytes — with each session played back on request; the administrator's password and every key were masked before it was kept. /about says more of what llmgw does (documents, guardrails, sign-in, high availability, pictures and 3D), and its install steps are the ones of today, leading to the guide.

• The company's MCP tools under the guardrails, a rate and a breaker.

  • The answer: the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a ticket is withheld as a pasted one would be, and personal data is redacted.
  • The arguments: they are read before they leave, so a secret pasted into one does not reach the company system.
  • Rate: ctl mcp add --rate N allows each caller N calls a minute to a server.
  • Breaker: five failures in a row take a server out for a minute, its tools off the list.

• How fast the judge is, measured. llmgw ctl guard judge check (and POST /api/v1/guard/judges/check) works out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once, and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged, and what to do: more cores, more judges, a GPU judge, a longer wait, or fail: open. server init runs it once the judge is up. If the machine is too slow it says so before the next steps, not after the first refused request.

• The company's own MCP servers through llmgw's /mcp. llmgw ctl mcp add jira --url … --secret-file Authorization=FILE registers one; /mcp then lists its tools as jira.search beside llmgw's own. Only keys granted them see them: llmgw keys limits --tools 'jira.*', patterns at any level of the tenancy tree. None granted anywhere is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The admin API has /api/v1/mcp/servers.

• The cluster as a read-only MCP server. Registered as llmgw ctl mcp add llmgw --url llmgw:admin and granted as llmgw.*, its tools say what llmgw ctl does:

  • cluster_status, list_nodes, list_models, list_replicas, explain_placement, node_logs;
  • nothing that changes the cluster.

  It is answered in the control plane's own process, so there is no token to keep and every control plane serves it. The admin API serves the same at POST /api/v1/mcp. ctl mcp add --bearer-file keeps a token, sent as Authorization: Bearer.

• Large models across cards and nodes. For a model of experts on more than one card, the plan now adds vLLM's --enable-expert-parallel. It reads the expert count from config.json under any of its names, and sizes a model that also reads pictures by its text_config. Several nodes are offered in any number, not only powers of two. Each way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This shows in ctl model plan, in the console's Serve a model, and in a serve job's first step. The serve form checks that a longer context still fits. ctl model serve --env KEY=VALUE puts engine variables such as NCCL_DEBUG=INFO on every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric.

• Machines of the judges' own in a three-machine install. The setup screen asks for up to two judge machines after the store's two; server init takes --judge-node user@host for each. Each is added as a node labelled llmgw.role: judge, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node of their own, 2 were. A node with an llmgw.role label takes only profiles whose node selector asks for that role, so no other model is placed on the judges' machine.

Fixed

• A machine too small for the models it would serve is said so first. The control plane that is also a node, serving the starter model and two judges, needs about 10.5 GB. On 8 GB the install finished every step and then had no room for the starter. Now the setup's checks fail its memory with what it would serve, and server init refuses before it installs anything, naming --no-model, --no-guard-model and --no-node.

• Turkish pages are Turkish to the end.

  • Errors: those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the catalog", the settings changed meanwhile, and others) are now in the page's language.
  • Percent: a percentage is written the language's way: %93 in Turkish, 93% in English.
  • Footer: the time at the foot of a page is in the language's date form.
  • Stale forms: they are refused with a sentence in that language rather than "bad request".

• llmgw keys limits no longer clears the limits it was not given. Setting one limit at a scope wrote the whole row, so --tools 'jira.*' alone took away that scope's rate, budget and model list. Only what is given changes now. --replace writes every limit, as the console's limits form does.

• The plan refused tensor parallelism past a model's KV heads. Qwen3-235B, with four KV heads, had no way to run on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now offered, as is one that divides them.

• A big model on a slow mirror was failed while it downloaded. The startup timeout counted the download, so hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing. The timeout now runs from when a node's download last moved.

• The judge init installs is no longer replaced by the leader's own. Just after the control plane started, the leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays.

• A judge the leader serves by itself is sized as init's is. When a guard policy names a judge the catalog lacks, the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to 3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, and fail: closed refused the request. It now gets what server init gives its judges: 2 GB, two cores each (fewer on a small node), and no prompt cache. Every judge served through modelserve runs with --cache-ram 0.

v0.14.9 — 2026-10-19

Sign-in set up from the console, with Active Directory tried before it is kept. A node's console now shows what the machine is doing, and a node's web page shows what it runs. Also fixes found trying Active Directory against a Samba domain controller on the test cluster. Sign-in through it, nested groups and a disabled account suspended by the sync all worked there.

Upgrading.

• No migration.
• auth.ldap and auth.oidc are in the cluster's settings, as before. A block written into server.yaml on a cluster with a store was never read; the log said so. Put it in with llmgw ctl config edit, or set it up again from the console's Sign-in screen.
• A Samba DC's own autogenerated certificate is refused (a negative serial and no SAN). Give it one from a CA.

Added

• Sign-in set up from the console. Under People → Sign-in, sudo llmgw shows how people get in: single sign-on, Active Directory and SCIM, each on or off. It also shows the directory's last sync. E sets up Entra ID from the tenant and client IDs. D sets up Active Directory, and the directory is tried before it is kept: each server reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The password, client secret and SCIM token are kept in the store as store:auth/…, sealed by the master key. The domain's CA goes into the settings (auth.ldap.ca), so no file has to be put on each control plane. The admin API has the same: GET /api/v1/auth, POST /api/v1/auth/ldap/test and PUT /api/v1/auth/secrets/{name}.
• A node's page says what it runs. It now shows the agent's version, marked when the control plane runs another, and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its GPUs.
• sudo llmgw on a node opens on what the machine is doing. A new Overview screen shows the agent's state and version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.

Fixed

• A broken auth.ldap no longer stops every control plane. It is the cluster's setting. A CA file that one machine could not use made llmgw server exit on every control plane, and no ctl config could then put it right. Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in.
• A second llmgw server on a machine stops before it competes to lead. llmgw server check (no such command) ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real control planes stood by. server now refuses words it does not know and takes its ports before anything else.
• The single sign-on and Active Directory guides said server.yaml. On a cluster with a store, auth is in the cluster's settings and the file's block is not read. They now say llmgw ctl config edit, and what certificate a domain controller needs.

v0.14.8 — 2026-10-18

Sign-in with an on-premises Active Directory, and Entra ID made safe for people in many groups. Also several keys for one outside provider, and a measure of what the gateway adds to a request.

Upgrading.

• Migration 0048 adds the user_directory table, applied when the control plane starts.
• Nothing changes until auth.ldap is set; docs/install.md, "Active Directory", has an example. It has been tested against a fake directory, not yet against a real domain controller.
• A judge installed by an earlier server init is refitted by the leader by itself (below); the steps given for v0.14.7 are no longer needed.

Added

• Sign-in with Active Directory (auth.ldap). People sign in on the gateway's form with their AD username or mail and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give staff roles and places in the tree, as with OIDC. Accounts are tied to the objectGUID and made at first sign-in when create_accounts is on. Every 15 minutes the leader reads the directory. People disabled or deleted there are suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend more than half the accounts is refused. See docs/install.md, "Active Directory".

• Several keys for one outside provider. An external model takes api_key_files, each with an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so a key answering 429 is passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint, never the key. Usage records which key answered by that name.

• llmgw ctl bench overhead: what the gateway adds to a request, measured against an engine that answers at once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks. It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).

Fixed

• Entra ID's groups past 200 no longer take a role away. For someone in more than 200 groups Entra sends none in the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app registration. docs/install.md has an Entra ID section.
• An account made by SCIM or OIDC could not be reinstated. Setting it active was refused as "never approved".
• An upgrade refits the judge an earlier server init installed. Upgrading to v0.14.7 left such a judge as it was: no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone, and the log says so once.
• A restart is no longer refused for room another restart is giving back. With every core of a node reserved, two models restarted together, one to need more and one less (as after ctl model resources on a judge and a starter), had the first refused: "fits on no node even with this replica stopped", though the second was about to give back its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained replica also counts a replica being restarted that still answers as serving, so replicas restarted one after another no longer wait out the five-minute grace each.

v0.14.7 — 2026-10-16

The judge installed by server init sized for what it does. On a machine of fewer than eight cores v0.14.6 left every judgment past its budget, so fail: closed refused every request. A single-machine install of v0.14.7 on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.

Upgrading.

• No migration.
• Upgrading does not change a judge already in the catalog. v0.14.6 and earlier installed two judges, reserving no cores or one, and 2 or 4 GB. To give them what they need: llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2, then llmgw ctl restart --model qwen3guard-0.6b. The two then need four cores on a single machine; below that, a longer latency_budget or fail: open.
• On a node whose cores are all reserved, ctl restart --model cannot start the new replica beside the old one and waits. Stop the old engine process, and the replica comes back with the new figures.

Fixed

• Both judges fit an 8 GB machine again. v0.14.6 had server init reserve 4 GB a judge replica. That figure came from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default, which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with --cache-ram 0, uses about 1.6 GB and stays there, and reserves 2.
• One machine runs one judge, on up to four cores. A second replica on the same machine shares its cores and is lost with it, so it bought nothing but half the speed of each. server init on a single machine now serves one judge on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two on two cores each, as in HA setups. --judge-replicas and the setup's Copies still choose.
• A judge gets the two cores it needs. v0.14.6 gave each judge replica one core on machines under eight. A judgment of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise latency_budget, where requests must not wait.

v0.14.6 — 2026-10-16

Setup that says what each part of an install is and lets every one be chosen and set; three machines that each run the gateway, so any one can be lost; a cluster that loses its store's primary without losing a request; one ComfyUI for an environment's models; pictures, 3D and workflows as MCP tools; alerts asked for at setup; extended thinking.

Upgrading.

• No migration.
• Control planes: with control.lease_ttl unset the lease is now 10 s (was 15). Upgrade the standbys first and the leader last; a mix of versions meanwhile is fine.
• The store (PostgreSQL): each member is dialled for 3 s before the next is tried. A store so far away that connecting takes longer needs connect_timeout=10 in its address.
• Nodes notice a control plane that died without closing its connection in 8 s (was 15) once they run this release.
• Judges installed by an earlier server init keep reserving no cores and 2 GB. To give them their own: llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2, then llmgw ctl restart --model qwen3guard-0.6b.
• Patroni replaces a lost primary within 20 s on a new cluster. On an existing one, run server init --patroni ... again from the machine with the authority; it changes only that.
• A floating address keeps VRRP router id 51 until server add --vip runs again, which moves it to one of its own; run it for every standby.
• A request whose judge is not there yet waits up to the policy's latency_budget for one before fail: closed refuses it.

Added

• Setup lets every part be chosen, each with its settings, and says what each is. sudo llmgw on a new machine lists the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model, pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is left out, with the same install as a command. Every setting is a server init flag too (--starter, --judge-fail, --media, --max-key-ttl, ...).

• Three machines are high availability for the gateway, not only the store. With three machines, setup makes the two database machines control planes too, standing by while the first leads, with an optional floating address (keepalived) that follows whichever leads. Any one machine can now be lost. By command: server init --patroni ... --standbys --vip 10.0.0.160/24. The mirror's management API answers the three control planes, so a standby that comes to lead can still fetch models.

• Pictures uploaded to work from: /v1/files takes purpose=vision or user_data besides batch, kept as long as job results, so a picture is sent once and named by its file-… id in the MCP tools and a workflow's inputs.

• One ComfyUI for an environment's models: ComfyUI models in one comfyui.environment (two or more) share one replica — the catalog makes it a hidden entry, comfyui-env-<name>, on the first member's profiles with the most memory any member needs, carrying every member's weights and custom nodes. Asking for any member starts it; it answers for each with that member's workflow and checkpoint, and the members get no replicas of their own. A picture model and a 3D model on one card stop handing it to each other with a cold start each way.

• Pictures, 3D models, vision and workflows as MCP tools at /mcp, beside the memory notes, each offered only where the gateway has a model for it: generate_image (with concept, side and texture styles), generate_mesh (from a picture or a prompt), review_images (a vision model, to a JSON schema), run_workflow, job_status and gateway_status. Each makes its request under the caller's key, so allowlists, limits, licences and budgets apply and it is charged like any request; what it makes is kept as the caller's file and answered by id and download URL. A client that takes a stream gets notifications/progress while a mesh is made and waits up to ten minutes; one that does not, 50 seconds, then the job id.

• Setup asks where alerts go: llmgw server init --webhook URL (repeatable) and a Notification webhooks field on both setup screens. Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text, any other the event as JSON (slack: or json: in front says which). They are added to the cluster's notify.webhooks beside those already there, and each is sent a test once the control plane runs on them.

• Extended thinking on /v1/messages: thinking: {type: "enabled"} tells a model that thinks by its chat template to (enable_thinking), and its reasoning comes back as thinking blocks before the text, streamed as thinking_delta events; disabled tells it not to. Not asked for, the reasoning is left out as before.

• How long on-demand models hold a card is a setting, in force on the leader within seconds: on_demand.evict_after_idle (2m unless set) and on_demand.min_lifetime (5m). On one card shared by a picture model and a 3D model, 1m each brings the wait between a picture and its mesh to about a minute.

Fixed

• The judges and the starter model divide a machine's CPUs instead of each taking all of them. server init gave the judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A judge also reserves 4 GB, not 2: it uses about 3.2.

• A lost store machine stops costing requests most of a minute. The store's connections try each member for 3 seconds, not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP keepalive, and TCP_USER_TIMEOUT on Linux). Patroni replaces a lost primary within 20 seconds (ttl 20, loop_wait 5, retry_timeout 5). Running server init --patroni again applies this to an existing cluster. Measured on three VMs with the primary's power cut: about 45 seconds of hanging requests before.

• Leadership stays put while the store's primary is replaced. Replacing a lost primary takes longer than the leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that, the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in 8 seconds (was 15), so it is on its way to the new leader as that one takes over.

• A judge replica that hangs no longer costs the request. All the tries of a judge request shared the policy's budget, so a replica that hung used it all and the try on another replica had none left: under fail: closed the request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before. On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge on a GPU.

• A new leader waits for the judge instead of refusing at once. The moment a control plane takes the lead it knows no judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's latency_budget lasts, so fail: closed no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A judge that is slow is not asked again, and an alarm is raised once per request, not once per try.

• Node discovery for Prometheus listed only the control plane asked and the store, so a machine that is both was a control plane to one and a store member to the other — scraped twice — and a standby asked alone lost every node. Every control plane lists all of them alike; deploy/monitoring's prometheus.yml says to scrape and ask each.

• A model whose weights and context pool did not fit its card was added and then failed to make its context on every start, taking the card from others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on 24 GB). ctl model serve shrinks the pool to what the card holds, saying so, and refuses weights that do not fit at all; the portal's serve form refuses such a pool and says the largest that fits.

• Removing weights from the mirror that a catalog model still loads is refused, naming the model: removed from the mirror alone, the model stayed in the catalog and its replicas went on loading the node's copy.

• Jobs ran one at a time, so a 3D job waiting for the card held up another model's job that could have run. They run side by side, one per model at a time and eight in all, on the leader as before.

• llmgw ctl model serve served an embedding model as a chat model: no --embedding, four slots, a 512-token batch, embeddings unset and a "say hello" smoke test it could not pass. A model whose GGUF says it pools (or --embeddings) is served for embedding: 32 slots of 1024 tokens unless sized otherwise, a batch that holds a whole passage, and one vector asked for as its smoke test.

• llmgw ctl upgrade checks every node's version once the waves are done and fails naming any not on the release, and says when the control plane has no nodes registered; llmgw upgrade on a control plane says how many nodes it left behind and the command that upgrades them. A site's nodes stayed a release behind with nothing to say so.

• The cluster and node pages swapped their live part every few seconds, forms and all, so filling one in raced the clock: what was typed, an open choice and the focus were lost. The refresh now waits while a field in it has the focus or a form in it has been typed into and not sent; the mirror page's reload during a download waits the same way.

v0.14.5 — 2026-10-14

Picture and 3D models that add themselves where a card fits them, with a card on the portal and a screen in the console to see and add them; a mesh's colour brought to the picture it was made from; and the whole licence written into every mesh.

Upgrading.

• The store gains a table (migration 0047: preset_state); nothing to do.
• presets.auto is on: once upgraded, the leading control plane adds flux-schnell and triposr, on demand, where a node's card fits one, the cluster installs ComfyUI (engines.comfyui) and the catalog makes nothing of that kind. Nothing is fetched until somebody asks for one. Turn it off with llmgw ctl model presets auto off.
• Nodes need this release for albedo_match (it is in the save node the gate puts in each replica); a ComfyUI replica takes it when it starts again.

Added

• A mesh's base colour brought to its picture's: LLMGWSaveMesh takes albedo_match with the picture and its mask, and scales the texture in linear light to the object's brightness in the picture. Stable Fast 3D takes the light out of what it sees and read a dark basalt (0.17) as a light stone dimly lit (0.67); matched, it is 0.17. On sf3d it is the albedo_match input (off unless asked).
• Picture and 3D models by themselves: once a node has a card that fits flux-schnell or triposr and the catalog makes nothing of that kind, the leading control plane adds the preset on demand — so a new cluster's portal has its Images and 3D pages without a definition written — where the cluster installs ComfyUI. presets.auto turns it off; a preset somebody removes is not added back. GET /api/v1/presets says how each stands and why one is not added; POST /api/v1/presets/{name} adds one (and sets engines.comfyui if the cluster has none); PUT /api/v1/presets/auto. Each automatic addition is in the audit log.
• Presets on the portal and in the console: the cluster page's Pictures and 3D models card lists each preset — what it makes, the classes it fits here, whether and by whom it was added, why the leader has not — with an Add button that says first when it would install ComfyUI and restart the control planes, and the switch for presets.auto. The console's models screen shows the same; A adds one, P flips the switch. ctl model add PRESET and ctl model presets go through the API (and ctl model presets auto on|off), falling back to the entry as v0.14.4 sent it on an older control plane.

Fixed

• A mesh carried only its licence's name, URL and notice; it carries the whole licence as /v1/models gives it — id, commercial use, excluded regions, revenue cap, attribution — for a pipeline that sorts assets by licence.
• A workflow's {{name|false}} default filled a BOOLEAN input with the word rather than a boolean.

v0.14.4 — 2026-10-13

Any ComfyUI workflow as a job, with typed inputs; picture and 3D models added by name; profiles that fit whatever card a site has, and ComfyUI on a V100; licences a machine can match and a person can leave out; a job waiting for the one card says what holds it. Many fixes to on-demand models taking turns on one card.

Upgrading.

• The store gains a table (migration 0046: user_excluded_licences); nothing to do.
• Nodes need this release for ComfyUI on Volta or Pascal cards (the cu126 build), for a driver from NVIDIA's open kernel module to be read, and for a node that loads its weights by repository name (sf3d); a ComfyUI already installed is kept.
• Control planes before nodes: an entry using the built-in triposr workflow, or a gpu: need instead of a class, is refused by an older control plane.
• Models added with llmgw ctl model add <preset> take the preset's name; remove an entry of that name first.

Added

• The client API described: OpenAPI 3.1 at /v1/openapi.json, no key needed, generated from a route table a test holds to the router's own; and /docs/api, Swagger UI (served from the gateway) over it and, for staff, over the admin API's document.
• A job waiting for the card says so: progress is {"state":"waiting_gpu","held_by":…,"eta_seconds":…} while another on-demand model holds it; a direct request's 503 carries X-Llmgw-Waiting-For and X-Llmgw-Eta-Seconds.
• Any ComfyUI workflow, as a job: POST /v1/workflows/runs with {model, inputs}. A model's catalog entry declares its inputs (comfyui.inputs: text, int, float, bool, choice, image, mask, audio, video, file, with bounds and defaults) and its output (comfyui.output: pictures, meshes, video, audio or files); a request is checked against them before a job is written, file inputs are base64, a data URL or a /v1/files id, and /v1/models lists each model's workflow so a client can build its form. A PNG result carries the licence in an iTXt chunk, a GLB in asset.extras as before.
• Environments: comfyui.environment names ComfyUI models that agree on their custom nodes and may share one replica; the catalog refuses one that cannot.
• Profiles that fit the hardware: a profile may say gpu: {min_compute_capability, min_memory_gb} instead of gpu_class, and stands for every class in the cluster that meets it, as <profile>@<class>; a ComfyUI profile may carry its own comfyui workflow, checkpoint and nodes, and any profile its own source. One catalog entry then runs fp8 where the card has it and fp16 on Turing.
• Custom nodes that compile: nodes[].build names directories of a node that are C++/CUDA packages; the gate builds each as a wheel for the card the replica runs on, keeps it on the node, and installs it beside the replica. comfyui.huggingface_repo shows a model's weights to a node that loads them by repository name.
• A licence a machine can match: license.id, commercial, excluded_regions, revenue_cap_usd, attribution.
• On-demand models in /v1/models, with loaded saying whether one is up now; a picture is cut out and put on grey before TripoSR sees it (LLMGWPrepareForMesh, rembg when it has no alpha).
• Licences a person leaves out: on the profile page, by licence, or by rule — every licence forbidding commercial use, every licence granting nothing in a region — so a model under one is neither listed to nor served for that person's keys and sessions (403 licence_excluded).
• Presets: a picture or 3D model added by name. llmgw ctl model add flux-schnell or triposr adds an entry that comes with llmgw — its workflow (TripoSR's now built in), custom nodes, packages and licence, and a profile by card need rather than class — and has the nodes install ComfyUI when the cluster's settings name none. llmgw ctl model presets lists them; the console's A on the models screen offers them beside a definition file.

Fixed

• A model waiting for another on-demand model to give up the card failed at once as "cannot be placed"; it now waits, and a job with it.
• A model evicted for being idle was placed again on the next pass, ahead of the one it made room for.
• A job or batch leased by a standby control plane, whose router has no endpoints, waited on "loading" for good; they are served only by the leader. A job told its model fits nowhere asks again for five minutes before failing, which an agent re-registering makes true for seconds.
• An on-demand replica adopted after a restart was never idle enough to evict, so a model waiting for its card waited out its whole idle timeout; it counts as used at adoption.
• A picture in a prompt was counted as its base64 by the context check, so one 1280x720 PNG was "138 thousand tokens" and a vision model refused it; a picture or a sound now counts as about a thousand.
• An on-demand model of a higher class was evicted, idle, for a lower-class one two minutes into a lull; it keeps the card through its idle timeout, and a model waiting on it is told so.
• A model removed from the catalog stayed wanted, so added again it loaded unasked.
• A node that loads its weights by repository name (sf3d) went to Hugging Face for a gated repository and got a 401: the agent did not hand the gate comfyui.huggingface_repo, so the weights on the node were never linked into its cache.
• ComfyUI on a V100. torch's CUDA 12.8 and newer builds have no Volta kernels, but its cu129 build was taken to run from compute capability 7.0, so on a V100 both ComfyUI builds were installed, about 10 GB each, and failed with "no kernel image". Cards older than Turing now get ComfyUI on PyTorch's CUDA 12.6 torch (build cu126); a Turing or newer node keeps the build it has.
• The open NVIDIA driver read by its version. With the open kernel module, the default since 560, the agent read the driver as "for" and so as unknown, and tried every engine build whatever driver it needs.

v0.14.3 — 2026-10-10

3D models from a picture, as jobs: made by ComfyUI on the cluster's cards, with their progress, their licence written into the mesh, a price per mesh and a page to make them. Also: a profile's own weights, a band for bulk work, and fixes for control planes behind a balancer and agents that restart.

Upgrading.

• The store gains tables and columns (migrations 0043–0045: notify_sent, jobs, usage meshes); nothing to do.
• Nodes need this release for their engines' output to survive an agent restart and for ComfyUI's custom nodes and progress; engines started by an older agent keep its pipe until they are restarted.
• A 3D model needs a model that makes them (see install.md, "A model that makes them"); none is added.

Added

• Jobs: 3D models asked after, not waited for. POST /v1/3d/generations answers 202 with a job id; GET /v1/3d/generations/{id} says its status, progress and, when done, the file its result is in (/v1/files/{id}/content); POST .../cancel stops it, in the engine's queue too. The request is replayed by whichever gateway holds the job's lease through the same path a client's request takes — key, allowlist, limits, admission, guardrails, accounting — so it is charged like any request, tagged job=<id>, and a model still loading is waited out for as long as the job has (24 h) rather than refused. A job survives the gateway that took it; results are kept for router.batch.retain_days. Migration 0044.
• ComfyUI models that make meshes (comfyui.output: meshes). The picture is uploaded to the replica's ComfyUI and the workflow's {{image}}; the mesh the workflow saves is the job's file, an OBJ turned into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards stronger ones can be added the same way.
• Custom nodes (comfyui.nodes): a repository at a full commit, put in each replica's own custom_nodes by the gate, with its packages installed beside the replica, never into ComfyUI's environment.
• A job's progress: the gate listens to ComfyUI's websocket and the job says queued and how many ahead, then the node running, its step and percent.
• A model's license (name, url, notice), in /v1/models, in the job, on the portal and in the GLB's asset.extras.
• per_mesh in auth.pricing; usage rows, sums and exports count meshes.
• /3d on the portal, beside /images: a picture chosen, the job followed, the mesh in a viewer with a download and its licence; and llmgw ctl 3d list|get|cancel for every owner's jobs.
• A profile's own source: the weights it loads instead of the model's, so an embedding model can run on vLLM from its repository on a card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers nodes holding each profile's weights, ctl model add checks every source, and ctl weights collects none a profile loads.
• A band's own queue_ttl (router.admission.bands[].queue_ttl): how long its requests may wait for room, instead of the admission's 30 s. A band for bulk work — a knowledge-base import — waits behind people rather than being turned away.

Fixed

• An engine's output no longer breaks when its agent restarts. Engines wrote to a pipe of the agent's, which an agent that restarted and adopted them left with no reader; an engine's next large write failed, and in a Python engine took the request it was reporting on with it. Each process now writes to its own file in the agent's work directory, and an adopted engine's log keeps what it wrote before the restart.
• Single sign-on behind a balancer. The cookie that carries a sign-in across the identity provider was signed with a key of each process's own, so with two control planes a callback that reached the other one was refused. The key is now a secret in the store, the same on every control plane.
• Notifications sent once by two control planes. Each kept its own hour of quiet per event, so a budget spent on a team both served was announced twice. The control plane that claims an event in the store (migration 0043) sends it; with the store unreachable it is sent anyway.
• A request turned away for a busy cluster says when to come back. A request that waited as long as it may (504) or found the queue full (503) had no Retry-After, so llmgw-kb counted the passage as failed instead of sending it again; both now say 10 seconds.
• The usage sums and exports count pictures (images), as the request records did; pictures made before are added to the sums.

v0.14.2 — 2026-10-10

Pictures: OpenAI's /v1/images/generations, made by ComfyUI on the cluster's own cards. The agents install ComfyUI themselves, the gateway prices pictures by the picture, and the portal has a page to make them. Also: on-demand models that survive a failover, and knowledge-base embeddings fixes.

Upgrading.

• The store gains a column (usage.images); nothing to do.
• ComfyUI is installed only where asked: llmgw ctl config set engines.comfyui 0.39.2 has every node with an NVIDIA GPU build it (about 10 GB under /opt/llmgw/engines). Nodes need this release to install it or run engine comfyui.

Added

• Pictures at /v1/images/generations, OpenAI's shape, through the same key, allowlist, limit, admission and guardrail checks as any request. A model with profiles on engine comfyui is placed on a GPU node like any other. A ComfyUI server outside the cluster can be an external model with api: comfyui instead. The workflow is a built-in (flux-schnell, the default, or flux-dev) or a JSON file in ComfyUI's API format, with {{prompt}}, {{width}}, {{seed}} and the like where the request's values go. Requests may also send seed, steps, negative_prompt and guidance. The seed used comes back in x-llmgw-seed. Pictures come back as b64_json; ComfyUI's workflow is stripped from each PNG, and a client that leaves takes its workflow off the queue. /v1/models says images for such a model.
• Engine comfyui. ComfyUI asks for no key, so the agent starts it behind llmgw comfyui-gate. ComfyUI listens on the loopback only. The gate listens on the replica's port, demands the replica's key, and lets through only what the router asks. The model's comfyui: block names the workflow and, for a source that is a set of files, the checkpoint.
• The agents install ComfyUI (engines.comfyui). The release's source is checked against a digest llmgw holds, since GitHub publishes none, and a version without one is refused. Its requirements are resolved as PyPI stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's CUDA 12.9 one for an older driver, and the result is checked on the card. A ComfyUI installed by hand is named by engines.comfyui.dir in agent.yaml.
• A price per picture: per_image in auth.pricing, on top of the prompt's tokens. A model priced only per picture is not free, so a spent budget still stops it. Each usage row keeps how many pictures were made, shown by llmgw ctl request.
• /images on the portal, beside the chat, for the models that make pictures. The chat no longer offers them. The page offers size, how many, seed, steps and a negative prompt, and shows each picture with its seed and a download link. Nothing is kept but the usage.
• An embedding model's query_prefix, put before the question a knowledge-base search embeds and never before documents, for Qwen3-Embedding and models trained like it.

Fixed

• An awake on-demand model survived no restart or change of leader of the control plane. A control plane that had heard no requests read every on-demand replica its nodes reported as wanted by nobody, and drained them all. An adopted replica now gets a whole idle timeout.
• ctl models said not placed for an on-demand model that had been woken and was serving; it now says ready (on demand).
• A batch of embeddings was refused as one over-long prompt when the passages together exceeded the context, though each one fit. The longest input is what has to fit.
• The chat's kept conversation sat in the browser under one key for everybody, so the next person to sign in on the same browser opened the last one's conversation. It is kept per account, and signing out clears what the pages stored.

v0.14.1 — 2026-10-09

Adapters trained on the nodes from a knowledge base and kept private to their owners; the cluster's and the agents' settings edited in one place; the tenancy tree and keys over the admin API and in Terraform; knowledge bases that hold up with many documents; the firewall setting on every machine.

Upgrading.

• Nodes need this release to follow agents.* in the cluster's settings; until a node is upgraded its agent.yaml is what it runs on.
• A separate mirror machine and store members that run nothing else follow the firewall setting once what reads it is installed: run sudo llmgw mirror init on the mirror again with the flags it was set up with, and llmgw server init --patroni again for the store's members. A control plane set up before v0.12.0 needs sudo llmgw firewall sync --install.
• The store's PostgreSQL memory applies after each member is restarted, one at a time, the leader last; until then it runs as before.
• The store gains tables and columns; nothing to do. Existing knowledge-base collections get their full-text index when a control plane starts, and keep vector storage; only new collections use halfvec, and only on pgvector 0.7 or later.
• Cached knowledge-base passages in vLLM are made again once, under the new cache_salt.

Added

• The cluster's settings on the portal (BACKLOG E4): /admin/settings for administrators. It edits the document llmgw ctl config edit opens: checked as a control plane would load it, refused when made from a version since replaced, with the history to view and roll back. Changes are audited by size, never by contents.
• The nodes' agent settings in the cluster's settings (BACKLOG E3): agents.default and agents.nodes.<id> hold everything agent.yaml can say but the agent's id, control planes, address and certificate. They are sent to the agents as they change; an agent starts again on new settings, or on an edited agent.yaml, with its engines serving throughout. Settings that would not load are refused on save, and set aside on a node rather than keep its agent from starting. Nodes need this release to follow them.
• LoRA adapters trained on the nodes (BACKLOG F4): llmgw ctl kb dataset writes question/answer pairs from a collection with a model of the cluster. llmgw ctl train start trains an adapter on a node with its installed vLLM's torch and a small overlay, on a card with enough memory free (beside an engine, if it leaves that much), converting to GGUF for llama.cpp. llmgw ctl train publish keeps it on the mirror (which now accepts uploads) and makes it the model's adapter.
• Adapters with owners. An adapter's owner (org, team, project, user or key) is the only part of the tree that sees it; to anyone else it is a model that does not exist. An adapter trained on a knowledge base belongs to a part of the tree that may read it, and is never made public.
• The knowledge bases' vLLM cache_salt follows the collections searched, not the organization, so two teams of one organization no longer share cached passages.
• Organizations, teams, projects, API keys and memberships in the admin API: /api/v1/orgs (the tree), POST /api/v1/teams and projects, PATCH to rename or move, DELETE for what is empty; /api/v1/keys to issue (the key is in the answer, once), list and revoke; /api/v1/users and /api/v1/users/{id}/memberships. They were only llmgw org, keys and users on a control plane before. The Terraform provider manages them as llmgw_org, llmgw_team, llmgw_project, llmgw_key (the key in a sensitive secret) and llmgw_membership.
• The firewall setting reaches every machine. A separate mirror and the store's members follow firewall.enabled and firewall_networks as nodes and control planes did: the leading control plane tells the mirror over its API, and keeps the state in the store for members that run no llmgw, which a timer on each reads. The mirror also lets in the network of every node the leader knows. mirror init and server init --patroni install what follows it; run them again on a cluster set up before.

Changed

• The store's PostgreSQL is sized to its machine's memory. Before, it ran on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one machine. Now each member gets a share of its own RAM, a smaller one where a control plane or agent shares the machine, and the share is read again every time server init --patroni runs. The new cache is applied by restarting one member at a time, the leader last.
• Knowledge-base searches hold up with many documents. New collections store their index as halfvec where pgvector is 0.7 or later (half the memory, and embeddings up to 4000 dimensions instead of 2000); older collections, and clusters on pgvector 0.6, keep vector. Each collection has its own full-text index, as it has its own vector index, made for existing collections when the control plane starts. The text arm ranks by the cheap measure first and scores only those candidates.

Fixed

• A knowledge-base search asking for more than 10 passages (X-Llmgw-Knowledge-K) got at most 40 candidates per collection from the vector index, whatever it asked for: pgvector's hnsw.ef_search is now raised to the number wanted, up to 1000.
• llmgw server backup and restore against a replicated store over TLS failed looking for ~/.postgresql/root.crt: pg_dump was given sslmode=verify-full without the cluster's authority.

v0.14.0 — 2026-10-08

Knowledge bases: a company's documents, put in by people from their own machines, kept apart per tenant, and answered from.

Upgrading.

• pgvector on a cluster set up before this release. Knowledge bases need pgvector, which server init and the high-availability database setup now install, but a rerun on a running cluster does not:
  1. Install postgresql-<version>-pgvector on every database machine.
  2. As postgres, run CREATE EXTENSION vector in llmgw's database.
  3. Restart the control planes. Until then everything else works, and llmgw ctl kb list says why knowledge bases are off. On a SQLite store they stay off.
• An embedding model. A collection needs a catalog model with capabilities.embeddings, for example bge-m3, whose weights are in the mirror.
• The client. People download llmgw-kb for their system from this release.

Added

• Collections and who may read them. A collection is owned by an organization, team, project or person. Grants add up down the tenancy tree and never cross an organization. A collection a key may not read is answered as one that does not exist. Chunks are kept in PostgreSQL with pgvector, partitioned by organization, with an HNSW index per collection, and row-level security on the table as a second line behind every query's own filter. A project, team or organization that owns collections is not deleted; a person's collections go with their account.
• llmgw-kb, its own download for Windows, macOS and Linux, run by people with their own key. It reads documents where they are and sends text and vectors, never the files, for what changed only; a SQLite ledger on the machine remembers what it did.
  • It reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML, MHTML, mail with attachments, notebooks, subtitles, ZIP archives, and text and source code in any common encoding.
  • Legacy Office files are read with LibreOffice, and scanned pages with Tesseract, when those are installed.
  • --dry-run says what would leave the machine; --delete removes only what that machine sent.
• Answers from documents. A chat request names collections in X-Llmgw-Knowledge, or its part of the tree has a default (llmgw ctl kb default). The best passages, chosen by meaning plus a little for the question's words, are added to the system message, numbered, and x-llmgw-sources says which were used. This happens before the guardrails judge the request, and the answer is never served from the answer cache. On vLLM replicas the organization's cache_salt keeps tenants' prefix caches apart. It works through the Responses and Messages APIs too.
• Administration: llmgw ctl kb (create, list, grant, revoke, default, docs, rm, search, delete), /api/v1/kb, the console's Knowledge screen, and a Documents picker on the portal's chat page.
• Metrics llmgw_kb_retrievals_total and llmgw_kb_retrieval_seconds; docs/knowledge.md.

v0.13.0 — 2026-10-08

The whole system stopped, started and restarted from one command, a guardrails judge that no longer refuses ordinary coding requests, and the rest of the field report from v0.10.6.

Upgrading. A cluster keeps the judge its policies name: one installed before this release goes on asking Llama Guard 3 1B until llmgw ctl guard judge use qwen3guard-0.6b, which the leader then fetches and serves by itself, and which replaces the policies' Llama Guard category codes.

Added

• A judge on a GPU and one on CPUs together: a policy's judge.models is a list asked in turn, the first that answers giving the verdict; the next is asked when one has no replica serving, fails, or misses its latency_budget. llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6b sets the order for every policy with a judge, and guard judge status says which judge stands in for which. A judge on a GPU is served as a standard model, so it takes a card only from what may give one up.
• A judge a policy names is fetched if it is missing: the leader fetches the weights of a judge llmgw knows (qwen3guard-0.6b, -4b, -8b, llama-guard-3-1b, -8b) into the mirror and serves it, when a policy names it and the catalog does not have it — and again if it is removed. guard judge enable --gpu serves Qwen3Guard-Gen 4B on the class with the most free cards.
• The whole system stops, starts and restarts from one command: llmgw stop takes every model down and answers every request for one with 503 (system_stopped), keeping the control planes, agents and mirror running and no model's definition changed; llmgw start brings every model back. It is the cluster's setting serving.stopped, live on every control plane. llmgw restart restarts every node's agent and every control plane, the leader last, with the engines serving throughout. llmgw status says whether the system serves. One machine's services are llmgw service status|start|stop|restart.
• A replica says what it computes on: ctl replicas (ON), the console and the admin API show CUDA with the GPUs its processes hold memory on and how much, as nvidia-smi sees them, or CPU — for every engine, kept by the agent across its own and the control plane's restarts. llama.cpp's layer count is added when it says it, which it does only at -lv 4.
• ctl engine-check llamacpp computes a token on the GPU: a small GGUF (--model, or the smallest in the weights cache) is loaded with every layer on the GPU at -lv 4 and asked for one token; a build whose CUDA backend does not load fails, by llama.cpp's own account of its layers. With no --bin it checks the build the agent installed.
• Engine versions and compute capability on nodes: ctl nodes shows llamacpp b10991 and 6000-24g (7.5); the admin API gives engine_versions and compute_capabilities. A profile's engine_version restricts it to nodes whose engine version starts with it, and explain-placement says which version a node has, or which engines.
• ctl restart --max-surge N and --max-unavailable N: that many of a model's replicas restart at once, beside themselves or stopped first.
• Multi-node readiness apart from the node's: doctor gives both, with fabric findings in a section of their own; node add says the multi-node one; ctl nodes shows a declared fabric's adapters that are down.
• A forgotten password is set through a link: /forgot mails one when the cluster's mail settings say how (SMTP, sender, the public base_url it points at), saying the same whether or not the address has an account; /reset sets the password once and closes every session. Without mail, llmgw users reset-link makes the link for an operator to hand over.

Fixed

• A replica restarted in place keeps its room: the replica was stopped and the pass went on to other models before placing its replacement, so a model waiting for room took the freed cards. On the reference cluster a GPU judge waiting for the only card took it from the model being restarted there. The replacement is now placed in the same step.
• The judge no longer refuses ordinary coding requests: Llama Guard 3 1B, the judge server init installed, could not tell "continue", "devam et" or "how do I kill all child processes in bash" (0.51 to 0.92 unsafe) from harmful requests (0.97 to 1.00), at any quantisation. The judge is now Qwen3Guard-Gen 0.6B (format qwen3guard), which gave the same ordinary requests 0.15 at the most and the harmful ones 1.00, on CPUs, in 0.8 GB. Its third verdict, controversial, counts as safe unless the policy says controversial: unsafe. guard judge use qwen3guard-0.6b moves an installed cluster to it, and replaces Llama Guard's category codes, which Qwen3Guard never names, with the baseline's.
• A long message is judged rather than refused: a judge read the whole last user message, and one with a file pasted into it overflowed the judge's context, so it gave no verdict and fail: closed refused the request. It now reads the start and the end, about 6000 characters.
• A replica no longer stays "starting" while it serves after a restart: an agent that registered before its adopted engines had answered their health check never said they had, so the control plane kept them starting and routed nothing to them — on the reference cluster, after llmgw restart, the only replica of a model. The agent now reports each shard the registration called not ready once it is.
• A failover no longer replaces healthy replicas: a standby that took over placed at once, before the agents had reconnected and said what runs, so it started a second copy of replicas it did not know of and then drained the healthy ones as surplus — on the reference cluster the guardrails' judge and a model's replica, within twenty seconds of a leader restarting. Placement now waits after a takeover as it does at start.

Changed

• A GPU placement llama.cpp says it cannot compute on is refused when it warns that it found no usable GPU, even where nvidia-smi cannot be asked; where it can, holding no memory on its GPUs refuses it as before.

v0.12.2 — 2026-10-06

The guardrails turned off and on as a whole, and a judge that refuses ordinary requests explained: the hazard it saw is told to the client and kept with the request, and llmgw ctl guard judge ask shows what it says of any text. Nothing to do to upgrade; the store gains a column.

Added

• The guardrails turn off and on as a whole: llmgw ctl guard off / on, which is guard.enabled in the cluster's settings — in force within seconds, the policies and their attachments kept. ctl status, ctl guard list and the guardrails page say when they are off.
• llmgw ctl guard judge ask TEXT has the judge read a text and shows its answer as given and as read — for finding out why ordinary requests are refused. Nothing of the text is kept, not even in the audit.

Changed

• A judge's refusal names the hazard: the client is told (judge: S6,Specialized Advice) rather than (judge), and the verdict is kept with the request (guard_rule): ctl request, the server's log and the guardrails page show it. Llama Guard 3's codes come with their names, S9 Indiscriminate Weapons, without a categories list.

v0.12.1 — 2026-10-06

The guardrails' judge, served and watched without server init: a judge missing, served as the wrong kind of model, or not answering is said on ctl status and the console, and llmgw ctl guard judge enable puts it right. Nothing to do to upgrade.

Added

• llmgw ctl guard judge status and enable: each judge model the policies ask, whether it is in the catalog, as a judge, with replicas ready and answering — and serving the missing one, Llama Guard 3 1B on CPUs under the name the policies give it, with --replace for a model of that name that is not a judge. No more server init to put a judge back. Also Do → Serve the guardrails' judge in the console.
• A missing or broken judge is said where it is looked for: llmgw ctl status and the console's Overview list each judge with a problem — not in the catalog, not a judge, no replica ready, or failing to answer — and what that does to requests (refused or unjudged).

Fixed

• A Llama Guard file served from the mirror page is the judge: it was added as a model clients use, under whatever name the form had, so the policies never found it and refused everything they judged (guard_unavailable). It is now added as a judge, under the name the policies ask.
• ctl says so when the API answers with a page: a command a control plane does not know — the leader on an older release, through a standby — failed with "invalid character '<'"; it now says which and why.

v0.12.0 — 2026-10-06

A minor version, because where the configuration lives changes: everything in server.yaml that is not one machine's own is now the cluster's settings, kept in the store, the same on every control plane and in force within seconds of a change. Nothing needs doing by hand to upgrade. The first control plane to start on v0.12.0 brings its server.yaml's cluster keys into the store, with the baseline guard policy when the file had none. From then on, change them with llmgw ctl config or the console's Settings, not in server.yaml, whose cluster keys are ignored, with a warning. Upgrade every control plane (llmgw ctl upgrade does, the leader last): until the leader runs v0.12.0, llmgw ctl config through a standby is refused. Nodes need v0.12.0 to follow the firewall switch.

Changed

• The cluster's settings live in the store, and every change is in force within seconds. Everything in server.yaml that is not one machine's own (guardrails, limits and prices, routing, engines, notifications, the mirror's policy, declared models, offline, federation, facility budgets) is a versioned document in the store, the same on every control plane. llmgw ctl config shows, sets, edits, lists and rolls it back, and the console's Settings screen edits it. A change is checked, kept with who made it and why, and applied: guardrails live, anything else by every control plane starting again on it, standbys first and the leader last. A change to a machine's own keys in its server.yaml is noticed and applied the same way. No "restart now?" any more, and no second copy to keep in step. The first start against a store brings in the file's cluster keys, with the baseline guard policy when the file had none, so an upgraded cluster gets its guardrails; after that, those keys in a server.yaml are ignored, with a warning.

Added

• llmgw's firewall can be turned off and on for the whole cluster: llmgw ctl config set firewall.enabled false. Every node's agent takes its table out within seconds, and each control plane's new llmgw-firewall-sync.path does the same there; true puts the rules back as they were. firewall_networks changes reach the nodes the same way.
• llmgw wipe removes llmgw from a machine — services, firewall table, configuration, the authority, secrets, state, users — after listing it and asking for the machine's name; --models, --store and --all take weights, the store's database here, backups and the binary too. Also Do → Wipe this machine in the console.

Fixed

• The mirror is given an address the nodes reach: server init and mirror init took the machine's first address for the mirror's --base, which on a machine with Docker could be 172.17.0.1 — a node then asked its own bridge for every model and was refused. They now take the source of the default route, else the first address not on a container's or a bridge's interface. And node add has the node reach the mirror's files before it is added: one that cannot fails the checks, naming the address and what to change.
• A standby that becomes the leader reads the store again: models added or removed, engine builds chosen and domains held through the leader before it, rather than placing on what it read when it started.
• A node on another network joins: node add set up the node's firewall but left the control plane's to the networks it was installed with, so a node on another subnet was installed and then never heard from (lost). It now lets the node in on the control plane it runs on — its subnet, or its address when a router is between them — and says so; a network already let in changes nothing.

v0.11.4 — 2026-10-05

Two control planes, run in anger on the reference cluster.

Added

• The console sets up a second control plane: Do → Add a control plane asks for the machine and a floating address and runs server add.

• Who leads, everywhere. ctl status and the console's Overview have a CONTROL line: which control plane leads, which stand by, one that does not answer, and whether the floating address is on the leader.

• A standby's console is the leader's. Its admin API passes /api/ to the leader over the agent port, with the cluster's certificates both ways, so sudo llmgw and llmgw ctl show the same cluster on either machine. Control plane certificates are now issued for client use too; server add reissues older ones.

• ctl upgrade in waves. Nodes are upgraded one, then two, four… up to --parallel (20) at once, so hundreds take minutes; a wave that does not come back stops it. The other control plane is upgraded first, over ssh.

• server add says when the mirror's --api-allow would refuse the new control plane.

• Guardrails in the console: a view of the policies and what each checks, whether their judge models are serving, where they are attached (attach and detach from it), and what they did in a window. GET /api/v1/guard/events answers the same, and GET /api/v1/guard now says what each policy does.

• A standby's portal is the leader's. Its client port passes everything but /health, /ready and /leader to the leader, and its admin port the dashboard as well as the API; a proxy left pointing at a standby showed an empty cluster. The leader trusts the control planes' X-Forwarded-For.

• ctl model serve --tools says in the catalog that a model makes tool calls, for agents and editors.

Fixed

• Continue's agent did nothing. The configuration the portal writes told Continue every model uses tools, and nothing of its context: a model whose calls came back as text stalled the agent, and prompts over a CPU replica's 4096 tokens were refused. It now says tool_use only for a model whose catalog says tools, and the context and longest answer the gateway allows.
• ctl model serve gave llama.cpp four slots whatever was asked: a --context-length 32768 --ctx-pool 32768 model had slots of 8192 and refused what the gateway admitted. Slots are now what the pool holds of the context. The longest answer grows with the context (a quarter, from 1024 to 8192) rather than being 1024 for every model.
• ctl upgrade upgrades the leader last wherever it is run; run on the standby, it upgraded the leader first and the leadership moved twice.
• Failover took 40 seconds; it takes a few. A stopping control plane held every agent's stream for the thirty seconds of a graceful shutdown; it now gives up its lease first and lets the agents go at once.
• A control plane that lost its lease while running kept its agents, leaving the new leader with no nodes. It now lets them go.
• An agent's reconnect backoff carried over from one outage to the next, so each failover made the next one slower; it starts again after a session that worked.
• The CONTROL line named no floating address.

v0.11.2 — 2026-10-05

Routing by what each vLLM replica's cache holds

• A vLLM profile can set kv_events: true. The agent then starts the engine publishing its KV cache events (--kv-events-config) to a port on 127.0.0.1 the agent listens on, with --prefix-caching-hash-algo sha256_cbor and PYTHONHASHSEED=1729, so its block hashes can be computed outside Python. The events never leave the node.
• The agent mirrors each engine's set of cached blocks and sends the net changes up its existing stream to the control plane four times a second (new KVEvents agent message), starting every new stream with a full copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or AllBlocksCleared start a replica's index again; blocks offloaded to CPU, remote ones and a hybrid model's second cache group are not indexed.
• The control plane passes them to the router's index, only for replicas the sending node runs, and empties what a stream described when it closes.
• For a model with kv_events, the router asks one of its replicas to tokenize the request (vLLM's /tokenize, with messages and tools, so the chat template is applied exactly), hashes the tokens into blocks as vLLM does, and scores replicas by the share of the prompt's leading blocks they hold. The imbalance guard still applies. With no answer — no events yet, an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls back to the approximate tree (router.prefix) or to load.
• The agent checks first blocks against llmgw's hashing; an engine that hashes another way (a profile overriding the algorithm, a changed vLLM) is reported in the agent's log and llmgw_kv_index_problem, and routed as though it published nothing.
• New settings router.kv_index.tokenize_timeout (250ms), min_blocks (4) and max_blocks (4096). New metrics llmgw_kv_index_decisions_total (kv_hit, kv_miss_min_load, load_balance_override, fallback_no_index, fallback_unsupported, fallback_tokenize_failed), llmgw_kv_index_tokenize_seconds, llmgw_kv_index_blocks.
• Experimental: tested against vLLM's event format and hashing as its source defines them, and against libzmq and msgspec publishing — not yet against a running vLLM.

LoRA adapters on vLLM, loaded on demand

• vLLM models can carry adapters. A PEFT adapter directory (a mirror version ending in /, a path on the nodes, or a Hugging Face repository id) is registered at start with --enable-lora --lora-modules, and a request names it as its model, as on llama.cpp. scale applies on llama.cpp only. SGLang and TensorRT-LLM profiles still refuse adapters.
• max_loaded_adapters: N on a model has each replica hold at most N of its adapters. Replicas start holding none (vLLM with VLLM_ALLOW_RUNTIME_LORA_UPDATING=True and --max-loras N, unless the profile's args set their own). The router loads an adapter on a replica through /v1/load_lora_adapter when a request first wants it there, and unloads the least recently used one no request is running on to make room. vLLM profiles only; llama.cpp keeps loading every adapter at start, since llama-server can rescale an adapter it started with but not load another.
• Adapter affinity: a request for an adapter goes to a replica that has it, or is loading it, while that replica is not saturated; otherwise to one with room for another adapter; otherwise where load sends it — which is how a busy adapter spreads to a second replica. Requests arriving during a load wait for it instead of loading the adapter again.
• Self-correcting: each replica's /v1/models is read when the router first meets it and every 30 seconds, so adapters loaded before a restart or by another router are known; a 404 for an adapter the router thought loaded loads it again and retries. An adapter vLLM refuses (not found, rank above --max-lora-rank) fails the request with vLLM's reason, without trying it on every replica.
• Metrics: llmgw_adapter_affinity_total{model,result=hit|joined|loaded}, llmgw_adapter_loads_total{model,outcome}, llmgw_adapter_evictions_total{model,outcome}.
• The admin API and ctl model adapters read the file as GGUF only for a model with llama.cpp profiles.

Federating clusters: another llmgw cluster as a provider

• llmgw ctl federation add NAME --url URL --key-file FILE [--send-tags] makes another llmgw gateway a peer. It is checked first (its /ready, timed, and its /v1/models with the key) and kept in the store (migration 0029, federation_peers) with the models it offers.
• A peer's models join the catalog as NAME/MODEL, always prefixed, and follow the peer's own list every federation.interval (10s). A model the peer drops goes, unless a model here still falls back to it: then it is kept withdrawn (in the catalog, sent nothing) until the fallback changes. The peer's aliases and its own peers' models are not taken.
• A peer whose /ready fails, which refuses the key, or which does not answer is taken out of routing until it answers again. /ready now counts only a gateway's own endpoints, not its peers'.
• llmgw ctl model fallbacks MODEL [FALLBACK...] (and PUT /api/v1/models/{name}/fallbacks) sets a runtime model's fallbacks, so overflow can go to a peer's model.
• Requests to a peer carry x-llmgw-hop with the ids of the clusters they have passed through (each cluster's id is kept in the store). A request that comes back to a cluster is refused with 508 federation_loop, and a peer a request came from is not sent it.
• The peer sees one key, this cluster's, and the request's X-Request-Id. With --send-tags, also x-llmgw-tags: federated_from=<federation.name>, org=,team=,project= (ids); off by default.
• In-cluster residency and no_store keep requests off peers, by name and as fallbacks. Usage records the peer's model as served, and peer:NAME/MODEL as the replica.
• ctl federation list|remove|sync, a Federation view in the console, llmgw_federation_peer_up, _latency_seconds and _models metrics, admin API /api/v1/federation, and a "Federating clusters" section in docs/install.md. New federation: block in server.yaml (name, interval).
• A response header the gateway already set (X-Request-Id, x-llmgw-served-model) is no longer repeated from the upstream's.

Facility-aware placement: racks, PDUs, holds and power caps

• Where a node is, from five labels: llmgw.io/site, room, row, rack and pdu (a list, for a dual-fed rack), in agent.yaml or set from the control plane with llmgw ctl label NODE key=value… [key-…], which is kept in the store (migration 0031, node_labels), wins over the agent's labels, works before the node joins, and refuses a misspelt llmgw.io/ key. A node with no location places as before.
• Replicas spread across racks and PDUs. A model's next replica goes to a node sharing no rack or PDU with its others when there is one: an ordering above weights already on disk and bin packing, never a filter.
• keep_within: rack (or row, room, site) on a profile keeps a multi-node replica's nodes in one such domain.
• Holds. llmgw ctl facility hold rack:r12-07 --reason … [--drain] places nothing new in a rack, row, room, site or PDU; --drain also drains its nodes. llmgw ctl facility release undoes both. Kept across restarts (migration 0031, facility_holds).
• Power caps. facility.budgets in server.yaml gives a domain's cap_watts and other_watts, with a margin; a replica's draw is the profile's power_watts, or its cards at the power limit they report (gpu_watts otherwise) and its cores at cpu_core_watts, with node_watts per node. Use is the larger of reserved and measured, node by node; a shard that would take a domain past its cap less the margin is refused, and a multi-node replica's shards are counted together. Unknown draw never blocks.
• Seen everywhere it matters. ctl explain-placement names the held domain or the kilowatts, and keeps the rejections of a replica that was placed, so one in an unexpected rack says why. ctl nodes has LOCATION, ready, held and --domain. llmgw ctl facility lists every domain with nodes, use, limit, reserved, measured and holds. The console's Nodes view has WHERE; a node's menu has Location… and Hold rack…/Release. Admin API: GET /api/v1/facility, POST /api/v1/facility/holds, DELETE /api/v1/facility/holds/{domain}, POST /api/v1/nodes/{id}/labels.
• Docs: docs/facility-aware-placement.md says what is built and where it departs from the design; docs/install.md and docs/console.md.

SGLang measured, checked and benchmarked as itself

• Fixed: the router read SGLang replicas with vLLM's metric names, and SGLang was started without --enable-metrics, so it served no /metrics at all: every scrape failed as though the replica were unreachable, quietly, and requests were routed to it blind. SGLang is now started with --enable-metrics and read in its own names — sglang:num_running_reqs, sglang:num_queue_reqs, sglang:token_usage (the KV pool's fill), the prompt and generation token counters and sglang:num_requests_total, as 0.5.21 publishes them.
• Fixed: llmgw ctl engine-check sglang asked the interpreter for --help and checked SGLang's flags against Python's. It now asks python -m sglang.launch_server --help, in the environment SGLang was installed into (or --bin).
• Fixed: SGLang was given --tp, which argparse accepts only as an abbreviation of --tp-size; it is given --tp-size.
• Fixed: llmgw ctl bench --metrics read an engine's metrics with the key given for --target — a client's key when the target is the gateway, or none — so llama.cpp and TensorRT-LLM, which guard their metrics with the engine's key, refused every read, and the run reported a queue depth of zero and suggested no thresholds without saying why. The engine key is now asked of the admin API for the replica at --metrics' address (or given with the new --metrics-key); metrics that cannot be read — refused, or in another engine's names than --engine — fail the run before any load, saying why; and a step whose metrics went unread shows - and the reason rather than zeros.

Reusing answers

• A response cache, per project, off until asked for. llmgw keys cache --project P --ttl 1h has the same request — same project, endpoint, model by its catalog name, and body with stream, stream_options, user, metadata, store and service_tier left out and its keys in any order — answered from the gateway's memory instead of the model, for the TTL. --off stops it; without either it says how it stands. The console has it as Cache answers… on a project.
• Only answers that come out the same are reused: chat and completions asked at temperature 0 or with a seed, and embeddings, rerank and score. --any reuses sampled answers too. Only a whole 200 from the model asked for is kept — not an error, a fallback's or a canary's answer, one a guardrail changed or stopped, or one over router.cache.max_entry_bytes.
• Streams both ways: a streamed answer is kept put back together into one completion (tool calls and reasoning included), and a kept answer is streamed as chunks — with the usage chunk if the client asked — to a client that streams. /v1/messages and /v1/responses get it too.
• Headers: x-llmgw-cache: hit|miss|bypass on every answer of a project that reuses answers, Age on a hit. Cache-Control: no-cache or no-store, or x-llmgw-cache: no, asks the model anew.
• Policies still apply: limits and the input guardrails run before the cache is looked in, the output policy judges a reused answer as it would a new one, and a request under no_store or continuing a stored conversation is never served from it nor kept in it. A project's answers are never looked up by another project's keys.
• A hit costs nothing: recorded as a 200 with no tokens and no cost, marked cache_hit on the usage record (migration 0027), and shown by llmgw ctl request; it counts towards the requests-per-minute limit only.
• In memory only: router.cache.max_bytes (128 MB) for all projects together in each gateway process, least recently used dropped first; nothing is written to the store. llmgw_cache_requests_total{model,result}, llmgw_cache_bytes and llmgw_cache_entries.

Offline sites and Kubernetes

• Offline bundles. llmgw bundle create (on any machine with internet) collects what a cluster would otherwise download — llmgw's release for each --arch with its signed checksums, every llama.cpp build for those architectures (or the --variant ones) with the checksums GitHub publishes, the starter model and guardrails' judge, and any --model name/version=hf:owner/repo/file pinned to the registry's commit and its checksum — each checked as it arrives, into a directory or one .tar (--tar), with a manifest whose sha256 it prints to carry apart from the bundle. Run again into the same directory, it downloads only what is missing. llmgw bundle show --verify reads one back.
• llmgw bundle import checks every file against the manifest and llmgw's release against the project's signing key, and only then puts them into the mirror — releases and engines under _llmgw/, models where the mirror keeps models, with their checksums — so a damaged bundle leaves the mirror as it was; a different file under a version already there is refused. On a control plane it also records the engines' checksums in offline.checksums (/etc/llmgw/offline.sha256); llmgw bundle trust does that alone, for a mirror on another machine.
• offline.mirror in server.yaml makes the cluster reach nothing outside: nodes install llama.cpp from the mirror, each file held to the checksum the control plane sends them over the agent's authenticated connection — never one from the mirror — and releases defaults to the mirror's copy, so node add, ctl upgrade and agents upgrading fetch from it; no registry is asked about newer weights. A release may now come over plain HTTP when the build trusts a signing key, since the signature holds whatever carries it. vLLM and SGLang come from PyPI and are said to be unsupported on an offline node rather than attempted.
• llmgw server init --bundle PATH --bundle-sha256 …, and the same two fields on the setup form: installs a control plane offline — the bundle into the mirror, the checksums trusted, offline: written to server.yaml, the starter model and judge served from the bundle or skipped when it does not carry them.
• docs/data.md no longer says only weights and ctl plan reach outside: it lists everything that does, and what an offline cluster changes.
• A Helm chart for the control plane (deploy/helm/llmgw): one replica in a StatefulSet, SQLite on a claim or an external PostgreSQL, the agent port's certificate and the admin token from Secrets, the API and metrics Service, a LoadBalancer for the agents' mutual TLS, an optional Ingress and ServiceMonitor, config laid into server.yaml, and the offline checksums from a ConfigMap; read-only root, non-root, ready on /leader. deploy/container/Dockerfile builds the image it runs. GPU nodes stay outside Kubernetes and join with node add (docs/kubernetes.md says why).

The admin API described, and a Terraform provider

• The admin API in OpenAPI 3.1. Every endpoint under /api/v1, its path and query parameters, and the schema of each request and answer with the meaning of each field, in docs/admin-api.openapi.json and served by every control plane at GET /api/v1/openapi.json (with the token, like the rest). The bearer token, the X-Llmgw-Actor header and the error shape are in it. Generated by go generate ./internal/docgen from a route table in internal/control/adminapi/routes.go and the Go types the handlers use, with the fields' comments as descriptions; a test fails when an endpoint is served and not in the table, another when the committed document is stale. Checked with openapi-spec-validator and openapi-generator-cli's validate.
• docs/admin-api.md: how to reach the API (loopback, an SSH tunnel, admin_remote), the token at /var/lib/llmgw/admin-token, what a request must carry — an address in Host, application/json on every change, even a DELETE — and some calls with curl.
• GET /api/v1/models/{name}: one model's whole definition as it is in force, with the defaults the catalog filled in. The listing has only its state and its profiles' names.
• A Terraform provider, in terraform-provider-llmgw/, a module of its own so the gateway does not carry its dependencies. llmgw_model (a model added at runtime: scaling, class, owner and adapters change in place, the rest replaces it), llmgw_quota, llmgw_guard_attachment, and the llmgw_models and llmgw_nodes data sources. Its client is checked against the OpenAPI document, endpoint by endpoint and field by field, so a change to the admin API that breaks it fails its tests; CI runs them. Built and installed by hand for now: see its README.

Tracing beyond the router

• The control plane traces what it does to the fleet. A reconcile pass that does something is a reconcile trace — passes with nothing to do are not traced at all — with what it did beneath it: scale up with place (the profile, engine, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why, as ctl explain-placement would print it), scale down, drain replica and stop replica each with its reason, evict, preempt with its victims, reap replica with the error and when the model is tried again, restart replica, restart refused, node unreachable and node lost.
• A replica's start is one trace, down to the node. replica start stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event and how long it took. The command to each node is a send start span whose context travels in the command (a new trace field on the agent protocol's ControlMessage), and the agent continues the same trace: start shard, with fetch weights (local, cached, downloaded, or downloaded by another shard; a download with its bytes and where a resume began, the source's credentials and query left out), launch engine (and wait for rendezvous on a worker of a multi-node replica), and wait for engine with how often the engine was asked and its last answer. "Why did qwen take four minutes to start" is that one trace.
• Stops, drains and adoptions on a node are in the control plane's trace too: stop shard and stop engine, saying whether the engine had to be killed; drain shard; and adopt when a registering node reports engines already running, with how many nobody claimed. An engine install is an install engine trace on the node.
• Model jobs are traces. ctl model serve and ctl model update are serve model and update model, with a span per step — registry, mirror, check, catalog, placement, smoke test, replicas — and from the placement step a link to the replica's own start.
• The agent has a tracing block in agent.yaml, the same as server.yaml's (endpoint, sample_ratio; OTEL_EXPORTER_OTLP_ENDPOINT when unset). Point both at one collector. A node follows the control plane's sampling for every start it is sent. Traces from a node carry its id as service.instance.id, and a control plane's its host name.
• With no endpoint nothing changes: every span is the no-op one, and a command carries no trace context.

Quality suites, and new weights held to one

• llmgw ctl eval MODEL --suite FILE|NAME asks a model a quality suite and scores its answers: per case, then the cases weighed into one score, 0 to 1, against the suite's threshold (or --min-score); it exits non-zero below it. --concurrency, --judge, --detach, --json.
  • Suites are YAML or JSON lines: a prompt or a conversation, a weight, and one way to score it — exact, contains, regex, number within a tolerance (the last number in the answer), json_schema (the answer read as JSON, in a code fence or a sentence or not, held to a schema), or judge: a rubric another model on the cluster grades 0 to 1. A reasoning model's <think> is not part of its answer.
  • Run on the control plane, through the router as its own requests, as a replay is: past admission, quotas, accounting and the guardrails. --suite is a file sent whole, or a suite's name in control.eval_dir (/etc/llmgw/eval) or built in: arithmetic and json-format, in deploy/eval/, sized for small models. --suites lists them.
  • Kept in the store (migration 0028): every run with its model, the repository and revision of the weights that answered, the suite's name and hash, its score and each case's answer and reason. llmgw ctl eval --history [MODEL] lists them, --run ID shows one.
• Updates and promotions held to a suite. llmgw ctl model update and model promote take --eval SUITE --min-score N, or a model carries one (llmgw ctl model gate MODEL --eval SUITE, eval: in its definition). A promotion asks the canary first and leaves it in place below the score; an update runs the new weights first as a canary given none of the model's requests, asks it, and promotes it or takes it out again, so the old weights serve until the new have passed. The job's eval step says the score; --skip-eval moves once without the model's suite.
• The console has Ask it a quality suite… (the score counted in as it comes), Its quality runs and Hold its updates to a quality suite… on a model's menu.
• Fixed: a change to the admin API with a body over 64 KiB reached its handler cut off at 64 KiB, by the audit log's reading of it.

MIG instances as GPUs of their own class

• A card in MIG mode is placed as its instances. Discovery reports a card in MIG mode as its MIG instances instead of the whole card, each a device of its own with its UUID (MIG-…), its own memory, and a class named from the card's and the profile: two 3g.40gb instances of an A100 80GB are two devices of class a100-80g-mig-3g.40gb. A profile names that class like any other, and placement puts a replica on a free instance of it. Both discovery paths do it: nvidia-smi (instances from nvidia-smi -L, their memory and ids from nvidia-smi -q -x) and NVML (GetMigMode, GetMaxMigDeviceCount, GetMigDeviceHandleByIndex). llmgw does not create instances; docs/install.md has a MIG section on making them.
• One instance a replica. A process opens only one MIG device, and instances have no NVLink or peer-to-peer between them, so a profile on a MIG class asking for more than one GPU, tp or pp above 1, or more than one node is refused when the model is added, with that reason; placement refuses it again. ctl model plan and ctl model serve plan an instance for a model one holds and never several together.
• Engines are given the instance by UUID. Placement numbers card 0's instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns that into CUDA_VISIBLE_DEVICES=MIG-<uuid> for vLLM and llama.cpp, and leaves GGML_VK_VISIBLE_DEVICES unset (Vulkan cannot see MIG). Every view shows the device as 0/mig2.
• Memory per instance, no made-up utilisation. Each instance's memory, and the processes holding it, are its own — read per GPU and compute instance from nvidia-smi's report, so a foreign process on one instance keeps that instance out of placement and not its neighbours. Temperature, power and wear are the card's. No driver reports utilisation per instance, so none is published for one: the console says n/a on MIG, and llmgw_gpu_utilization_percent has no series for it rather than a zero.
• A card in MIG mode with nothing carved is kept out of placement, with the reason in ctl nodes, since CUDA will not open the whole card while MIG is on.
• llmgw doctor reports MIG: each card's mode and instances, and warns of a card in MIG mode with no instances, of GPU instances with no compute instance, and of a mode change waiting for a reset, with the nvidia-smi command for each.
• ctl nodes counts a node's devices by class when it has MIG instances or more than one class (a100-80g-mig-3g.40gb×2,a100-80g×1); the console's GPU list shows instances under their card with their own memory and the card's heat and power, and a node's page lists its classes with counts. ctl status and the plan count each device under its own class, where a node's cards were all put under its first class before.
• The agent protocol's GPU gains mig (parent card, its memory, profile, GPU and compute instance ids); the node API's GPUs gain label, class, card, mig_profile and memory, and nodes gain gpus_by_class.

Engines and the mirror

• TensorRT-LLM as an engine (engine: trtllm), on nodes where you installed it: name its interpreter in the node's agent.yaml (engines.trtllm.python) and the node offers it while that interpreter has the tensorrt_llm package. llmgw does not install it. The agent starts trtllm-serve serve with the replica's host, port, --tp_size, --pp_size and served name, through a few lines of its own that make it demand the gateway's key on every path but /health (trtllm-serve has no key option), turn on the Prometheus endpoint and iteration statistics, and keep usage reports to NVIDIA off. The router reads it at /prometheus/metrics — trtllm_num_requests_running, trtllm_num_requests_waiting, trtllm_kv_cache_utilization, tokens and successes. Needs TensorRT-LLM 1.3 (rc13 or later) and an Ampere or newer GPU; one node per replica. Profiles may not give it an options file, --trust_remote_code, --grpc, --num_serve_frontends or the flags the agent sets; --set PATH=VALUE changes a setting instead. llmgw ctl engine-check trtllm --bin <python> checks an installed one. Written against TensorRT-LLM's documented command line, server and metrics and tested with stand-ins for them; not yet run on a datacenter GPU.
• llmgw mirror import-ollama [NAME[:TAG]] puts a model you already have in Ollama into the mirror for llama.cpp: its GGUF, and a vision model's projector beside it, each checked against the sha256 Ollama named its file by. With no name it lists Ollama's models; it finds Ollama's directory itself (OLLAMA_MODELS, ~/.ollama/models, Ollama's Linux service) or takes --ollama-dir. The version is the tag and the start of the digest, since a tag moves in Ollama. It prints a model definition to add, with Ollama's num_ctx as --ctx-size. The console's Mirror view does the same with I, for the mirror beside the control plane.
• llmgw mirror import FILE... --model NAME copies files already on the mirror machine into it the way a download is kept: whole or not at all, the checksum beside it, a version never replaced, a .gguf checked to be one, and --sha256 checked when given.

The portal in Turkish

• The web portal speaks English and Turkish — the landing page, sign-up and sign-in, the account page and its editor setup, the claim page, chat, the about page, and every administrators' page (cluster, nodes, NVLink and fabrics, people, limits, usage, organizations, mirror, guardrails), and the docs portal around its documents, which stay English. Flash messages and form errors built by the server are translated too.
• A switcher at the foot of every page (English · Türkçe) remembers the choice in a cookie; without one the browser's Accept-Language decides, and without that the new landing.language (en or tr, empty is English). An unknown code stops the server starting. <html lang> and Content-Language follow the page.
• Figures and dates in the page's language: a Turkish page writes 1.234,5, 04.10.2026 and "3 saat önce"; chart tooltips name the day in Turkish. Form values, CSS and SVG coordinates are unchanged.
• Ready for more languages: the catalogue is JSON under internal/web/locales/<code>/, and a language is a directory. Tests refuse a key a page uses that some language lacks, a translation with different arguments from English, and any template text left in English — so a new page cannot quietly ship English-only.
• Messages that come from the control plane and other packages (node warnings, engine errors, store refusals) are still shown as written, in English; the API, CLI and console are English.

Documentation

• A docs portal at /docs. The manual the binary carries — the pages llmgw docs prints and the console's Manual shows, the configuration reference among them — rendered on the client port in the portal's own look: a sidebar of the documents in reading order, the open one's sections under it, a link on every heading, and links between documents kept inside the portal (links to files that are not in the binary, such as deploy/, go to the repository). No script on the page, so the Content-Security-Policy has nothing to allow. Public by default, like the landing page; set landing.docs_sign_in: true to send visitors to sign in first. Signed in, the page carries the usual menu, which now has a Docs link.
• A client developer's guide, docs/clients.md. How to use llmgw from the official OpenAI and Anthropic SDKs — there is no llmgw SDK, and none is needed — in Python, TypeScript, Go and curl: the base URL and key, streaming, tools, embeddings, the Responses API, batches, the Anthropic SDK against /v1/messages, and what llmgw adds as headers (x-llmgw-thread, x-session-id, metadata and x-llmgw-tags, x-llmgw-variant, /v1/memory, /mcp), with the statuses a client meets. The portal's index leads with it, and the landing page and the account page link to it.
• The console's Manual and the portal read one list of documents in one order (llmgw.Manual), so the two cannot drift.

Administration

• An audit log. Every change made through the admin API (llmgw ctl, the console) or the pages under /admin is recorded: who, when, through what, the fields it was given with passwords, keys and tokens taken out, and how it ended — refusals included, in words. So are a person's own new keys, revoked keys and password changes. Read it with llmgw ctl audit (--since, --actor, --contains, --detail), GET /api/v1/audit, or the console's Audit view. It moves with the store.
• Staff roles besides administrator: viewer looks at every administration page and changes nothing, operator also runs the cluster from them, billing sees usage and quotas. Set with llmgw users role or the console's Staff role….
• Single sign-on through OpenID Connect (auth.oidc). A "Sign in with …" button sends people to the identity provider and back, with PKCE, state and nonce. The ID token is verified, and the account is found by the verified email address, or made when create_accounts is on. Groups give staff roles (roles, set at every sign-in) and places in the tree (memberships). no_passwords leaves the provider as the only way in, except for administrators.
• Provisioning through SCIM 2.0 (auth.scim.token_file, /scim/v2).
  • Users: the identity provider makes, suspends and reinstates accounts — the PATCH forms Entra ID and Okta send are both read. Deleting a user suspends the account.
  • Groups: joining and leaving the groups auth.oidc maps gives and takes away staff roles and places in the tree.
• Key lifecycle.
  • Lifetime limits: auth.max_key_ttl limits how long any key may live, and llmgw keys ttl team:Acme/ml 720h sets shorter limits for an organization, team or project. A key is made within the shortest that applies, and one asked for with no expiry is given that longest — from the CLI, the profile page and the editor connect page alike.
  • Rotation: llmgw keys rotate <id> --overlap 48h issues a replacement where the old key was, and the old one stops after the overlap.
  • Listing: keys list --all shows each key's kind — service (a project's), member, personal — with service keys first, and when it expires.
• Usage export. llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model writes CSV by hour, day or month and any of organization, team, project, user, key and model.
  • Data: read from the hourly and daily sums, so it reaches back past the request records' retention.
  • Format: cost is given in micro-units and units, and names that look like a spreadsheet formula are quoted.
  • Other ways in: GET /api/v1/usage/export (CSV, or format=json) and E on the console's Usage view.

Forgetting a person, and requests that stay in the cluster

• llmgw users forget erases a person for a request to be forgotten: their usage kept under a random pseudonym so totals stay true, their kept requests, personal files, batches and limits deleted, their name taken out of the audit log, the account removed.
• llmgw keys residency --project P --in-cluster: no model served by an outside provider for the project's keys, asked for, as a fallback or as a canary.
• Both are on the console's menus.

Secrets in Vault

• Secrets from Vault or the environment. Every setting naming a secret's file also takes env:VARIABLE or vault:mount/path#field (HashiCorp Vault or OpenBao, KV v1 or v2; a token, AppRole, or VAULT_TOKEN).
• The store's own secrets sealed under secrets.master_key: Vault's transit engine (vault-transit:KEY), or a 32-byte key read from a reference. A store, or a backup of it, then does not hold what engine keys are derived from. Values from before are sealed as they are next read.

Scaling by the clock

• scaling.schedule: windows of the week with their own min and max — more replicas through working hours, an on-demand model kept loaded while people are at work — in control.timezone.
• scaling.prewarm opens a window early, so its replicas have loaded when it starts.

Token-level timing, and admission by SLO

• Per-token metrics: llmgw_tpot_seconds, llmgw_inter_token_seconds and llmgw_output_tokens_per_second by model and pool, and TPOT and tokens a second by tenant.
• A model's slo counts in admission: past its ttft_p99 or tpot_p99 (the recent p99), it is treated as saturated, so the lower bands wait. llmgw_model_slo_seconds and the llmgw-slo alerts make the promise visible in Prometheus.

Restoring a backup, and copies off the machine

• llmgw server restore ARCHIVE puts a backup back: checks it first (whole, opens with --identity, not newer than this llmgw, where the store goes), says so with --dry-run, then moves /etc/llmgw aside, restores it with modes and owners, restores the store (SQLite, or PostgreSQL with pg_restore --clean), and starts the server. A drill is in docs/install.md.
• backup_copy_to copies each archive to a directory or over ssh as it is written, keeping as many there as here; refused without backup_recipients.
• Archives now record each file's owner by name, so a restore on another machine gives the llmgw user its files back.

Anthropic's Messages API

• /v1/messages (and /v1/messages/count_tokens) translated to chat completions and back, streamed or not: text, images, tools and their results, the system prompt, stop sequences. Claude Code and Anthropic's SDKs reach the models here; the key may come in x-api-key. Errors come back in Anthropic's shape.

More of OpenAI's API

• Responses: text.format asks for structured output, input_image sends images, and previous_response_id continues a conversation the gateway keeps as a thread (for a project that keeps conversations).
• Passed through: /v1/rerank and /v1/score for vLLM's rerankers, and /v1/audio/transcriptions and /translations as the multipart form came, for an engine serving Whisper.
• /v1/moderations is answered by the guardrails' judge model.

Requests kept, and replayed

• A project can keep its requests and answers (llmgw keys requests --project P --retain-hours N [--sample 0.05], or Keep requests… in the console): what the model was sent, after redaction, and what the client got. Off until asked; never under no_store; pruned hourly.
• llmgw ctl request ID --body shows one; --replay MODEL sends it to another model and shows both answers.

Batches

• OpenAI's Batch and Files APIs. POST /v1/files (purpose batch), POST /v1/batches, and reading, listing, cancelling and deleting them, so OpenAI's clients send batches here unchanged.
  • Serving: each line goes as a request of the key that made the batch, in the lowest admission band, a few at a time; rate limits are waited out. Answers and failures are written to files kept 30 days.
  • Accounting: each line is an ordinary usage record tagged batch=<id>.
  • Restarts: answered lines are kept as they are answered, and any gateway on the store carries a batch on under a lease.
  • Limits: router.batch bounds the file, the lines and the concurrency. Refused to a project under a no_store policy.

New weights tried on a share of the traffic first

• Canaries. llmgw ctl model canary MODEL --percent 10 fetches the newer weights of a model's repository and runs them beside it as MODEL-canary, given that share of its requests.
  • Routing: a session stays on one side; x-llmgw-variant pins a request to canary or stable and says which served it. Clients cannot name the canary.
  • Rolled back by itself when, over 50 of its requests or more, its error rate is five points above the model's or its time to the first token half again as long; canary_rolled_back is sent to the notification webhooks.
  • Then: --percent changes its share, llmgw ctl model promote moves the model to it, --abort takes it out, and llmgw ctl model canaries compares the two sides. All of it is also on the console's model menu.

Requests can be tagged

• Tags and end users on every usage record. OpenAI's metadata and the x-llmgw-tags header (feature=search, env=prod) tag a request; OpenAI's user field names the client's own end user. Both are kept with the request's usage and shown by llmgw ctl request. Tags are not sent to the engine. A malformed one is refused with 400.
• Counted by tag, for the tags named. router.tag_labels lists the tags counted in llmgw_tagged_requests_total and llmgw_tagged_tokens_total, the first 100 values of each apart and the rest as other.

A hardened install by default

• server init and the setup screens write a hardened configuration.
  • Guardrails: a baseline policy on every request. It refuses secrets in prompts and answers, masks card numbers and records injection attempts.
  • The judge: Llama Guard 3 1B, two replicas, fetched and run on CPUs (about half a second for a short text). It refuses violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content. The policy fails closed: a request it cannot read within 5 seconds is refused (503), and judge_unavailable goes to the notification webhooks.
  • Keys: they live at most 90 days.
  • Opting out: --no-guard-model (or unticking it on the setup screen) leaves the judge out and keeps the pattern checks. A server.yaml written earlier is never changed.
  • What is left to you: the closing text now names the two things init cannot do — a certificate for the client port, and the keys backups are encrypted to.

Notifications

• Webhooks for what needs a person (notify.webhooks): a model parked, a node lost, a budget spent, new weights in a model's repository — sent by the gateway itself, with no Prometheus in between.
  • Formats: JSON signed with HMAC-SHA256 (X-Llmgw-Signature) when secret_file is set, or a Slack-compatible {"text": …}.
  • Delivery: queued, never holding up what raised it; tried three times; the same event about the same thing at most once an hour, and new weights once per revision.
  • Checking: llmgw ctl notify test.
  • A judge not answering (judge_unavailable) is one of the events.
• Alertmanager receiver templates for Slack, e-mail, PagerDuty and a webhook in deploy/monitoring/alertmanager.yml, commented out until chosen.

Guardrails

• Hooks for guardrails on the request path (nothing judges yet; B2 adds the policies). A guard, when one is set, judges a request once it is authorised, within its limits and has its stored conversation spliced in, and before it is dispatched; and filters the answer, buffered or streamed, before the client sees it. A blocked request is refused with 400 and code: "content_policy_violation" — the first error with a code — and costs nothing; a blocked answer ends with finish_reason: "content_filter", and a stopped stream with that chunk and [DONE], the engine cancelled. The Responses API reports either as incomplete with incomplete_details.reason: "content_filter". x-llmgw-guard and x-llmgw-guard-policy say what was done; the usage row records the action, policy, category and time taken — never the text.
• Guardrail policies (guard: in server.yaml; see docs/guardrails.md).
  • What is found: personal data, each kind validated before it counts (email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits, IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt injection seen through disguises (look-alike letters, invisible characters, base64/hex/URL encoding); deny patterns.
  • What is done: flag, redact as [EMAIL_1], or block. A request can be shaped too: a pinned system prompt, the caller's own system messages dropped, an output limit, tools stripped or refused.
  • Answers: judged buffered, or a sentence at a time when streamed, never releasing half a value; tool calls are checked against allow lists, a count and the request's own schemas.
  • Modes: shadow records what it would do; no_store refuses conversations, notes and models outside the cluster.
  • Where they apply: guard.default everywhere; others attached with llmgw ctl guard attach to an organization, team, project, user or key, and narrowed like limits.
• A judge model on the cluster (role: guard in the catalog, judge: in a guard policy). It reads what patterns cannot — Llama Guard 3, ShieldGemma or a JSON-answering model — and its verdict flags or blocks input, output or both.
  • Calls: past admission and accounting, bounded by a latency budget (250ms by default), after which the policy's fail decides.
  • Caching and streams: answers are cached for five minutes, and a streamed answer is read every 600 characters.
  • Isolation: a judge model is hidden from clients and refused to them. It is critical by default, and cannot be loaded on demand.
• Seeing what the guardrails do.
  • Metrics: llmgw_guard_checks_total (stage, policy, action, category, mode), latency for checks and for the judge, judge errors, fail-open, redactions by kind, stopped streams, and blocks per tenant.
  • Alerts: judged traffic served unjudged, a slow or unreachable judge, a tenant's blocks spiking, a secret in an answer.
  • Pages and commands: /admin/guard shows counts by place, policy and category and the recent requests by id; ctl request shows the verdict; the profile page lists the policies in force where the person works; the console's organization rows have Guardrails… and a column.
  • The chat page says when a request was refused or an answer stopped by policy, instead of showing an error.

Security

• Built with Go 1.26.6. Eleven standard-library vulnerabilities reachable from this code at 1.26.3 (net/http, crypto/x509, os.Root, net/textproto) are fixed by it. make vuln runs govulncheck; make check runs it with the tests and vet.
• Releases are signed, and only signed releases are installed. The build now carries the release signing key (internal/release/trusted.pub); the control plane upgrading itself, an agent told to upgrade and a node being added all refuse a release whose checksums are not signed with it. The CI release job stops rather than publish an unsigned release. Adding a node no longer has the node fetch the release and compare it with a checksum from the same server: the control plane fetches it for the node's architecture, checks the signature and the checksum, and hands it over. Releases are fetched over https only (loopback excepted), and LLMGW_RELEASE_KEY replaces the trusted key in development builds only.
• The agent port is not opened without TLS by accident. A control plane with no agent_tls refused nothing: its agent port was plaintext, on every interface, and any machine reaching it could register as any node — be given replicas and so be sent prompts. It now refuses to start unless agent_insecure: true is set, and then listens on loopback unless agent_listen names an address. server init writes agent_tls, as before.
• A model definition cannot run code on its nodes. A profile's env and args reach a process the agent runs as root. The environment was checked against a short denylist that missed NCCL's and CUDA's plugin and injection libraries; arguments were not checked at all, so --trust-remote-code, llama.cpp's --log-file or --slot-save-path, or vLLM's --tool-parser-plugin were taken. env is now an allowlist of the families engines are tuned with (CUDA_, NCCL_, VLLM_, OMP_, HF_…) less any name for a path, plugin, library, address or credential, and llama.cpp's LLAMA_ARG_* are checked as the flags they are; args refuse what reads, writes or loads files, loads code, or moves what the agent sets (host, port, key, model). Both are refused where a model is defined and again on the node before the engine starts.
• Engines no longer run as root. The agent starts them as the system user llmgw-engine (in the video and render groups for the GPUs), with a home and caches of its own under the agent's work directory. llmgw node add makes the user; a node without it keeps running engines as root and the agent says so in its log. supervisor.run_as in agent.yaml names another user, or root.
• The admin API wants a token. It listened on loopback with no authentication, so any process or user on the control plane could add a model, upgrade the fleet, read the engines' keys or dump the heap through pprof. Its /api and /debug paths now demand the token the server writes, 0600, to its state directory (/var/lib/llmgw/admin-token) on first start; llmgw ctl, the console and node add read it, so they run with sudo, and LLMGW_ADMIN_TOKEN or ctl --token-file give it from elsewhere. pprof is off unless debug_pprof: true. admin_listen off loopback is refused unless admin_remote: true. The read-only dashboard pages, /metrics and /health are unchanged.
• The replicated store asks for a password and speaks TLS. Patroni's REST API took changes from anyone who reached port 8008 — rewriting the access rules to trust, or moving the leader — and PostgreSQL traffic between the machines was plaintext (sslmode=disable). Patroni now asks for a password before a change (kept with the cluster's others in /root/llmgw-pg, and on the control plane in /etc/llmgw/secrets/patroni-rest for server add); each member has a certificate from the cluster's authority, PostgreSQL has ssl on, every remote access rule is hostssl, and the control planes connect with sslmode=verify-full against the authority's certificate. A new cluster is set up this way; an existing one is changed in place, by reload, the cluster serving throughout, by running Replicate the store in the console again — or sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni <user>@<a> --patroni <user>@<b> --no-mirror --no-node --no-model — on the control plane that holds the authority, then on each other control plane so it reconnects by TLS. etcd is not yet given TLS: keep ports 2379 and 2380 to the cluster's machines.
• A capacity fallback respects the key's allowed models, and is charged at its own price. When the model asked for had no capacity, the next in its fallbacks served the request even if the key was not allowed it, and the request was priced as the model asked for. Fallbacks a key may not use are now skipped, and a request is charged what the model that answered costs.
• Suspending an account stops its keys. Suspension closed the person's sessions and left every API key they had made working. A key whose owner's account is not active — suspended, pending, or gone — is now refused as a revoked key is, within auth.cache_ttl, and works again when the account is reinstated.
• A join code is a credential only for its hour, and only on its address. The setup checked a code's expiry on the joining side alone, so a code stayed good for as long as the setup ran; it now refuses an expired one itself. Its listener binds the address the codes name rather than every interface, and a joined machine is known by the address it connected from, not the one it claimed.
• The setup's sudo grant cannot outlive it. Given a sudo password, the setup writes a temporary NOPASSWD grant and removes it when it finishes; a setup that crashed or was interrupted left it for good. The grant now schedules its own removal within the hour as it is written, and the password is dropped once the grant stands in for it. An address given to the system ssh can no longer be read as one of its options.
• Made-up keys cannot fill memory or the database. Every well-formed key presented, real or not, was cached for good and cost a database read the first time. The cache is now bounded, and keys not in it are looked up at most 20 a second per client address (50 at once); beyond that the request is answered 429 without a read.
• Slow or idle connections cannot pile up. The servers bounded only how long headers took. Every listener now also bounds how long a connection may sit idle and how large its headers may be, and reading a request (5 minutes on the client port, a minute or less elsewhere — never on the agents' long-lived streams); the client port holds at most max_connections (10,000) at once, and a message from an agent at most 16 MB.
• The sign-in and signup forms resist scripts. Ten wrong passwords from anywhere locked an address's owner out for the window; attempts are now counted per address and client, so somebody guessing elsewhere cannot lock the owner out, with a looser count per address alone so guesses spread over many clients are still bounded. The sign-in form carries a token its cookie matches, so another site cannot sign a visitor into an account of its choosing. Posting either form is rationed per client address (sign-in 30 a minute, signup 5), and a password hash that cannot get a turn within ten seconds answers "busy" instead of holding the request. server init now writes signup_open: false.
• Forwarding headers are believed only from the proxies named. X-Forwarded-Host and -Proto were taken from anyone, and decide the gateway address written into the /setup scripts clients pipe to a shell; behind a caching proxy one request could poison them for others. They — and X-Forwarded-For, now used for the per-address limits — are believed only from router.trusted_proxies (addresses or CIDR prefixes). Behind a reverse proxy, set it, or the pages and scripts name the address the proxy reached and every client counts as the proxy. /setup scripts are served no-store.
• Removing a node revokes its certificate. ctl node remove only dropped the node from the inventory; its certificate, valid for a year, still connected and was given work. Now the removal is recorded in the store and the agent port refuses any certificate issued for that node before it — onboarding the node again issues one that works. A mirror serving its management API with --tls-ca now requires --api-client (the control plane's name), since the same authority signs every node.
• Engines listen on the management address only, the one the control plane reaches them on, instead of every address the node has — a public one included. (When that address is a name, or not one of the node's, they still listen on all.) SGLang's key is off its command line, where every user on the node could read it: it reaches the engine in its environment, and a short launcher hands it to SGLang inside the process. Both take effect as engines restart under an upgraded agent.
• Backups can be encrypted. backup_recipients in server.yaml (age recipients or SSH public keys) has llmgw server backup encrypt each archive to them with age, as .tar.gz.age: an archive holds the CA's key and every secret in the store, and is meant to be copied off the machine. Open one with age -d -i <key>.
• PyTorch's index gives torch and nothing else. vLLM's CUDA 12.9 build was resolved with PyTorch's index beside PyPI and unsafe-best-match, so either index could supply any package — how torchtriton was hijacked in 2022. Now that index is explicit, named only for torch, torchvision and torchaudio; every other package comes from PyPI alone, and the lock is PEP 751's (pylock.llmgw.toml), pinning each file's address as well as its hash. Checked against vLLM 0.30.0: the same 195 packages resolve.
• The control plane's own fetches stay off link-local addresses. Checking a model's source and reading a GGUF's header went wherever the URL said, cloud instance metadata (169.254.169.254) included. They now refuse link-local, multicast and unspecified addresses, while still reaching the cluster's network, where the mirror is. The mirror's fetches, which refuse private addresses too, now also see through IPv6 addresses carrying an IPv4 one (NAT64, 6to4, IPv4-compatible), and through a proxy: the target's addresses are checked before the request is handed to it.
• A CPU-only node reporting a problem no longer panics the agent handler, and thread ids are bounded to 128 letters, digits and - _ . :. A thread is readable by every key of its project that knows the id; the docs now say to use unguessable ids.
• /v1/models lists only what the key may use, rather than every model the gateway serves to anyone.
• Portal sessions end after a day unused (still at most 14 days in all), and expired or idle sessions and used or expired key claims are deleted hourly — they were refused, but never removed.
• Every page carries a Content-Security-Policy: scripts only from the gateway itself and the page's own inline ones, each named by its hash — no inline handlers, no eval, nothing a model's reply or a model card could slip into a page. The few onchange/onsubmit attributes moved to /_static/pages.js, htmx runs with allowEval: false, and the live pages' "only while visible" refresh no longer needs an evaluated filter. Pages now carry their headers even when answered with an error status.
• KaTeX 0.16.47 (from 0.16.11), past the advisories fixed in 0.16.21.
• Smaller hardening. The mirror no longer serves its bookkeeping files (a file's .source URL and .sha256) on :8081. A GGUF header nesting arrays more than 4 deep is refused rather than recursed into. Serve and update jobs keep the last 100 finished, and the mirror forgets finished downloads after a week, instead of remembering every one until restart.
• A firewall for llmgw's internal ports. server init (and the store members it sets up with --patroni), node add, server add and mirror init now keep the machine's internal ports — the agents' stream, engines, the mirror, PostgreSQL, Patroni, etcd, node exporter — to the cluster's own networks, in an nftables table of llmgw's own loaded by llmgw-firewall.service. Only those ports are touched: the table accepts everything else, so other services on the machine and any firewall it already has are left as they were. Extra networks go in firewall_networks; --no-firewall skips it; llmgw doctor says when a node lacks it.
• etcd and Patroni's API speak TLS. etcd's peers and clients use certificates from the cluster's authority, and a client must present one; Patroni reaches it with its member's certificate and serves its API over TLS. server init --patroni, run again, moves an existing cluster: with Patroni in maintenance mode, so PostgreSQL serves on and nothing fails over, etcd is rebuilt on TLS and Patroni's keys carried into it. The control plane asks the API over HTTPS, verified, and over HTTP only where a member has not moved yet.
• A cluster keeping its access rules per member got no hostssl. The TLS step looked only at the shared configuration, and on a cluster set up before the rules moved there said the rules were hostssl while plaintext was still accepted. Each member's own rules are now made hostssl too.

Fixed

• An engine that died after an agent restart went unnoticed. The watcher that reports a ready engine dying was tied to the stream it was started on, so after a reconnect nothing reported it, and an engine adopted from the previous agent had no watcher at all: its replica stayed "ready" and was routed to. Every connection now watches every ready shard, started or adopted. Found on the PoC: a killed engine on node06 stayed ready for minutes.
• server init rewriting the store's address no longer leaves /etc/llmgw/secrets unreadable to the service when it was made 0700 for the service user, which kept the gateway from starting.

Everything that can be changed on a running cluster, from the console

sudo llmgw only watched the cluster and edited the configuration files; changing a model's replicas still meant llmgw ctl. Now Enter on a row opens what can be done to it, as forms and questions:

• Models: replicas (min, max, idle timeout), class, owner, resources, LoRA adapters, restart, retry when parked, why it is placed where it is, remove; A adds one from a definition file.
• Nodes: cordon, uncordon, drain, the vLLM and SGLang builds, upgrade its agent, delete model files nothing uses, remove.
• Replicas: restart, replaced or in place.
• Quotas (new): capacity quotas, set and removed.
• Mirror (new): what it holds and which models use it; download into it, remove from it.
• People (new): accounts — approve, suspend, reinstate, the cluster dashboard, memberships, limits, a new password, delete, create; API keys — issue (shown once), limits, revoke; organizations, teams and projects — add, rename, move, remove, limits for the level and for each member, capacity quotas, how long conversations and agent notes are kept.
• Serve a model (new view): GGUF models on Hugging Face — trending, most downloaded, most liked or newest, or searched with / — then one of the chosen repository's files, a name and a replica count, and it is fetched, checked, added and started. The registry is mirror.registry's when set.
• Do: upgrade the cluster, restart every replica, back up now.

The people side runs the same commands as llmgw users, keys and org, against the same store, so their checks and messages are the ones shown. A sidebar taller than the terminal now scrolls with the current entry.

New weights in a model's repository, found and moved to

A repository on Hugging Face can publish new weights under the same file names — a fixed chat template, a better quantisation — and nothing said so. A model served from the registry now records its origin: the repository, the commit, and each file's checksum there; a model served before is worked out from the mirror's record of where it fetched each file. Every mirror.update_check (six hours by default; 0 turns it off) the control plane asks whether those files changed. A new commit that leaves them alone is not an update.

• llmgw ctl model updates [--check] lists what each repository has.
• llmgw ctl model update MODEL fetches the new files into the mirror beside the old ones, checks them against the registry's checksums and the nodes' llama.cpp, points the catalog at them, replaces the replicas one at a time and asks one to say hello.
• llmgw ctl model rollback MODEL goes back to the files the update replaced, which stay on the mirror.
• The console shows UPDATE on Models and offers all three on a model's row.

A model from server.yaml is reported but not changed: its source is edited there.

The monitor: the cluster drawn like btop

The console has a Monitor under Watch: CPU, memory, GPU busy and memory, requests, tokens, time to first token, in flight and queued, network and disk, as Braille charts filled and coloured by height, over 5 minutes, an hour, a day or a week (1–4 or R); then each node's CPU, memory and network, each card's busy, memory, heat and power, and each model's requests, as sparklines. A node's and a model's own pages carry their charts too.

The history is read from the Prometheus in monitoring.prometheus, through the admin API (POST /api/v1/metrics/range); without one, the charts draw what the console has seen since it opened.

The control plane now lists the cluster's machines for Prometheus' service discovery, at /prometheus/targets on its metrics port — nodes, itself and the store's members, each labelled with its llmgw id — and llmgw node add installs prometheus-node-exporter on the machine (--no-node-exporter leaves it off). deploy/monitoring/prometheus.yml scrapes them as the nodes job.

Serving and updating run in the background

Serving a model or moving one to new weights from the console starts a job on the control plane and leaves the console free at once, instead of closing it to follow a download that can take an hour. A new Jobs view shows each job's progress and steps, and the sidebar how many are running. llmgw ctl model serve --detach and llmgw ctl model update --detach do the same; llmgw ctl jobs lists the jobs and llmgw ctl jobs ID follows one.

The file that suits the hardware, and how to spread it

llmgw ctl model plan REPO now plans the files the repository actually has against the fleet as it is, before anything is downloaded: the model's shape comes from one GGUF file's header, read from the registry, or from its config.json. Every file is listed with the best way it runs here — one card; several of a node's cards (llama.cpp splits by layer, vLLM by tensor parallelism, dividing the KV heads); several nodes' cards over one fabric, pipeline-parallel between them (vLLM); or a CPU node's memory — what each card or node would hold and whether that hardware is free now. Best first: on a GPU, the fewest cards, then the better quantisation up to eight bits; whether the cards are free now only breaks a tie. Busy is not unfit: a model for cards that are in use is added and starts when there is room, the job saying so rather than failing. --precision keeps the old plan in the abstract.

llmgw ctl model serve without --quant takes the plan's file, and without --cpu or --gpu-class/--gpus/--nodes the plan's way of running it — a GGUF model on GPUs is now given a GPU profile rather than always the CPU. --fallbacks writes the next best ways too, lower in priority, so placement uses one of them when the first is busy. The console's Serve a model shows the plan for each file, marks the best, and asks for the way and the fallback.

Models of several files: vision projectors, original weights

• A vision model's projector (mmproj) is fetched beside its GGUF weights and loaded with them: llmgw ctl model serve REPO --projector FILE, or the choice offered by the console. The model is marked as taking images. Nodes load it once their agent is this version or later.
• --format safetensors serves a repository's original weights, with their config.json, tokenizer and shard index, on vLLM across --tp GPUs of --gpu-class (the fleet's one class when there is one). In the console, F on Serve a model switches to them.
• The mirror keeps where it fetched each file beside it, and reports that and the checksum after it restarts. The console's Mirror view shows one row per version of a model, with its files on Enter.
• An update watches every file of a model, the projector and the tokenizer included; one whose files the mirror cannot trace is found on the registry by name, commit and size, or given with llmgw ctl model origin.
• llmgw_model_update_available is published, and ModelUpdateAvailable alerts on it.

The manual, in the binary

llmgw docs reads every document — installation, operation, the configuration reference, the README, the changes — and a page for every command, built from the commands' own help, like info: topics on the left, T for a page's contents, / to find, n for the next match. Piped, it prints the page. In the console it is Help › Manual, and ? opens the new page about the console itself from any screen.

v0.11.1 — 2026-10-03

The console's overview no longer spills over the screen

On a cluster with a GPU, sudo llmgw printed the overview's GPU table onto the terminal under the console every three seconds, scrolling across the full screen. The summary's tables were written to standard output whatever they were given; they are now written where the summary is, which for the console is its panel. The test cluster the console was tried on had no GPU, which is why it was not seen there.

v0.11.0 — 2026-10-03

A minor version: the store can be replicated and the control plane doubled, both set up by llmgw itself; sudo llmgw opens setup screens and a console; usage is read from sums kept as it is written. Released without candidates.

Upgrading. Migrations 0015 (usage sums, filled from the request table already recorded — once, at the first start) and 0016 (control planes) run by themselves. An agent older than this ignores the list of control planes and keeps its configured one. Nothing in server.yaml or agent.yaml changes meaning; auth.usage_retention and router.tenant_label are new and optional.

The GPU and fabric work below began from the field report in issue #2, after a deployment on GPU nodes.

GPU metrics that move, and charts of them

A GPU's utilisation and memory in use were read once, when the agent started, and published from then on: a card busy since looked as idle as it was at registration, and one idle since as busy. They are now sampled with the temperatures and power every heartbeat (ten seconds at most), with the memory bandwidth in use, the SM clock and uncorrected ECC errors besides. An agent too old to send them still has its registration figures published, and a figure the driver does not give is left out rather than shown as zero.

• New series: llmgw_gpu_memory_utilization_percent, llmgw_gpu_sm_clock_mhz; llmgw_gpu_utilization_percent, llmgw_gpu_memory_used_bytes and llmgw_gpu_ecc_errors now from the sample.
• The node page shows each card's utilisation, memory, clock and ECC errors in its table, and charts of utilisation, memory, temperature and power, one line per card.
• The cluster page charts the fleet's GPUs by class: average utilisation and memory in use.
• Grafana (deploy/monitoring/grafana/llmgw-overview.json) has a GPU row: utilisation, memory, memory bandwidth, temperature, power against the limit, clock, and what holds cards back.
• Alerts: GPUThermalThrottling (ten minutes slowed by heat), GPUUncorrectedECCErrors, GPUHot (above 87 °C for ten minutes).
• The cluster page's GPU map: every card a square, a row per node, coloured by how busy, how full or how hot it is (?gpumap=util|memory|temp), with what holds it and its figures under the pointer; free cards dashed.
• Cards held and doing nothing: the replicas whose every card reads under 5% busy, with when they were last asked — the first place to find room.
• What each model gets from its cards, on the cluster page and in Grafana: output tokens a second per GPU and per watt-hour, with the cards' draw and average utilisation, over the last quarter hour.
• What wears a card out, read every minute: XID errors from the kernel's log (counted from when the agent started), memory rows remapped or pages retired and whether either waits for a reset, a card out of spare rows, PCIe link retries and NVLink errors. Published as llmgw_gpu_xid_errors, llmgw_gpu_last_xid, llmgw_gpu_remapped_rows, llmgw_gpu_remap_pending, llmgw_gpu_remap_failure, llmgw_gpu_retired_pages, llmgw_gpu_retirement_pending, llmgw_gpu_pcie_replays, llmgw_gpu_nvlink_errors; said in words on the node page; a red outline on the GPU map; a Grafana panel; and alerts — GPUXidError, GPUOutOfSpareRows, GPUNeedsReset, GPUPCIeRetriesClimbing, GPUNVLinkErrorsClimbing.
• GPU time by owner on the usage page: the cards each owner's models held over the window, charted and summed into GPU-hours, whether or not the models were asked anything — the time a GPU quota is about. From the metrics store, as far back as it keeps history.
• llmgw_gpu_assigned{node,gpu,model,replica,owner}: which replica holds each card and whose quota it counts against, for joining the GPU series to models and tenants.

Usage is read from sums kept as it is written

Every chart and total — the usage tab, quotas, the dashboard, a person's own page, and budgets restored at start — summed the request table again on each view: a row per request, scanned whole. Usage is now also summed as it is written, in the same transaction, into five-minute slices, hours and days per key, person, project and model (usage_5m, usage_1h, usage_1d; migration 0015 fills them from what is already recorded).

• Charts read the coarsest of the three their slices are made of: a day in 20-minute slices from the five-minute table, a week or a month from hours.
• Totals and budgets read the rows up to the next five-minute slice, then slices, hours and days — exact, not rounded to a slice. On three million requests over thirty days, a month's total per person went from 705 ms to 18 ms on PostgreSQL.
• Latency and time to first token are kept as histograms in the sums, so a percentile per team or model can be read after the rows are gone.
• auth.usage_retention (e.g. 2160h) deletes request records older than that, hourly. Totals, charts and budgets are unchanged by it — they read the sums — but ctl request finds nothing older. Unset keeps every record, as before.

Alerts per tenant and per budget

• llmgw_tenant_requests_total{tenant,outcome}, llmgw_tenant_tokens_total and llmgw_tenant_ttft_seconds: requests by outcome (ok, error, limited), tokens and time to first token per tenant. The tenant is the level router.tenant_label names — team by default, or org, project, or none — named by its path, "acme/platform". One level rather than every project, so a cluster of thousands of projects does not become a Prometheus of millions of series.
• llmgw_budget_spent_ratio{scope,id,name,period}: this period's spend over the budget for every organization, team, project and person given one, refreshed every minute.
• Alerts in deploy/monitoring/llmgw-rules.yml: BudgetMostlySpent (80%), BudgetSpent, TenantHittingLimits (a tenth of requests refused), TenantErrors, TenantTrafficSpike (three times the weekly average) and TenantSlowFirstToken.
• A Grafana dashboard per tenant, deploy/monitoring/grafana/llmgw-tenants.json (uid llmgw-tenants): requests and tokens by tenant, the share failing and refused by a limit, first-token p95, every budget's share spent, and who used most in the last day — with a tenant chooser at the top.

How each team's requests fared, not only what they cost

• Percentiles from the sums. First token and latency, p50, p95 and p99, per organization, team, project, person or model, read from the histograms kept with the usage sums: on the usage tab's table, and in a branch's or person's panel on the people page, beside the error rate and requests refused by a limit.
• Refused requests are counted against their owner. A request a rate, token or budget limit refused was counted only in Prometheus, by model; a team running into its limit looked like a team that had gone quiet. It is now recorded as a 429 against its key, person and project — no tokens, no cost — and shown as "Limited".
• Usage by model is a grouping of the usage tab, by the model that answered.

The console watches the cluster and holds every setting

sudo llmgw on a control plane is a console in three groups down the left:

• Watch, refreshed every three seconds: the overview; nodes, with CPU, memory, GPUs free, disk and agent version, and a node's detail — load, engines, each card's busy, memory, temperature and power; every GPU in the fleet and which model holds it; models, with a model's profiles, warnings and replicas; replicas and where they run; the last day's use by team, project or organization; and a node's agent log.
• Settings: every key of server.yaml, block by block, with what applies when the file sets nothing and what each key is for, from its comment in the code. Enter changes a key or opens a block; lists and maps of blocks (endpoints, models, prices) take entries added and removed. A save is read by the same loader the service uses first — nothing is written that would not load — keeps the file's comments, leaves the old one beside it, and offers to restart the service.
• Do: add a node, replicate the store, leave.

On a node, sudo llmgw opens the same settings for its agent.yaml.

The keys come from the configuration structs themselves, through the generator that writes docs/reference.md, so a setting added to the code is on the screen without anyone listing it, and a test fails if the two drift. That generator told blocks apart by name alone, so tlsconf.Config, engines.Config and supervisor.Config were each taken for the agent's Config and dropped: the reference had no TLS, engine or supervisor settings at all. It now has 37 blocks, six more than before.

A second control plane with llmgw server add, and the screens as windows

• sudo llmgw server add [user@]host [--vip 10.0.0.160/24], run on a control plane, makes another machine a standby: this llmgw copied over, a certificate from the cluster's authority, the same configuration and store address, the machine let into the store (for a store under Patroni, one change through Patroni's API), and the service started. Every node is told of it within half a minute and fails over to it untouched. --vip keeps a floating address on whichever control plane leads, with keepalived on both. Tried on new machines: switching the leader off, the standby led after 14 seconds and the floating address answered from it after 23.
• Nodes learn every control plane from the leader (a new ControlPlanes message, kept in /var/lib/llmgw-agent/control-planes.json), after the ones in their configuration.
• A node notices a dead control plane within a quarter of a minute. A leader whose machine stopped dead left its nodes' streams open until the kernel gave up retransmitting, about fifteen minutes; the agent now pings the stream after ten quiet seconds, and bounds dialling too.
• GET /leader: 200 on the control plane that leads, 503 on a standby, open like /ready — what keepalived follows.
• The store's access rules live in Patroni's shared configuration for a cluster set up by server init --patroni, so another control plane is let in without touching each member.
• The screens are windows: full screen, a title bar, the steps or actions down the left, the content in a bordered panel, the keys along the bottom, questions and errors as dialogs over it — the setup, the console, adding a node and one-machine init alike. Keyboard only.

Machines nothing may log into: llmgw join

For sites that allow no ssh into a machine, the setup screens offer a fourth way to reach one: a join code. The setup listens on port 7444 while it waits and shows a code for each such machine, with the commands that fetch llmgw onto it; there, sudo llmgw join <code> dials out — nothing dials in — checks by the certificate fingerprint in the code that it reached the setup that made it, admits itself with the code's secret, and runs what the setup sends, printing each step, until the setup says it is done. A code is good for one machine and an hour, and is under 90 characters so it can be copied whole from one screen into another.

To the setup a joined machine is one more way of running a script there: the checks, PostgreSQL, Patroni and etcd go over it unchanged. Tried with one machine joining by code and another logged into with a password, in one setup.

Setting up from nothing in screens: sudo llmgw

Run with nothing after it in a terminal, as root, llmgw opens full-screen setup on a machine with nothing installed, and a console on a control plane. In a pipe or a script it prints its help, as before.

• Setup asks one machine or three. For three — this one as the control plane and etcd's third vote, two more for the store under Patroni — it asks how to reach the other two: the ssh agent, a key file, or a password, and a sudo password where sudo asks for one. Passwords stay in memory for the run; a sudo password grants the user passwordless sudo until the install finishes, and the grant is removed after, failed or not.
• Every machine is checked before anything is installed: login, system (Ubuntu 22.04/24.04, Debian 12), architecture, CPUs, memory and disk for its role, clock against this one, NTP, the ports its role needs, internet, and whether it reaches the other machines. Failures stop it, with what to do; warnings let it go on.
• Then the plan, and the command line that does the same from a script, then the install with its progress, and the administrator's password.
• The console shows ctl status, refreshed every five seconds, and offers to add a node or, for a store on one machine, to replicate it.
• Machines are reached with Go's ssh client rather than the system's, so a password can be used; a host key is recorded on first sight and refused if it changes.

Tried end to end on three new machines whose user logs in and sudoes only with a password.

A store that survives losing its machine

deploy/ha/ now covers the store as well as the control planes: PostgreSQL under Patroni on two machines, its leader lock in a three-member etcd, and llmgw given every member with target_session_attrs=read-write. Measured: a planned switchover cost one failed lease renewal; switching off the primary's machine promoted the standby in 24 seconds with the gateway serving throughout. Example configurations are beside it.

llmgw server init --patroni user@host --patroni user@host sets it up: PostgreSQL, Patroni and etcd on those machines over ssh — this one as etcd's third vote — the role and database, and the store already in use moved in. Every step looks first; run against a cluster that is up, it changes nothing. Tried from scratch on three new machines (two minutes, then a failover by switching the primary off) and against the reference cluster (nothing touched).

Watching it: for a store of several hosts the control plane publishes llmgw_store_member, llmgw_store_replica_lag_bytes and llmgw_store_patroni_up from each member's Patroni, a member that stopped showing as missing rather than vanishing; llmgw ctl status has a STORE line; and four alerts — StoreHasNoLeader, StorePatroniUnreachable, StoreReplicaNotStreaming, StoreReplicaBehind.

Making an administrator on PostgreSQL (init --admin, users promote) failed: the flag was sent as a bool to an integer column, which SQLite takes and PostgreSQL refuses. Every existing PostgreSQL store got its administrator by being copied from SQLite, which is why nobody had met it.

llmgw server backup reads a store address of several hosts. It took the list for one host name and reached the database only by falling through to the second; it now hands libpq the list, and dumps from a standby when there is one.

People, limits and usage are one page

Managing a team meant its members on the people page, its quota on the quotas page and its spend on the usage page, each linking to the others. They are now one page in three tabs — Organizations & people, Limits, Usage — each at the address it had before, so links and bookmarks still land. The menu has one entry for them, People.

Choosing an organization, team, project or person in the tree, or a person in the account list, opens a panel beside it with everything about that one thing: its own limits and what it is held to from above, spend against its budget, its per-member limit, its capacity quota, and its requests, tokens and spend over the last week, drawn. Its limits can be set or changed there, and the answer comes back to the same panel.

The administrator's menu no longer has an Organizations entry beside it: /org and /org/<id> send an administrator to that organization's panel here. The page stays as it was for an organization's managers, who cannot see this one.

The chat page says whether a GPU or a CPU answered

A model can have replicas on a card and on CPUs at once, and the two answer at very different speeds; the chat page gave no way to tell which one a reply came from. The picker now says where each model runs — GPU, CPU or GPU + CPU — and the line under each reply names what answered it: GPU 6000-24g or CPU, beside its tokens a second.

Every answer through the gateway carries the same as x-llmgw-hardware: cpu, or the GPU class. An endpoint configured by hand, with no pool, sends none.

A card shared with something else is judged every heartbeat

A card with another process on it when its agent started was kept out of placement, and stayed out — "5014 MiB already allocated by another process" — after that process let go of it, until the agent was restarted. Found on a node shared with ComfyUI. The other way round, a tenant arriving after registration was never seen, and a card the agent's own engine held when the agent restarted read as someone else's.

The agent now reads every card's memory per process on each heartbeat, leaves out what its own engines hold, and sends the rest; the control plane keeps a card out of placement on that reading, and lets it back in when it falls under 512 MiB. llmgw_gpu_memory_used_bytes reads the same figure. An agent too old to send it keeps the reading from registration.

A GPU placement no longer runs on the CPU unnoticed

llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not where the loader looks), came up on the CPU, answered at CPU speed, and was reported ready while its GPU sat reserved and idle. Three checks now stop it:

• Installing the CUDA build asks it which devices it sees (llama-server --list-devices). None, on a node with GPUs, is a failed install, with what ldd says libggml-cuda.so is missing. The binary's own ldd could not say it: the backend is opened at run time.
• After the upgrade, engines the agent finds already running are held to the same check when it adopts them: one serving a GPU placement on the CPU is stopped, and its replacement starts through the check. The installed CUDA build is checked again within ten minutes, and installed afresh if it sees no GPU.
• Starting a shard on GPUs is ready only when its process holds memory on each of them (nvidia-smi --query-compute-apps, its process group included). One that does not is stopped and reported failed: "placed on GPU 0 but computing on the CPU", with the devices the engine lists. This covers a hand-installed binary (engines.llamacpp.bin), which nothing else checks.
• llmgw ctl engine-check llamacpp, run on a GPU node, now also fails when the engine sees no GPU, and otherwise prints the devices it computes on.

Shards are also given CUDA_DEVICE_ORDER=PCI_BUS_ID: placement numbers GPUs as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a node with two kinds of GPU, a shard could get a different card from the one it was given.

A model of many files, mirrored whole for vLLM

vLLM loads a directory — shards, config.json, the tokenizer — and does not read GGUF. Mirroring one could go wrong several ways: the explorer offered the shards alone, a node could be handed a version still downloading or missing a shard, and a repository's page given as a model's source was fetched as one file of HTML.

• The explorer fetches a safetensors set with what the engine reads beside it: config.json, generation_config.json, the tokenizer files, the shard index, the quantisation config. Never code: a .py would be the engine running what the repository ships, which is to be chosen on purpose.
• A version is offered to nodes only whole. The mirror refuses its manifest (409, with what is missing) while a file is still downloading, or while a shard its model.safetensors.index.json names, or a part of a split (-00002-of-00004), is absent. A node reports "not complete on the mirror yet" and is tried again.
• Checksums go with the files. The mirror keeps each file's sha256 beside it and the manifest carries it; a node checks every file of a set against it before using the set, and fetches one that differs again.
• llmgw mirror add takes several URLs into one version.
• A Hugging Face repository page given as a source is read as the repository id for a model vLLM or SGLang runs, which the engine downloads itself (the node then needs to reach Hugging Face), and refused for one llama.cpp runs, which needs a file.

With a vLLM profile of tp: 4, placement reserves four GPUs of one node for such a model, as before.

Signing up with an address that has an account

On PostgreSQL, signing up with an address already registered showed "something went wrong; try again": the store recognised a duplicate by the word "UNIQUE", which is SQLite's, and PostgreSQL says it in lower case under code 23505. It now knows both, and the address is answered as on SQLite — with the same page as a new sign-up, so the page cannot be used to find out who has an account. That page now also says what to do if the address had one: sign in, or ask an operator to reset the password.

An error the page does not explain says the account was not created and gives a reference (err-…), logged beside the error, for an operator to find it; the database's own words stay in the log.

An adapter no fabric uses does not keep a node out

llmgw doctor failed a node for a RoCE port with no RoCE v2 GID entry, or for ib_uverbs not being loaded, whether or not any fabric used them; on a node added for single-node inference, with no fabric given, that refused it for RDMA nobody had asked for.

• Those are failures only for ports a declared fabric uses, and only when a fabric is declared. Otherwise they are warnings saying nothing uses them yet.
• Which fabric an undeclared InfiniBand port is on, and whether it has a subnet manager LID, are notes rather than warnings; node add still warns once that adapters are up with no fabric given.
• node add --fabric hands its fabrics to doctor (doctor --fabric name:mlx5_0,…), so they are checked as declared even though agent.yaml is written afterwards.

A restart with no room beside it

ctl restart places each replacement beside the replica it replaces. With no room for it — one GPU, every card that fits held by the replica itself — the old replica was stopped only after the drain grace period, five minutes of "restart replacing" with nothing saying why, which read as a restart that never happened; scaling to zero and back was the way out. And a replacement that fit nowhere even with the old replica gone was stopped for all the same, leaving the model with nothing. Now:

• The replica is the only thing in the way, found by placing the replacement on a copy of the inventory without it: stopped at once, the replacement taking its place.
• The replacement fits nowhere even then: the restart is given up and the replica keeps serving. ctl replicas shows restart refused: and why.
• Room being made — a lower class preempted, an idle on-demand replica stopping — is waited for, as before.
• ctl restart --in-place stops each replica first, whatever room there is.

Every GPU node gets engine builds it can run, by itself

Which build of an engine a node installs follows its hardware — the CPU architecture, the driver, and the oldest GPU on it — from what each project publishes and how it was compiled. Each engine has a list of builds, best first; the node takes the first its hardware meets, and the first of those that installs and is seen to compute on the GPU. A build the table is wrong about costs its download, and the next one is tried.

	llama.cpp	vLLM
Driver 580+, Turing and newer	CUDA 13.3	the version asked for, CUDA 13 (PyPI)
Driver 525–579	CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570	the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch)
Volta (V100)	CUDA 12.8 on driver 570+, else Vulkan	0.19.1, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels
Pascal, Maxwell	CUDA 12.8 on 570+, else Vulkan	none: no vLLM is built for them
arm64 GPU (GH200, Grace)	CUDA 13.3, else Vulkan — never the CPU build	CUDA 13 or 12.9, 8.0 and newer
No NVIDIA GPU	the CPU build	none

Read from the projects' release pipelines: vLLM's x86 builds start at 7.5 from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build starts at 7.5.

• A node running an older vLLM says so: the cluster page's vLLM build column shows the version installed, and the installation's detail says why ("vLLM 0.19.1, the newest built for compute capability 7.0, instead of 0.30.0"). Profiles choose GPUs by class, so a model meant for the newer vLLM is not placed on a V100 by accident.
• The install computes on the GPU before it counts: a small tensor operation for vLLM, the devices llama.cpp lists for its GPU builds.
• The build can still be chosen per node on the cluster page (auto, CUDA 13, CUDA 12.9, none), with llmgw ctl engine-build NODE vllm BUILD, or POST /api/v1/nodes/{id}/engines; kept in the store and sent to the node at once.
• Whether an engine computes on its GPUs is read from every process nvidia-smi lists, not compute processes alone: the Vulkan build is a graphics one. A llama.cpp shard is given GGML_VK_VISIBLE_DEVICES beside CUDA_VISIBLE_DEVICES.

Engines per node, and nodes that do not run them all

A cluster with older GPUs could not add them cleanly: node add refused a V100 node for the vLLM it could not run, though it was meant for llama.cpp, and --ignore-checks passed over every other problem with it.

• What vLLM cannot do is a warning, in llmgw doctor and node add: the node joins without it and serves what it does run. node add checks the vLLM the control plane installs, not the binary's own default.
• node add --disable-engine vllm keeps an engine off a node for good: written as engines.disabled in its agent.yaml, never installed or advertised there, and not checked by doctor.
• engine_overrides in server.yaml give nodes picked by GPU class, id or labels a different version of an engine, or none — for exceptions; the build each GPU generation needs is chosen without it (above).
• The requirements follow the version, as the table above sets out; the threshold had been CUDA 13's for every version.
• An engine binary named in agent.yaml is used. The agent had installed its own beside it, which was then used first.
• The node page shows each GPU's compute capability.

Serving from the mirror says where a model will run

The serve form put every GGUF model on the CPU: it offered no choice, and its "Profile name" field, pre-filled "cpu", read as though renaming it would move the model to a GPU. Placement is decided by gpu_class, which the form never set. Now:

• Runs on lists the fleet's GPU classes with how many cards are free, and for llama.cpp the CPU. On a fleet with GPUs the class with the most free cards is chosen.
• GPUs per replica sets how many cards a replica takes on one node (gpus_per_node for llama.cpp, tp for vLLM), and parallel requests how many a llama.cpp replica serves at once.
• The profile is named for where it runs (t4, t4-x4, cpu) rather than typed.
• A line under the form says what the model will run as — engine, CPU or how many of which GPU, memory, context pool and how it is divided — and turns red when the CPU is chosen with GPUs free. The message after adding repeats it.
• Choosing the CPU while GPUs are free is refused unless run on the CPU anyway is ticked.

llama.cpp on several GPUs of one node

A llama.cpp profile takes several of a node's GPUs with gpus_per_node: N: placement reserves N cards of its gpu_class on one node, the engine sees those N (CUDA_VISIBLE_DEVICES) and spreads the layers over them (--split-mode layer, in proportion to each card's free memory, which for whole cards is even). tp or pp on a llama.cpp profile was a width nothing placed — the replica stayed unplaced or ran on one card. It is now read as gpus_per_node, so a profile written that way is placed on that many cards after the upgrade, rather than dropped; only both given and disagreeing, or nodes above one, is refused.

v0.10.6 — 2026-10-01

Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its format check and has no binaries.

Tried on the PoC cluster before release: LoRA adapters asked for by name through the gateway, two on one model, each answering as trained and the model itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent requests on the preferred replica, 4 on the other); and a bronze request displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried live: a model served from an external provider, which needs a provider key.

Capacity from the admin pages

What v0.10.5 made settable from the command line is now on the pages too:

• Quotas has a Capacity section: every scope's guaranteed and most GPUs, cores and memory beside what it holds, with a form to set, change or remove one. It goes through the same checks as llmgw ctl quota.
• A model's card on the mirror page sets its class, its owner (when there are quotas to count against), and the memory and cores a replica reserves, beside its replica counts. Only what changed is applied, and each change is said.
• Room counts what a class may take. The card, llmgw ctl model target and the headroom API count the replicas a model could get by stopping those its class outranks, separately from free room, and name whose they would be. A critical model's card no longer refuses a replica that preemption would make room for.

LoRA adapters

A llama.cpp model can carry adapters, each a name and a GGUF LoRA file. The nodes fetch them with the model's weights and llama.cpp loads them all at start, applied to nothing (--lora-scaled FILE:0); a request that names an adapter goes to the model's replicas with "lora": [{"id": i, "scale": s}], so one replica serves the model and all its adapters. Adapter names share the namespace of models, are listed in /v1/models, and follow key allowlists on their own — a key allowed an adapter does not reach the model under it. Adapter files are kept from weight cleanup like the model's own. Not yet for vLLM or SGLang profiles, whose adapters are loaded differently.

Finding and attaching adapters from the mirror page

• Explore has a LoRA adapters view: adapters for any model, or — from a held model's card, find adapters on Hugging Face — those made for the model its repository was made from, since a GGUF repository is a quantisation and adapters are made for the original.
• Opening a repository reads the first bytes of each GGUF file and marks it a LoRA adapter or a model. Repositories tagged "lora" hold a model with the adapter merged in about as often as an adapter, and only the file says.
• Held adapter files are marked in the list, and a model's card lists its adapters, offers the held adapter files of its architecture to attach under a name and scale, and removes them. A file that is not an adapter, or is for another architecture, is refused before it reaches the catalog.
• llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME] lists and changes them from the command line, through POST /api/v1/models/{name}/adapters, with the same checks.

Models served outside the cluster

A catalog model can be an OpenAI-compatible API elsewhere: external: {url, model, api_key_file} and no profiles. Nothing is placed for it; the router sends its requests to the provider, rewritten to the provider's name for the model and with its key, read from the file on every publish so a rotated key is picked up. Named in another model's fallbacks, it is used only when the cluster has no capacity left. It is never scraped for metrics, and never reads as stale. Prices, budgets, limits, usage and key allowlists apply as to any model — and since its prompts leave the cluster, the allowlist is how to decide who may send them (docs/install.md). ctl models and the cluster page mark it external.

Pool spillover

A model served from pools of different priority — H100 before A100, say — was routed to the preferred pool however full it was. Now a pool whose replicas are saturated, by the same measure admission uses (queue past its healthy depth, or KV cache past its ceiling), passes requests to the next pool that is not. When every pool is saturated the preferred one is kept: a pool that is just as full gains nothing.

Displacement: a higher band is not refused while a lower one waits

Each band's queue was bounded, and nothing bounded them together, so a full gateway turned an interactive request away while batch requests sat queued behind it. admission.queue_capacity_total now bounds everything queued — by default half of every band's capacity together. Reached, a request is not refused while a lower band has any queued: the newest request of the lowest band is turned away instead, with a 503 and Retry-After: 5, so it backs off and returns. The newest, because the oldest have waited longest and are nearest their TTL. Counted as displaced in the admission rejection metric.

Releases can be signed

llmgw release keygen makes an ed25519 key pair. With the private key in the CI secret RELEASE_SIGNING_KEY, releases carry checksums.txt.sig; with the public key in internal/release/trusted.pub, every binary built after refuses a release not signed with it, wherever it installs one. Off until then, and docs/install.md says in which order to turn it on.

Restarts borrow a lower class's room

A restart places the replacement beside the replica it replaces, and where there is no free room it now makes some the way any placement does: by stopping a replica its class outranks, which gets its room back once the old replica has gone. The in-place restart — the model unserved while its replacement loads — is left for when nothing of a lower class is running. On a cluster where every model is standard, that is every full one: mark the models that can wait opportunistic (llmgw ctl model class, or the model's card).

v0.10.5 — 2026-09-30

Published first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its format check and has no binaries.

After upgrading:

• Declare what CPU models need. A CPU model added from now on must say its memory; one stored before this still loads and is flagged in ctl models until llmgw ctl model resources MODEL --memory GB --cores N is run. Cores are then reserved per replica and become the engine's thread count, for replicas started after the change: llmgw ctl restart --model MODEL.
• Upgrade the mirror on its own host, now with one command: sudo llmgw upgrade v0.10.5 --service llmgw-mirror (by hand once, for a mirror older than this). llmgw ctl upgrade says so at the end.

CPU cores are reserved, and memory has to be declared

The first step of resource allocation for larger clusters: what a replica holds of a node is reserved, not assumed.

• Cores. A profile's cpu_cores is reserved out of its node's CPUs, as memory is, and is what its engine computes on (llama.cpp's --threads). A node holds as many CPU replicas as it has cores for, and two on one node no longer spin against each other. Agents report their CPU count; a node that does not, and a profile that declares no cores, constrain nothing.
• Memory is required of new CPU models. A model with no GPU is all host memory, and one declaring none looked to placement as if it fitted everywhere, any number of times. Adding one without min_memory_gb is now refused. Models stored before this still load, and say what is missing.
• Declared against measured. Each model shows what one replica reserves and the most one of its replicas uses — in ctl models (RESERVES) and on the cluster page — with a warning when nothing is declared or when it uses more than a tenth over what it reserves.
• Setting them. llmgw ctl model resources MODEL --memory GB --cores N (and POST /api/v1/models/{name}/resources) for a model added at runtime. ctl model serve and the mirror's serve form fill in cores — half the smallest node the model fits on — beside the memory they already estimate. Running replicas keep their old figures until restarted.

The commands are llmgw ctl model resources and llmgw ctl model class; in rc.7 the first was briefly llmgw ctl resources.

Capacity classes, and preemption

A model has a class — critical, standard or opportunistic — saying what its capacity is worth against another's when there is not enough for both. Unset, an on-demand model is opportunistic and every other model standard, which is how the catalog behaved before. llmgw ctl model class MODEL CLASS sets it.

When a model cannot be placed, replicas are stopped to make room for it, by these rules:

• A lower class gives way to a higher one.
• Within a class, only replicas above a model's minimum — room it grew into under load — and only for a model still below its own minimum.
• A critical model's minimum is never taken, nor a replica that has only just loaded, nor one spanning several nodes.
• Idle on-demand replicas go first, then what lower classes grew into, then the same within the class, then lower classes' minimums; the longest unused first within each, and of equally idle ones the smallest.
• The set is tried on a copy of the inventory first, at most three replicas, and nothing is stopped unless it makes the room. Of a set that does, each replica whose stopping turns out not to be needed is left running, so an idle replica tried first on the wrong node is not stopped with the one that made the room.

A replica stopped this way is drained and given 30 seconds to finish what it is serving. Its model is held back for up to five minutes, or until the model it gave way to has its replicas, so it does not take the room straight back. ctl models shows each model's class and "gave room to X" while held; the cluster page shows the class and how many replicas were lost; the metric is llmgw_model_preempted_total{model,class}, and every preemption is logged.

Capacity quotas for the tenancy tree

An organization, team or project can be guaranteed GPUs, CPU cores and memory, and limited to at most some amount of each. What it holds is what the replicas of the models it owns hold — and those of the teams and projects beneath it. llmgw ctl model owner MODEL team:Acme/ml says whose a model is; one with no owner is shared and counts against nobody.

• A guarantee is kept. A replica within its owner's guarantee is never taken for another scope's model, whatever the classes: a guarantee a higher class elsewhere could take would not be one.
• Beyond it is borrowing. Free room is used; when a scope within its own guarantee needs room, what others hold beyond theirs is taken back first, whatever their class — except a critical model's minimum.
• A limit is a ceiling. A model whose owner would go past a limit anywhere up the tree gets no further replica, and says which limit; headroom and the mirror card stop there too.

llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16 (also --cores, --memory and their limits), llmgw ctl quota list with what each scope holds, llmgw ctl quota clear. Scopes are named by path or ID; a name that could mean two is refused. Changes take effect within half a minute. Metrics: llmgw_capacity_holds, llmgw_capacity_guarantee and llmgw_capacity_limit by scope and resource; alert: CapacityAtLimit.

Sized by what a replica was measured to serve

What llmgw ctl bench MODEL --save measured — the most requests at once one replica served while meeting its SLO, and its throughput there — now drives decisions rather than only being printed:

• Scaling up when requests per replica pass what was measured, not only when the engines' queues fill: an engine takes requests into its slots long after latency has gone.
• Scaling down only when the replicas left would stay under 70% of what they were measured to hold, so a replica taken back does not put the rest straight over.
• Choosing a profile. Among profiles of equal priority, the one measured to serve the most per GPU — or per core — is tried first. Priority set by an operator still wins.
• Sizing. llmgw ctl model target MODEL --tokens-per-sec N says how many replicas that takes, how many run, and whether the rest fit, with the reason when they do not; --apply sets it if they do. Headroom reports what one more replica would serve.

A model never measured behaves exactly as before. Measurements are read from the store every half minute.

Upgrading a machine the control plane does not reach

sudo llmgw upgrade VERSION --service UNIT upgrades the machine it runs on — the mirror's host above all, whose service runs unprivileged and cannot replace its own binary as an agent does. The release's checksums are checked, the old binary kept, the unit restarted and watched until it stays up.

Nodes report their own heat and power

The first step of facility-aware placement (docs/facility-aware-placement.md), observe-only: nothing is decided on these yet. Every agent reads, at most every ten seconds, and sends with its heartbeat:

• GPUs from nvidia-smi: temperature, HBM temperature where the driver has it, power draw and the enforced limit, and what is holding the clocks down — power cap, thermal slowdown, power brake. A GPU's temperature was read once, at registration; it is now current.
• CPUs from hwmon, which is what lm_sensors reads, with nothing to install: the package temperature, and the board's inlet air sensor where it has one.
• CPU power from RAPL's energy counters.

A sensor a machine does not have is not reported, never shown as zero. New metrics: llmgw_gpu_power_watts, llmgw_gpu_power_limit_watts, llmgw_gpu_memory_temperature_celsius, llmgw_gpu_throttled{reason}, llmgw_node_cpu_temperature_celsius, llmgw_node_cpu_power_watts, llmgw_node_inlet_temperature_celsius. Each node's page shows them, with a GPU table of temperature, power against its limit, and what is throttling it.

The mirror page in tabs

/admin/mirror was one long page: what the mirror holds, the Hugging Face explorer and the fetch form, one under another, with every visit waiting on Hugging Face for a list it might not show. It is now three tabs — Held (with how many are held, downloading and failed), Explore Hugging Face, and Fetch by URL — each with its own address. Only the tab shown does its work, and only Held reloads while something downloads. A download started from the explorer links to Held, where its progress is; a fetch the mirror refused comes back on the form it was sent from.

A held model opens to where it runs, and how many

Clicking a model under Held opens a card beneath it: each replica with its node, state, age and the CPU and memory its engine uses, and how many more replicas the cluster has room for now, on which nodes, and why not more. The replicas it always runs and the most it grows to under load are changed there. Room is found by trying the scheduler itself on a copy of the inventory — the same decisions the next passes would make, not an estimate beside them — and a number of always-running replicas it cannot place is refused with the reason, rather than accepted and left short. A ceiling above the room is taken, and said. The same answer is at GET /api/v1/models/{name}/headroom on the admin API.

Each row also says how its model runs without being opened: replicas ready of those it should have, how many are still starting, and the nodes they are on — in amber when short, in red when parked or when it does not fit. It is one line per model beside its mode, and nodes drop the part of the name every node shares (node05 for llmgw-node05, the full name on hover).

One line per record, in every table

The same rule across the admin pages. Headers do not wrap; a record is one line; free text is cut to its column with the whole of it on hover rather than given a row of its own; a row across the whole table is only a note or an open form. On the cluster page that means a node is one line (its address and logs beside its name, not under it), a model's description is a column rather than a second row, node names drop the prefix they all share, and the GPUs, classes, fabrics and fallbacks columns appear only when some row has something in them — on a CPU-only cluster they were three columns of dashes pushing the table past the page. On People, a person in many teams shows two and a count, the rest on hover.

v0.10.4 — 2026-09-30

Published first as release candidates v0.10.4-rc.1 to rc.9.

After upgrading, run llmgw ctl restart --all once: CPU replicas take the new thread count below only when they start.

CPU replicas no longer fight over the CPUs

llama.cpp computes on every CPU of the machine unless told otherwise, and the agent did not tell it. Two busy CPU replicas on one node then spin against each other: on an 8-vCPU node, 7B generation fell from 10 tokens a second alone to 0.8 each with both busy, in two rounds of the same test. The agent now starts a CPU replica on half the machine's CPUs, where one alone makes 6.5 and two busy make 5.5 each. llamacpp.threads in the agent's configuration sets it for a node — all of them suits a node that only ever runs one CPU replica — and a profile's own --threads still wins. Measured on VMs whose host is oversubscribed; the proportions, not the numbers, are the point.

Tracing

With tracing.endpoint set (or OTEL_EXPORTER_OTLP_ENDPOINT), each inference request is traced over OTLP/HTTP: a span for the request, carrying its X-Request-Id, with admission, scheduling and each attempt at an engine under it. A caller's own traceparent is continued, and the engine is sent one for its attempt. tracing.sample_ratio traces a share of the requests that arrive without a decision. Off, it costs nothing.

Restarting replicas

llmgw ctl restart REPLICA... | --model M | --all replaces replicas with new ones. Each replacement is started beside the replica it replaces, which keeps serving until the new one is ready; replicas of one model go one at a time. Where no second copy fits anywhere, the old replica is stopped after the drain grace period and the new one starts in its place. This is how running engines pick up the keys introduced in v0.10.3, an edited profile, or a new engine version.

A model that keeps failing is parked, and everybody is told

A model whose replicas failed at every start was retried forever: kev-0.5b, whose architecture the nodes' llama.cpp did not know, was started 155 times in eight hours. After five failures in a row a model is now parked — not placed again until its definition changes or llmgw ctl retry MODEL is run — and ctl status, ctl models and the admin API say so with the last error.

A request for a model that will not come up — parked, or fitting on no node — is answered at once with the reason, instead of "the model is loading; retry shortly", which a client would retry forever. Each request still makes the control plane try placing it afresh, so a model that fits once room is made is not refused on an old answer. "no placement for model" is logged when its reason changes or every ten minutes, not on every pass: it was 250 identical lines in two days for one model.

Metrics and alerts for what goes wrong

New series: llmgw_model_replicas_desired, llmgw_model_start_failures, llmgw_model_parked, llmgw_model_unplaceable, llmgw_node_engine_install, llmgw_control_plane_leader and llmgw_certificate_expiry_timestamp_seconds. llmgw_model_replicas_ready is now published at 0 for every model that should be running; it was published only while a replica was ready, so ModelHasNothingServing could never fire. New alerts: ModelParked, ModelFailingToStart, ModelCannotBePlaced, EngineInstallFailed, ControlPlaneLeadership, CertificateExpiresSoon and CertificateExpiresVerySoon.

ctl status counts a model against the replicas it should have now rather than its minimum, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied.

Backups

llmgw server backup writes the store and /etc/llmgw — configuration, secrets, and the certificate authority's key — to one archive readable by root only, keeping the newest 14. deploy/systemd/llmgw-backup.timer runs it nightly. The control plane publishes the newest archive's time as llmgw_backup_last_success_timestamp_seconds, and BackupTooOld fires when it is more than 26 hours old or there has never been one. docs/install.md now says how to restore, including why running engines survive it: the engine-key secret comes back with the store.

Replicas follow load between min and max

A model's max was a ceiling nothing reached: every model ran exactly its minimum however long its queue. Now, when requests are waiting at the engines — every slot of every replica busy — at one or more per replica for a minute, one replica is added, up to max; ten minutes with nothing waiting takes one of those away, down to min. One step at a time, and not while the previous one is still starting, so a spike that is over before a replica loads does not leave several behind. Each decision is logged with the load that made it. llmgw ctl scale MODEL --min 1 --max 3 turns it on for a model.

A model from the registry in one command

llmgw ctl model serve OWNER/REPO takes a model from the registry to a replica that answers. It picks the GGUF file (--quant, Q4_K_M by default), has the mirror fetch it and checks what arrived against the registry's published sha256, checks that the nodes' llama.cpp can load it, sizes its memory from its header, adds it to the catalog, waits for a replica — stopping early with the reason if it fits nowhere or keeps failing — and asks it to say hello. Each step is shown as it happens. It runs in the control plane as a job, so a download of hours does not depend on a terminal staying open. --on-demand adds it without starting a replica.

Upgrading from one command

sudo llmgw ctl upgrade VERSION, run on the control plane, upgrades that machine and then every node in turn. Each agent is told the version over its existing stream, fetches the release for its own platform, checks it against the release's checksums, keeps the old binary beside the new and restarts, while its engines keep serving. A node that does not come back on the new version, ready, with its replicas, stops the upgrade before the next one. Running it again resumes. Agents from before this have to be moved by hand once.

Following a request

Every response carries X-Request-Id — the client's own when it sends a plain one, otherwise one the gateway makes — and the engine receives it too. One access log line per request records it with the model asked for and served, the replica, status, time queued, time to the first token, total time, tokens, and the key, project and user. Usage records keep the id, replica, queue time and time to first token, and llmgw ctl request ID reads one back: the answer to "why was this slow?".

Rate-limit headers

Responses carry OpenAI's x-ratelimit-limit-*, x-ratelimit-remaining-* and x-ratelimit-reset-* for requests and tokens, from whichever level of the tenancy tree will refuse first, so SDKs and editors can slow down before they are refused.

Memory sized from the model

The mirror's serve form sizes a llama.cpp model's memory from its GGUF header: weights, plus the KV cache for the context pool from its layers, KV heads and head width, plus the engine's own. At a 16k pool that cache ranges from 0.44 GB to 5.25 GB across the models on the live mirror, which the old "weights plus half again plus two" could not see.

Security

• Alertmanager answered anyone on the network, including requests to silence alerts. The monitoring README now binds it to loopback, where Prometheus beside it is its only client.
• Login and signup could be made to use gigabytes. Each attempt hashes with 64 MiB of argon2, with no bound on how many ran at once, and the per-address throttle grew by one entry per address tried. Hashes now run a few at a time and the rest wait; the throttle expires old entries, and when it is still full it refuses new addresses rather than dropping live ones — which would have let a flood of made-up addresses reset the throttle on a real one.
• Any key holder could grow Prometheus without bound by asking for made-up model names, each of which became a metric label. Requests are counted under the catalog's name for their model, or "unknown".
• A SQLite backup left a full copy of the store readable by local users while it ran. The snapshot is now made in a private directory, and llmgw-backup.service sets UMask=0077.
• An engine archive could write outside its directory through chained links. Archives are unpacked through os.Root, and a link may only point at a file beside it. Every llama.cpp b10991 archive, CPU and CUDA, already follows that rule.
• Onboarding made every node trust the cluster's root for everything. It was added to the system store, so a certificate the control plane signed would pass for any site with every TLS client on the node. The agent trusts it through its own configuration only; onboarding no longer adds it, and takes it out of a node that an older version onboarded.
• Any node could manage a mirror running mutual TLS, since one root signs every node's certificate. llmgw mirror serve --api-client NAME answers only the certificate with that common name, and the mirror warns when mutual TLS runs without one.
• The administrator's password stayed in init.log. It is shown once, on the screen, and the log keeps a placeholder.
• Auth off on a public address was silent. The server now warns at start when the API listens beyond loopback without keys.
• The actor recorded with a change was whatever the caller sent, line breaks included. It is cut to a short printable name, and sudo llmgw ctl records the person who ran sudo rather than root.
• vLLM's Ray head ran its dashboard, whose job API runs code for anyone who reaches it. It is off. The head itself still has no authentication, and SGLang still takes its key as a flag; docs/install.md says what to do about both. The monitoring README says how to keep Prometheus's queries to the control plane.

Fixes

• Tests that slept a fixed time and then asserted now wait for their condition. They are the likeliest cause of CI's intermittent failures on a loaded runner.
• ctl reported a change the API had queued (202 Accepted) as an error, as ctl restart did on its first real use.
• The status each failure is answered with — 499, 502, 503 with and without Retry-After, 504, 429 — is now tested case by case.

Releases

Small fixes are published as release candidates of the next version — v0.10.4-rc.1, rc.2 — which sort after v0.10.3 and before v0.10.4. Their release notes come from this section, and they are marked as pre-releases. Onboarding treats a release candidate as a release and downloads it.

v0.10.3 — 2026-09-30

Security fixes from a review of v0.10.2. Three of them were confirmed on the live cluster before being fixed.

Engines demand a key only the router holds

Engines listened on every address of their node with no key. A request from a neighbouring node reached the model directly, past the gateway's authentication, quotas and accounting. Each replica's engine is now started with a key derived from the replica id and a secret in the store. The router, its metrics scrape, the agent's readiness check and ctl bench send it. llama.cpp and vLLM receive it through their environment, not their command line. Engines started before this keep running without a key until their replica is next started.

The mirror's management API answers only who it was told to

Without --api-allow, the management API answered anyone who could reach it, including every node. It now answers this machine only; the control plane is let in with --api-allow or mutual TLS. Upgrading a mirror set up by hand: add --api-allow <control plane address> to its unit. Downloads started through the API go to http and https only, and never to loopback, private or link-local addresses (--allow-private-sources to permit that). Model, version and file names can no longer be .. or start with a dot.

The admin listener refuses what a browser page could send

The admin API has no login and trusts whoever reaches it. A page open in an operator's browser, reaching it over an ssh tunnel, could post a model definition. The listener now requires Host to be an address or localhost, refuses cross-site requests, and takes changes only as application/json. A profile's env may no longer set variables that make an engine run other code (LD_*, PATH, PYTHON* start-up variables, BASH_ENV and similar). They are refused where a model is defined and dropped again before an engine starts.

Fixes

• The agent installs only engine versions that look like one, whatever the control plane sent; a version of ../x could have installed outside /opt/llmgw/engines.

v0.10.2 — 2026-09-29

• After server init's screen closes, only the administrator's password and the next steps are printed, not the whole run again under the screen that just showed it. The whole run is kept in /var/lib/llmgw/init.log, root's only, and printed in full when init fails.
• The screens' spinner and text field are llmgw's own rather than from bubbles, one dependency fewer. Every llmgw command still asks the terminal for its colours as it starts, which bubbletea v1 does on purpose; a terminal that never answers waits for its timeout.

v0.10.1 — 2026-09-29

llmgw ctl fabric-test: two nodes, rail by rail

llmgw doctor checks a node on its own, and every check can pass while two nodes still cannot reach each other: a cable into the wrong switch, a partition key, a RoCE network with no route. llmgw ctl fabric-test A B runs ib_write_bw from each of A's adapters on a fabric to the same adapter on B, one rail at a time, through the agents — no SSH between nodes, and perftest is installed the first time — and grades each against its link's rate: ok at 80%, slow at 50%, fail below or when nothing got through.

Agents now say which requests they answer beyond the usual (features), so one too old for the test is refused at once instead of waited for.

Written against perftest's real output and tested with it, but not yet run over real InfiniBand or RoCE: this cluster has neither.

The mirror on its own machine, in two commands

llmgw mirror init --control-plane <address> sets a machine up as the model mirror: the user, /srv/models, and a service serving files to every node on :8081 and the management API on :8082. server init --mirror http://<mirror>:8082 uses it instead of running one beside the control plane. It took a unit written by hand and an edit to server.yaml before.

The management API can fetch and delete models; mirror serve --api-allow makes it answer only the addresses given and the machine itself, and mirror init gives it the control plane's. With --no-node, the starter model is fetched and catalogued anyway, unpinned, and starts on the first node added.

Tried on three empty VMs: mirror, control plane and a node, and a request answered by the starter model on the node.

server init asks, and node add takes several nodes

• sudo llmgw server init in a terminal, without --admin, asks for the administrator's email, the other names nodes will reach the machine by (showing the addresses it already found), SQLite or PostgreSQL, and whether to run the mirror, a node and the starter model here. Then its steps fill in as a list, downloads updating in place, and the administrator's password stays in a box until the end — and is printed again after, with the whole run, so it is in the terminal's history. The last screen says how to add a node and to back up the authority's key. --plain, or no terminal, prints lines as before.
• llmgw node add a b c adds each in turn and ends with which were added. A fabric named for one node's adapters is recognised on the next by what the adapters report, and not asked again.

Both tried on empty VMs: init from the form to a starter model answering, and two nodes added to it in one command in 40 seconds.

v0.10.0 — 2026-09-29

Independent InfiniBand fabrics and NVLink domains, told apart, drawn and placed on; a node checked by llmgw doctor and added by llmgw node add; vLLM installed by the nodes themselves; people in many places at once; and a chat page, editor setup and a model explorer for the people using it.

Independent InfiniBand fabrics, told apart and drawn

A fabric was whatever agent.yaml called it, and nothing checked. At a site with several independent fabrics, two given one name would have had a replica spread across both — started, then hung on its first collective. Now:

• Identity. Each active InfiniBand port reports the master subnet manager it sees (sminfo) and every subnet manager on its fabric (saquery SMIR), asked once a minute through one port per fabric. Ports whose managers are listed together are one physical fabric, so a failover does not split one in two. A RoCE port's fabric is the IP network of its RoCE v2 address. The agent installs infiniband-diags where there are adapters and no sminfo.
• Placement divides a declared fabric by that identity and forms a replica on one part only. A node whose adapters for one name are cabled to two fabrics is refused; a node that cannot say is kept apart from those that can.
• Subnet managers. The cluster page lists each fabric's master, its priority, its standbys and its master changes over the last day, and warns when a fabric has no standby, when its ports disagree on the master, when a port is up with no subnet manager (INIT), and when one name covers two fabrics or two names one fabric.
• Ports while running. The agent re-reads every InfiniBand and RoCE port on each heartbeat: state, physical state, rate, error and data counters. A port going down takes its fabric off the node within seconds, and back when it returns — it used to wait for the agent to restart.
• Maps. Every five minutes one node on each fabric runs ibnetdiscover, and the control plane parses it. /admin/fabrics draws the fabric: switches in their tiers, adapters under their leaves, cables coloured by rate, degraded ones — trained below the rest of their kind — marked in red, rail alignment, and the cluster's own adapters live, with their traffic running along the cables. It zooms, pans, highlights what a switch or adapter touches, and opens detail on a click. Placement prefers the nodes of one leaf and puts nodes with a degraded cable last.

NVLink, on every build and across nodes

• The release binary sees NVLink. It is built without cgo, so without NVML, which was the only source of NVLink peers: on every NVLink machine the portable agent reported none. Discovery now reads nvidia-smi topo -m: NVLink peers, each card's NUMA node, and how near each network adapter is.
• GPUs are chosen by it. Every set of the right size is scored: NVLink pairs within it, then — for a multi-node shard — nearness to the fabric's adapters, then NUMA nodes spanned. It used to grow one card's peers, which took cards joined to that one but not to each other.
• Multi-node NVLink domains — an NVL72 rack, an NVLink Switch system — are found from each GPU's fabric state, ClusterUUID and CliqueId, with nvidia-imex and its channels. A replica spanning nodes is placed inside one domain first, on nodes whose every card is joined and whose IMEX is up, and its shards get NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback. /admin/nvlink draws each domain as a rack of trays and says what keeps a tray out. A node's page shows its GPU-by-GPU NVLink matrix.

Multi-node launches fixed before they meet hardware

• vLLM launches multi-node its own way by default. An empty engines.vllm.launch_mode meant ray, never checked against a real cluster; it now means mp, every node running vllm serve with --nnodes, --node-rank and --master-addr, the workers --headless, as vLLM documents. The mp path used to give workers no rank and the leader's address as their own. Set launch_mode: ray to keep Ray.
• NCCL bootstraps over the node's own interface. The agent's default of eth0 meant it never looked for the interface holding its management address, and the control plane sent one name for every node. A node on ens3 or bond0 had NCCL bind nothing.
• RoCE shards get NCCL_IB_GID_INDEX, the RoCE v2 entry with an IPv4 address; NCCL's default is a v1 link-local one that does not route.
• A Ray worker now waits for its head before joining.
• The agent unit sets LimitMEMLOCK=infinity, which RDMA needs.

llmgw doctor

Checks a node for what llmgw needs of its GPUs and fabric, and says what is wrong and how to fix it, most serious first: NVSwitch without the fabric manager, GPUs not joined to their NVLink domain, IMEX, nvidia_peermem and ib_uverbs, each port's link and subnet manager, a fabric with no standby, RoCE without a v2 GID, the agent's declared fabrics against the adapters, the interface NCCL will use, the unit's memlock limit, and the NCCL environment a shard on each fabric would be given. --json writes every fact it read, to send back; it exits non-zero when something fails.

It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU model, where llama.cpp runs several times slower and nothing else looks wrong — and whether vLLM can be installed on the node, and why not.

vLLM installs itself on GPU nodes

The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU that can run it, with nothing done by hand. A Python 3.12 environment is built by uv — itself a GitHub release checked against its checksum — from PyPI, resolved as PyPI stood a day after that vLLM was published and installed with --require-hashes, so every node gets the same files whenever it is built. The resolved list stays in the environment as llmgw-requirements.txt.

• The default is vLLM 0.30.0, which takes every flag llmgw passes. Set engines.vllm in server.yaml to change it, or to "" to keep your own.
• It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and 25 GB free while it installs. A node that lacks one says which on the cluster page; a node with no GPU skips vLLM without a word.
• gcc is installed with it, since Triton compiles its kernel launcher on the first model load.
• SGLang installs the same way, on GPU nodes, when engines.sglang names a version.

Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB, torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM itself was not started there.

A node is asked only for engines its agent says it can install. An agent older than this release says nothing, and is asked for llama.cpp alone, so a cluster half way through its upgrade does not show every node as failing vLLM.

llmgw node add: a node in one command

Adding a node was a bash script from the repository, not installed anywhere, that needed the node's name, the control plane's address and a dozen flags. It is now part of the binary:

sudo llmgw node add ubuntu@10.0.0.21

run on the control plane. The node id is the machine's hostname and the address its agent dials is this machine's on the route to it, checked against the server certificate. It installs llmgw on the node — the release for its architecture, fetched and checked there, which works on RHEL too — runs llmgw doctor there and stops on a problem, issues and installs the certificate, writes agent.yaml and the unit, starts the agent, and waits for the node to join and install llama.cpp.

In a terminal it is a checklist that fills in, and asks what a script cannot: which fabric each group of active adapters is on — grouped by the subnet manager or RoCE network they report — and whether to go on past a problem. Elsewhere it prints one line per step. Running it again on a node is safe.

llmgw node remove forgets a node that is gone for good, which otherwise stays listed as lost until the control plane restarts; only a lost node, or an unreachable one with nothing placed on it, is removed. llmgw ctl nodes no longer lists vLLM as skipped on every CPU node.

Tried on an empty VM, from the terminal screen and plain: joined with llama.cpp ready in under a minute, and 17 seconds to add it again.

People belong to many places, or to none

A person was in exactly one project, chosen at approval, and could not use the gateway before that. Now:

• Memberships. A person can be a member of any number of organizations, teams and projects, or of none. A member of a team or organization may use every project in it. llmgw users join/leave, and the people page adds and removes them from either side. Approving no longer requires a project.
• Personal keys. A key can belong to a person with no project. On /profile a person chooses what a new key spends against: one of their projects, or Personal. llmgw keys create --for <person> does the same. auth.personal sets default limits for personal use, or turns it off.
• Limits for people. --scope user limits a person across every key, project and the chat page. --per-member on an organization, team or project gives each member their own share there. Both are on the quotas page, and are enforced alongside the tree, each in its own pool.
• The chat page asks what to spend against when a person has more than one choice, and checks it on every message.
• Spend is recorded with who spent it, so a person's budget survives a restart, personal use and the chat page included.

Leaving a project revokes the keys the person made there, unless another membership still reaches it. A personal key keeps no conversations or notes: those are kept per project.

An organization's own page

A member of an organization can be made its manager (users join --org … --role manager, or the star on the people page). A manager gets /org: the organization's usage over time by team, project or person, its teams and projects with limits and spend, and its members — read-only, and only what was spent in its projects. Administrators can open every organization's page.

The chat page

• A model that is not running shows a progress bar while it loads — waiting for a node, downloading, starting — and the message is sent once it is up, rather than failing after 45 seconds with "the model is loading".
• A finished reply is rendered as markdown, by the server under the model cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
• Stop, while a reply streams; its tokens, speed and time to first token under it; a system prompt and a temperature; and the conversation kept in the browser across a reload — the gateway still keeps none.

Connect an editor, and the Responses API

• The account page makes a key for Continue, Codex or Cline and shows what the tool needs, filled in: Continue's assistant file, Codex's provider and profile, Cline's four settings. Continue and Codex get a one-line command (curl -fsSL <gateway>/setup/<tool>?model=… | sh) whose script holds no secret — the key comes from the environment — keeps a copy of whatever it replaces, and leaves the rest of the file alone.
• POST /v1/responses translates the Responses API, which Codex speaks, to chat completions and back — instructions, messages, function and custom tools with their calls and outputs, usage, streamed or not — down the same path as any request.

Your own use

The account page (was "Your keys") shows the last thirty days of a person's own use and, for each place they spend, a bar for every budget that applies — their own, or a project's shared with others, said as such.

The mirror page

• Explore models opens on what is trending on Hugging Face, with views for new and most downloaded, filters for task and size, and each repository's parameters, architecture (✓ where the nodes' llama.cpp loads it), context, base model and license. "Only what these nodes can run" is on by default.
• A repository opens beneath its row with its model card, and a download started there starts in place, its progress followed in the table above.
• What the mirror holds is grouped by model family.
• An importance matrix published beside a model's quantisations is no longer offered as a model.

How a model runs. Each model a mirrored file is serving now shows how it runs — always on, on demand, or stopped — and it can be changed there, without the CLI. The serve form asks the same: on demand loads the model when somebody asks for it and stops it after 15 minutes unused. A model declared in server.yaml shows its mode and says to change it there, because the file is read on every start.

Weights the nodes cannot load are not fetched. kev-0.5b was a GGUF whose architecture, ggmlc, llama.cpp does not know. It was downloaded, catalogued and placed, and failed on every node it was tried on. The architecture was in the first kilobytes of the file all along. Now:

• The mirror page's registry listing shows each file's architecture and, for one the nodes' llama.cpp cannot load, says so instead of offering to fetch it. A vision projector (mmproj) is marked as not a model on its own.
• A fetch reads the file's header first — a range request, not a download — and refuses one that cannot be served, unless "fetch anyway" is ticked.
• Serving from the mirror and llmgw ctl model add refuse the same.

The list of architectures is llama.cpp's own (LLM_ARCH_NAMES) at the build llmgw pins, b10991: 152 of them. A different build configured in engines: is not refused, only not checked.

Each node's CPU and memory, and a page per node

The agent reads /proc between heartbeats: CPU across all cores, load, memory used and available, and each engine's cores and resident memory. The cluster page's node table shows CPU and RAM as bars; each node has a page with its current load, its engines, its fabric ports, its GPUs, and an hour of charts from Prometheus.

The admin pages

• The cluster page leads with capacity, nodes, models and replicas, then usage; it refreshes in place instead of reloading, and its usage table names people and projects instead of ids.
• The usage page's legend gives each line's total, not its last hour.
• The people page keeps an account on one line and its rarer actions in a menu.
• Every page fits a phone: a wide table scrolls in its own box.
• Static files are named by their content, so a deploy's new styles and scripts reach browsers at once instead of a day later.

One menu on every page. Each page drew its own menu, and no two agreed: different links, in a different order, the page you were on sometimes listed and sometimes not. Every signed-in page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the left for an administrator, Chat, Account and Sign out on the right for everyone — with the current page marked. The chat and key pages name the person's project by its path rather than its id.

The quotas page says what actually applies. A branch that sets no limit of its own used to read "—" in every column while its organization held it to sixty requests a minute. Each limit now shows the value it is held to from above, muted and marked ↑, with where it comes from on hover; a larger value set below a stricter one says what caps it, for rate and token limits as well as budgets. Spend and budget share a column with a meter that turns red past 90%. The tree is indented like the people page, opening a form or saving one no longer reloads the page, and the explanations moved under "How limits combine". Projects are placed under their team by id rather than by the start of their path, which misplaced them when two organizations shared a name.

The project's website, in every gateway

/about is llmgw's own page: what it does, InfiniBand and NVLink, installing, the reverse proxy configurations, and the release's files to download. It is served without a key like /, whose footer links to it, and says nothing about the installation. Its fonts are in the binary, so it makes no request elsewhere. The published website is rendered from the same template.

Metrics

llmgw_node_cpu_utilization_percent, llmgw_node_cpus, llmgw_node_load, llmgw_node_memory_used_bytes, llmgw_node_memory_available_bytes, llmgw_shard_cpu_cores, llmgw_shard_memory_bytes, llmgw_ib_port_active, llmgw_ib_port_rate_gbps, llmgw_ib_port_counter_total, llmgw_fabric_master_sm, llmgw_fabric_subnet_managers, llmgw_fabric_problems, llmgw_gpu_nvlink_joined, llmgw_imex_up.

Fixes

• PostgreSQL queries with an apostrophe in a comment failed: the placeholder rewriting read it as the start of a string. Comments are now skipped.
• Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
• Usage by team, project or organization, and the spend on the quotas page, left out every request made on the chat page: they were joined through the key, and the chat page has none. They now count where a request was spent.
• /org was answered with "no api key was provided" for a signed-in browser; a test now drives every session page through the authentication middleware.

Upgrading

Upgrade the control plane first, then the agents: an older agent is served as before and simply reports none of the new measurements. Existing agent units keep their old limits until LimitMEMLOCK=infinity is added to them. A node that set mgmt_interface keeps it; one that did not now uses the interface holding its management address rather than eth0. Multi-node vLLM replicas start with mp unless launch_mode: ray is set.

The store migrates on its first start. Migration 0010 turns everyone's project into a membership and attributes past usage to people where a key had an owner. On SQLite it rebuilds the keys table; it was run against a copy of a production PostgreSQL store before release. llmgw users move is gone: use join and leave.

vLLM is installed on every GPU node that can run it, once its agent is upgraded: about 8 GB each, from PyPI. Set engines.vllm: "" in server.yaml before upgrading to keep vLLM your own. deploy/bootstrap/onboard-node.sh is replaced by llmgw node add; it still works, for a machine holding the authority but not this llmgw.

v0.9.2 — 2026-09-28

A new control plane serves a model by itself

llmgw server init now also runs a model mirror on the same machine, makes the machine a node of its own control plane, and fetches Qwen2.5 0.5B into the mirror and serves it there — on the machine's NVIDIA GPU when nvidia-smi finds one (the node installs the CUDA build of llama.cpp), on CPU otherwise, with a CPU profile as the fallback. An NVIDIA card on the PCI bus with no working driver is reported. --no-mirror, --no-node and --no-model leave these out, and every step is skipped on a rerun when it is already done.

Not yet run end to end on a real machine.

v0.9.1 — 2026-09-28

The mirror page is one line per model

What the mirror holds used to take two table rows per file — seven columns and the address under them — so a sharded model filled the screen. Now each version is one line: model and version, the file (or "4 files" for a set), total size, and which catalog models use it. The address moved into a "copy address" button; the checksum is in the file's tooltip. A version already in use is not offered for serving again, and removing one warns that the catalog uses it. The header counts versions rather than files.

v0.9.0 — 2026-09-28

Managing people and the tenancy tree

Organizations, teams and projects can now be renamed, moved and removed, and an approved person can be moved to another project or removed — none of which could be done at all before, from the web or the CLI.

• /admin/users shows the whole tree, each project opening onto its people and keys, with approve, move, suspend, reinstate and remove for people, revoke for keys, and add, rename, move and remove at every level. A search over every account sits below it. Changes update the page in place (htmx, embedded in the binary) and it still works with scripts off.
• llmgw org — tree [--people], add, rename, move, remove.
• llmgw users move and llmgw users delete.
• llmgw keys list --all lists every key with its project and owner.
• Every command that takes a project, team or organization accepts an id, a path such as Acme/platform/gateway, or a unique name, and --project may be omitted when there is only one. Users are named by id or email.

Moving a person takes the keys they made with them; keys an operator issued stay with the project. Only empty things are removed, and a project that ever had a key is never removed, because usage history refers to it.

Changes

• --project no longer defaults to default in keys create, keys list, users approve, users create, keys threads and keys memory: a tree built from the web has no project of that name. With one project it is chosen; with several the command lists them.
• keys limits --budget is in whole units, as the quotas page takes it, rather than micro-units. --budget 50000000 now means fifty million units; divide existing scripts' figures by 1 000 000.

Fixes

• keys init could silently move an existing team or project to another organization or team by redeclaring it. It now refuses.
• An account rejected while pending and then reinstated became active with no project, and failed at its first key. Reinstating one is refused; approving gives it a project. Rejecting a signup now removes it.

v0.8.8 — 2026-09-28

Fixes

• server init --postgres failed on Rocky Linux with "postgresql-setup: executable file not found". The PostgreSQL project's own packages (PGDG) provide postgresql-server too, so dnf can answer that name with postgresql16-server, which names its unit postgresql-16.service and keeps its tools in /usr/pgsql-16/bin. init now finds the server unit systemd has, reads the data directory from it, and runs initdb itself when the directory is empty. A PostgreSQL that was already installed is used as it is.

Known gaps

Not yet run on a real machine. Three arrangements are not handled: a PostgreSQL initialised with ident for TCP logins (the connection check fails and says so), one listening on a port other than 5432, and several PGDG versions where the highest is not the one on 5432.

v0.8.7 — 2026-09-28

PostgreSQL in one command

sudo llmgw server init --postgres

Installs PostgreSQL with the machine's own package manager when there is none, creates the llmgw role and database with a generated password kept in /etc/llmgw/secrets/store-dsn, and checks that it connects. On a control plane already running on SQLite it stops the gateway, copies the store across so every account and key survives, points auth.store at PostgreSQL and starts the gateway again; if the move fails, the gateway is started again on SQLite. Given on the first run, there is never a SQLite store.

server.yaml is edited in place with its comments intact, and the previous file is kept beside it. A PostgreSQL that init did not install is left sized as it was, since something else may depend on it.

The PostgreSQL installation itself has not yet run on a real machine; the move, the configuration edit and the rerun behaviour are tested.

Fixes

• A new control plane with no models did not start. A configuration with no static endpoints and no models was refused as having nothing to serve, so under systemd the unit restarted every two seconds and nothing listened on 8080. It now runs as a control plane with an empty catalog.

v0.8.6 — 2026-09-28

Reverse proxy configurations

deploy/proxy/ has a Caddyfile and an nginx configuration for the client port, and both are attached to every release beside the binaries. They do not buffer streamed replies, their read timeout outlasts a model's wait before its first token, their body limit matches the router's, and they set X-Forwarded-Proto and X-Forwarded-Host themselves — llmgw marks session cookies Secure from the first and shows the second as the base URL on its landing page.

v0.8.5 — 2026-09-28

A control plane in one command

sudo llmgw server init --admin you@example.com

Creates the llmgw user and its directories, the certificate authority, the control plane's certificate for the machine's hostname and every address it has, server.yaml, the default project, and an administrator whose password is generated and printed once — then installs the systemd unit, starts it and waits until it answers. What used to be two steps of install.md is now this.

Each step is skipped when its result exists, so running it again finishes an install that stopped and changes nothing on one that did not. --host adds an address the machine cannot see of itself (a DNS alias, a NAT or floating address) and reissues the certificate keeping the old ones. It never overwrites server.yaml or an authority, and certificates issued by an authority kept on another machine are left as they are.

Fixes

• The configuration install.md showed could not start. An empty catalog without control.enabled: true is refused as having nothing to serve. The documentation and the file init writes both set it.

v0.8.4 — 2026-09-28

Node logs on the admin page

Each node on the cluster page links to its logs — the agent's own, and the output of every engine running there — with an option to follow them. llmgw ctl logs NODE reads the same. They come over the agent's existing stream, so no ssh, jump host or root is needed to see what a node did. The agent keeps its last 2000 lines in memory; each engine its last few hundred.

v0.8.3 — 2026-09-28

Found by building a node from nothing with v0.8.2.

Fixes

• A new node's first engine install failed on apt's lock. cloud-init and unattended-upgrades run apt in a machine's first minutes, and DPkg::Lock::Timeout does not cover the lock apt-get update takes. A package command held up by a lock now waits for up to ten minutes, and says so; a failed install is retried after a minute rather than ten.
• A new node reported no disk until its first download created the weights directory. It is created when the agent starts.
• Onboarding took one control plane address. It takes several, comma separated, so a new node fails over like the rest.

v0.8.2 — 2026-09-28

A node no longer runs whatever engine somebody once put on it. It installs its own.

Engines are installed by the node

engines: in server.yaml names the engines every node has and their versions, defaulting to the llama.cpp build this release was checked against (b10991). Each agent converges on it: it picks the build for its hardware — CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise — installs the OS packages that build needs with the machine's own package manager, downloads the archive and checks it against the sha256 GitHub publishes, unpacks it refusing anything that would land outside its directory, and runs it. ldd names a missing library or a system too old for the build ("libstdc++.so.6 is too old (needs GLIBCXX_3.4.30)"); --version proves the rest. Installs are checked again every ten minutes.

An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose. A node that has installed nothing yet is not a candidate for any model, where before it was a candidate for all of them. ctl nodes, the cluster page and placement's refusals say where each installation stands, and warn about a CPU without AVX2.

Not every system is refused for its name: the build runs on Ubuntu 22.04+, Debian 12+ and RHEL-family 10, and ldd decides. vLLM and SGLang are not yet installed this way and must still be named in the agent's configuration.

Onboarding

Onboarding installs llmgw from the CI release for the node's architecture, checked against the release's checksums, and writes an agent configuration of only the control plane's addresses and the TLS files.

Fixes

• A race on the weights fingerprint, which two heartbeat loops touched during a failover.

v0.8.1 — 2026-09-28

Nothing here changes a configuration format: every file that worked with v0.8.0 still works. What changes is how much of it needs writing, and how much of running the cluster can be done from a page rather than a shell.

A node describes itself

An agent file used to name its node, its address, the path to llama-server and its weight directories — different on every node and kept true by hand. The agent now finds these out: its id is the hostname, its address is the one on the route to the control plane, llama-server is looked for on PATH and in /opt/llama.cpp, a node without nvidia-smi simply has no NVIDIA cards, and fetched weights land in /var/lib/llmgw-weights, which is always scanned. Memory is re-read on every heartbeat and installed engines once a minute, so a VM given more RAM or a node given an engine becomes a candidate without an agent restart; the control plane logs both. The agent file shrinks to the control plane's addresses and the TLS files, the same on every node.

Models from the mirror, without writing YAML

Search the registry from /admin/mirror, fetch a sharded model in one action, and serve a mirrored version from its row — name, source, engine and an estimated memory requirement filled in. A model can say what it is for, and /v1/models shows it. Placement no longer hands the same memory to two models or puts an engine on a node that lacks it, and says when a replica runs an edited profile.

Tenancy, quotas and usage on pages

The tenancy tree, quotas and budgets are managed from the admin pages; an organization's budget bounds its teams, a group's quota counts against the group, and a budget that cannot be spent against says so. /admin/usage charts requests, tokens and spend over time by organization, team, project or person, and a request is priced by the model that served it.

Watching the cluster

The cluster page has its own charts, backed by a year of Prometheus history, with alerts that were made to fire before being trusted — including for the disk the metrics share with the models.

Chat

The picker offers each model once rather than once per alias, never asks for more tokens than the model allows, and offers a model that loads on demand — sending it a message is what loads it.

v0.8.0 — 2026-09-27

The store stops being a file on one machine's disk. PostgreSQL, a way to move an existing installation onto it without reissuing every key, and the two things a second control plane needs so that running one is safe rather than clever.

A second control plane, and a way to fail over to it

Leadership is a lease in the store, so two control planes against one database cannot both place work — which they otherwise would, both seeing a model wanting replicas and none running, leaving twice the replicas each holding GPUs the other believes are free.

A standby also refuses agent streams. Commands travel down the stream the agent opened, so a standby holding one is a node the leader cannot command, and the leader's placements would fail with "no agent connected" for a node that is plainly connected. Agents therefore take a list of addresses rather than one, trying each in turn: a standby refusing, a stopped control plane refusing the connection and a crashed one never answering are the same thing from a node.

Measured on the cluster. Leadership moved three seconds after the leader was stopped — it released the lease on the way out rather than letting it lapse — and all five agents had registered with the standby six seconds later. The standby adopted every replica rather than restarting any: six replicas, all ready, engines that never stopped serving.

Everything is unset-means-leader, so a single control plane and every deployment that existed before this behave exactly as they did.

What this does not do is move client traffic. The standby served inference correctly on its own port while the public address returned 502, because nothing in front of them moved — that needs a load balancer or a shared address, and it is the remaining piece.

PostgreSQL

The store runs on PostgreSQL, chosen by what the address looks like: a path is still SQLite, a postgres:// URL is not, and @ followed by a path reads the address out of a file so a DSN carrying a password stays out of a configuration that gets copied and pasted into issues. Every existing installation keeps working untouched.

One set of queries rather than one per database, because the SQL here is almost entirely portable and two copies would drift one query at a time. The rewrite of placeholders is a wrapper around the handle rather than a change at sixty call sites — which found six places reaching past it, including all three manual transactions and two prepared statements, every one of which worked on SQLite and would have failed on PostgreSQL.

Exactly one query is not the same sentence in both: the note search, MATCH against @@. SQLite gets full-text search from an FTS5 table kept in step by three triggers; PostgreSQL derives a tsvector from the row, so there is nothing to keep in step and no trigger to forget. It uses the 'simple' configuration rather than English — these are an agent's own notes, in whatever language it chose, and stemming English rules over Turkish would quietly stop finding things.

llmgw keys move-store copies an existing store rather than starting a fresh one, because keys are stored hashed: a migration that drops them cannot recreate them, and every person and every script would need a new one.

Testing it against a real database found three things nothing else would have. schema_migrations is created in Go rather than in a migration file, so it escaped the port and still said INTEGER — 64 bits in SQLite, 32 in PostgreSQL, and a millisecond timestamp does not fit. The copy order followed the order the migrations create tables in, and users is created after api_keys even though api_keys references it. And key_claims.sealed is emptied the moment a claim is used, so a claimed row holds an empty blob in a NOT NULL column — an empty blob scans as a typed nil slice and a real NULL as an untyped one, a difference pgx cannot see, so it arrived as NULL.

The driver is pgx, which is pure Go: the whole binary is built with CGO_ENABLED=0 so that it runs on a node whose libc is older than the builder's, and a cgo driver would take that away from every deployment, including the ones staying on SQLite.

v0.7.0 — 2026-09-25

What a cluster needs to be lived with rather than demonstrated: somewhere to keep a conversation, somewhere to keep a note, and an honest answer about the disk underneath both.

Memory notes for agents, and threads without API keys

Notes an agent writes for itself and reads back in later conversations, offered as REST (/v1/memory) and as MCP tools at /mcp (remember, recall, forget). A note is a key, text and tags, searched by words or tag, private to a project and to the agent id in x-llmgw-agent. llmgw does not inject the tools into chat requests: an agent that is not given the server never sees memory.

Off for every project until someone runs llmgw keys memory --retain-hours N, and the retention is required, as it is for threads.

On a gateway with no authentication, a threads: block (store, retain_hours) now keeps conversations and notes in one shared pool. Before, x-llmgw-thread was silently ignored there; now a gateway configured to keep nothing refuses it with an explanation instead. The shared pool is for one trusted machine: anyone who can reach the port and knows an id can read it.

Conversations that can move between models

A thread a client names with x-llmgw-thread, carrying only its new turn. llmgw replays the stored history to whichever model is serving, which is what lets a conversation begun on a large model continue on a small one — including when a spent budget moved it there and the caller never chose.

Off for every project until someone runs llmgw keys threads. Until then llmgw stores no message content anywhere, as it never has, and that default is worth more than the feature. The retention is required rather than optional, because a conversation store with no expiry is a transcript of everything a team ever typed kept forever by omission, and it is measured from the last turn so a conversation in progress does not expire underneath it.

History that does not fit the serving model is trimmed oldest-first and the reply says how many turns went, in x-llmgw-thread-trimmed. System messages are never dropped — losing one silently changes what the model is for while it carries on answering — and neither is the turn the caller just sent.

Threads are scoped to a project, and reading, writing or deleting one that belongs to another answers exactly as one that never existed: two answers would let a caller map which ids are real in projects they cannot read.

Capturing the reply is opt-in and bounded, so a request without a thread accumulates nothing on the hottest path in the system, and one with a thread stops at a cap rather than growing to it.

DELETE /v1/threads/{id} forgets a conversation, as a client rather than as an operator: the person who said the thing should not have to ask someone to read it before it can go.

Disk, and taking back only what llmgw put there

Disk was the one resource nothing reported. statfs was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space in the same struct was discarded, so the only way to find out why a node was filling up was to ssh to it.

Nodes now report it, per filesystem rather than per configured directory: free space belongs to the disk, and two model directories on one disk are one pool of bytes. Shared filesystems are kept out of a node's own figure, because space on a filer belongs to everyone mounting it and twenty nodes would otherwise each report the same free terabyte. It shows in ctl nodes, the dashboard and Prometheus, and a node that reported nothing reads as unknown everywhere — an older agent must not look like a full disk.

A profile may declare min_disk_gb, and a node without the room is no longer a candidate, with a reason explain-placement prints. Skipped on a node that already holds the weights: it needs no room for them, and without that the node best placed to run a model is the one most likely to be rejected.

ctl weights lists model files llmgw downloaded for models no longer in the catalog, and --collect deletes them. The design is mostly about what it will not do. Only files llmgw fetched are candidates, and that is a calculation rather than a heuristic — the cache name is a hash of the source, so the wanted set is computable exactly. Anything an operator put on a node is left alone however unreferenced it looks. Each node checks every path again before unlinking it, refusing anything outside its cache directory or without the cache name, so a control plane that is wrong cannot talk a node into losing a model that took a week to copy.

Fixes

• A capability was lost at an indirection. The controller is handed a forwarding dispatcher, and reaches past its interface by type assertion to find what lies beyond scheduling. Deleting weights was implemented on the agent service and not on the forwarder, so it was simply absent at runtime — with the whole suite green, because the dispatchers the tests use do not implement it either and its absence there is the expected case.
• ctl tool-check blamed the engine for what the model did. It had one verdict for a failed attempt and told operators not to try another model. Measured: the engine renders the tool prompt correctly, the coder models do not use the <tool_call> wrapper it asks for, and qwen2.5-3b-instruct on the same engine returns a populated tool_calls array. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper.

v0.6.0 — 2026-09-18

Phase 4: a replica can be bigger than one machine, and the fabric it spans decides where it is allowed to go. Around that, everything a cluster needs before anyone other than its operator is let near it — keys, budgets, a way in through a browser, and enough visibility to answer "why is nothing scheduling".

Multi-node replicas

A replica is now a set of shards with ranks, a leader and a bootstrap address, rather than one process. Two invariants are enforced rather than documented: a replica never spans two fabrics, and a tensor-parallel group never mixes GPU classes. Both are the kind of mistake that does not fail — it hangs, or it runs several times slower, and nothing says which.

A replica missing a shard is not ready. It was previously derived from the shards that had reported, so a two-node replica with one node up looked servable and took traffic into a model that could not answer.

Adoption after a restart happens one agent at a time, because that is how agents reconnect. Merging them as they arrive is what keeps a multi-node replica whole instead of being torn into fragments that are then killed as strays — and an adopted engine is health-checked before it is called ready, rather than being assumed alive because its process exists.

Fabrics are distribution domains too

An isolated fabric is not only a compute boundary: weights present in one are not present in the other. Node weight inventories are reported and summarised per fabric, so "the model is on the cluster" is never mistaken for "the model is reachable from the nodes that would run it". A fabric name that matches nothing a node actually has is refused at registration rather than becoming an unschedulable replica hours later.

Who may use it

API keys, tenants and per-model allowlists, on SQLite (modernc.org/sqlite, so still no cgo). Rate limits and budgets are enforced on the request path; usage is written off it, in batches, so accounting never adds latency to inference. Budgets survive a restart — counters are rebuilt from the store, because a limit anyone can clear by bouncing the process is not a limit.

A spent budget is now served a cheaper model where the catalog offers one, rather than refused. For a team trying models out, being dropped to something smaller is a far better answer than a wall, and the response says which model answered.

People can sign up, be approved, and issue and revoke their own keys. A key is shown once; only its hash is kept.

The browser surface

A landing page that says how to connect, a read-only cluster dashboard, and an in-browser chat that never puts an API key in the browser. The dashboard shows who is using the cluster and what they are running — per person and per model, with requests, tokens, errors, spend and latency.

All of it is served from the client-facing port behind a session, deliberately not from the admin port, which also serves pprof — and pprof can dump the heap that holds every API key in plaintext.

Models without a restart

The catalog is live. Models can be added and removed from the command line or the API, take effect immediately, and outlive a restart; configuration still wins over the store, and the whole catalog is validated before anything is written.

A model is only listed once it can actually serve. It was listed as soon as it was known, so the first thing a new user did was ask for a model that returned 503.

Loading on demand, and letting go

A model whose scaling allows it is started when someone asks for it, and stopped when nobody has for a while. An idle replica is evicted to make room for one that was asked for, with guards that exist because each is a way to lose work quietly: only on-demand models, only single-node replicas, never one that has never been used, and never one younger than its minimum lifetime.

Unknown is not idle. A replica with no recorded use is left alone rather than treated as idle since the epoch.

Knowing whether it will run before it runs

• ctl plan works out whether a model fits the cluster and how it would be split, in the precision the chosen engine can actually serve — recommending a quantisation the engine cannot load would cause the failure the command exists to prevent.
• ctl tool-check verifies llmgw's engine flags against the engines actually installed. Flags drift between releases, and the alternative is finding out during a model load.
• Placement now knows a model's context length and memory footprint, and node RAM, so a CPU profile is not placed on a machine that cannot hold it.
• Placement waits for the fleet to settle after a restart rather than deciding against a half-reported cluster.

Weights

Nodes report the model files they can see, without llmgw owning the storage: a shared filer, an image exported over the fabric, rsync or a copy made by hand all look the same to placement, which only cares whether the files are there. Reporting nothing means "not asked to look", not "has nothing".

Where the simple arrangement is not enough, a node can fetch a model by URL — single-flight, resumable, written to a temporary name and renamed atomically, so a killed download never leaves something that looks like weights.

Seeing the fleet

Prometheus metrics for nodes, GPUs and replicas, collected from live state at scrape time rather than from a background goroutine, and served on their own endpoint rather than through the one that also exposes pprof. The same view is in the dashboard, so nothing else needs installing to see it.

Onboarding a node

Adding a node is one command. There are systemd units, a private CA and a from-nothing install guide that starts with a machine that has no CUDA and no binary.

llama.cpp, and CPU profiles

A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI API without a GPU, so placement, supervision, scraping, scheduling and admission can all be exercised on ordinary machines.

A profile may now omit gpu_class, which is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it — for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. CPU replicas are spread across nodes rather than bin packed: unlike GPUs there is no allocation to stop two of them sharing a machine's cores.

Metric mappings are per engine

Running, Waiting and KVCacheUsage now have a vocabulary per engine rather than one for the cluster. A mixed fleet has two names for the same three numbers, and a single global mapping left half of it unmeasurable.

A mapping that leaves a metric unnamed now means the engine does not publish it, and the field stays zero. Naming one the engine does not serve is still a contract violation. Running and Waiting can never be omitted.

Mutual TLS on the agent stream

A private CA (llmgw ctl cert init-ca | issue-server | issue-client | inspect), certificates on both ends, and the control plane refusing any agent whose claimed node id does not match its certificate's common name. Public authorities cannot supply these: they stopped issuing the clientAuth usage.

Plaintext HTTP/2 remains the default. agent_tls: on the server and tls: on the agent turn it on.

Fixes

• An agent told to stop did not stop. Cancelling the context is not enough — HTTP/2 watches it while the request body is being written, and a bidirectional stream spends its life blocked on the read side. Measured on a deployed node: systemctl stop took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
• An agent could not explain why it was refused. The session cancelled its own context and then asked that context whether it was shutting down, so every stream error read as a clean close: no message, and the backoff reset.
• A broken metric contract was invisible. It was logged per scrape at debug level, so a cluster could look healthy while every routing decision was made blind. It is now a warning, once, with a matching line when it clears.
• Token estimates came from the routing prefix, not the prompt. Every prompt over about 2 KB was undercounted, so tokens-per-minute limits let through far more than they were set to.
• Model-fitting double-counted overhead. Whole-model overhead was charged per rank and a 10% reserve taken twice, so plan reported that a 70B model in bfloat16 does not fit two 80 GB cards. It does.
• x-llmgw-served-model was never sent. It was set after the proxy had already written the response header, which is the one point at which headers stop being changeable. A client had no way to tell a fallback from the model it asked for.
• Fleet metrics published nothing. They were registered after the registry had been built, so the collector existed and was never scraped.
• GPU discovery broke on older drivers. A field added for utilisation was required rather than optional, so a driver that does not report it produced no inventory at all instead of an inventory without that number.
• Replica ids and ports were reused while the processes holding them were still alive.

v0.5.0 — 2026-09-16

Phases 2 and 3. The cluster runs itself: agents report their hardware, the control plane places replicas, and ctl makes all of it visible.

This release covers both phases because their exit criteria depended on the same piece — the agent stream — and neither was demonstrable without it.

ctl

llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain

• nodes surfaces unhealthy GPUs and why they are excluded, which is usually the answer when a node looks idle but nothing schedules onto it.
• explain-placement prints every candidate considered and the reason it was rejected. "Unschedulable" on its own says nothing about which of six constraints bit.
• drain is a rolling move, not a stop: the replica is drained, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. Bounded by a grace period, because draining the only node that can host a model would otherwise never finish and leave it undrainable.
• cordon is the weaker operation: no new placement, nothing disturbed.

The management API is on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes.

ctl logs is not here. It needs a request/response pair the agent protocol does not have. The failure path already ships an engine log tail with every status report, so the case that matters most is covered; reading a healthy shard's output on demand is not.

Phases 2 and 3, in detail

Written as the work landed, which is why it reads as a log rather than as notes. Everything here shipped in v0.5.0.

Added

• IB discovery (internal/agent/discovery): reads /sys/class/infiniband, parses the formats sysfs actually uses ("4: ACTIVE", "200 Gb/sec (4X HDR)", hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node without IB is not an error — it is a single-node island.

• GPU discovery via nvidia-smi, the cgo-free path. NVML needs cgo, which would force the whole binary, server included, to be built with a C toolchain. Health flags anything that should keep a card out of placement: uncorrected ECC errors, and memory already held by another process.

• GPU class derivation. Sizes come from the driver in GiB, which does not always match the marketing name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g. Consistency matters more than matching the box, because the number decides whether a model fits.

• Engine adapters for vLLM and SGLang: command and environment construction, kept free of process handling so it is testable without GPUs. vLLM supports both Ray and multiprocessing launch modes, because which one works depends on the version.

• NCCL environment generation, the part plan §5.3 marks critical. NCCL_IB_HCA lists only the chosen fabric's adapters — leaving it unset lets NCCL pick an adapter on another fabric where the peer is unreachable, and the job hangs rather than failing. Bootstrap is pinned to the management interface for the same reason.

• Validation before launch: a tensor-parallel group may not mix GPU types, the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below compute capability 8.9 rather than silently falling back to a slower path.

• Supervisor (internal/agent/supervisor): processes run in their own process group and are stopped as a group, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader. Bounded log tail, a state machine that separates Starting from Loading (a 70B model takes minutes, and an operator needs to know that is normal), and orphan adoption through pid files fingerprinted with the command line — pids get reused, and adopting an unrelated process is worse than leaking the original.

• Readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a model that is still loading.

• llmgw agent inventory reports what the scheduler would see on a node, as a table or JSON. Run it before adding a node: it is the fastest way to find a card another process is using, or a port with no subnet manager.

• NVML backend behind the nvml build tag, which adds NVLink topology. Nothing else exposes which GPUs share an NVLink domain, and that is what keeps a tensor-parallel group off the slow path inside a node. Discover falls back to nvidia-smi if NVML fails, and says so loudly rather than silently losing the topology. CI compiles the tagged build so it cannot rot unnoticed.

Not verified

The NVML path compiles but has never run against a driver — there is no GPU machine to try it on. Treat the NVLink domains it reports as unverified until someone runs llmgw agent inventory on a real node.

• Control plane wiring (internal/control): Ready replicas become router endpoints, with no static configuration involved. The router keeps live state for endpoints that survive an update — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes this becomes a watch stream and nothing above it changes.

Phase 3 exit criterion, measured in test/e2e

A node hosting a live replica is killed while four clients stream continuously. The control plane marks it unreachable, then lost, rebuilds the replica on a survivor, and republishes. Across three runs: ~370 requests completed, zero failures, and no replica remained on the dead node.

This proves the control logic, not the transport: the agent fleet is simulated, and the real path still needs the stream.

A real bug the race detector caught

startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock. Both were racing with agent status reports, which arrive from another goroutine and mutate exactly those fields. Under the simulated fleet the reports came back fast enough to hit the window every time. Everything is snapshotted under the lock now and the functions work from values.

The agent stream

• Protocol (api/proto/llmgw/v1): one bidirectional connect-go stream carrying registration, heartbeats and shard status upward, and start, stop and drain commands downward. The agent dials out, because GPU nodes sit behind firewalls far more often than a control plane does and an outbound connection needs no inbound rule on every node in the fleet.
• Registration carries the whole inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it — they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement.
• Agent service (internal/control/agentsvc) is both the stream handler and the reconciler's dispatcher: it holds the open streams, so it is the only thing that can deliver a command to a node.
• Agent (internal/agent) validates a profile before launching anything — an engine that refuses a profile does so after minutes of loading — then supervises the process and reports every transition, including the engine's output tail on failure.
• Reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock the control plane over again. Engines keep serving throughout: the router talks to them directly.
• llmgw agent is real, and llmgw server runs the control plane whenever any model declares a profile. The static endpoint list remains for router-only runs against sim-engines.

Verified with real binaries

Server and agent as separate processes over a real stream: the agent registers, a replica is placed, an engine is launched through the vLLM adapter's actual argument construction, streaming traffic is served through the router, and killing the agent moves the node through unreachable to lost with placement explaining exactly why it can no longer run anything.

Surviving a control plane restart

Not persisting replica state left one hole that had to be closed before it mattered: restarting the control plane orphaned live engines. They kept holding GPUs the inventory believed were free, so the next placement failed to allocate with an error naming the new process rather than the old one.

The fix is on the agent, not in a database, and it cannot disagree with reality the way a stale row can:

• The supervisor now writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica.
• Adopt takes over a still-running engine from a previous agent. It cannot be waited on — it is not this process's child — so it is watched by polling, and stopped by signalling its group directly. Its earlier output is gone, and the log says so rather than presenting emptiness as silence.
• Every Register, not only the first, reports what the agent is running. A reconnect may be a control plane restart and has to recover the same way.
• The reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped: nothing will route to it and it is holding cards.
• An engine the agent cannot identify — a corrupt state file — is killed. A process holding GPUs that nothing can describe is worse than no process, because placement will hand those GPUs out again.

One mechanism covers agent restarts and control plane restarts.

A test that was passing for no reason

TestAgentReconnects used httptest.CloseClientConnections, which does nothing to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops tracking hijacked connections. The agent never reconnected and the assertion — on the connection map, which still held a stale entry — passed anyway. Tests now sever connections through their own listener and count opened streams.

Multi-node launch

Placement and environment generation had landed early, which made multi-node look further along than it was: a multi-node replica could not come up at all. Three things were missing, and all three came down to two assumptions that do not hold once a replica spans nodes.

Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers. Workers timed out, reported Failed, and took the replica with them. Engine.ServesHTTP(spec) now says which ranks serve; a worker is judged by staying alive instead. That is weaker evidence on purpose — the replica's real readiness is the leader answering, which only happens once every worker has joined.

A shard is not always one process. Engine.Commands(spec) returns a sequence: every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's exit. vLLM under Ray is why — the leader runs a Ray head and then the server against it, and a worker runs only the join, with no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist.

A command can also require the rendezvous point to be listening before the next one starts, bounded by a timeout so a head that never appears fails the shard rather than hanging it. Sidecars are killed with the shard: a Ray head left behind holds the port the next replica needs.

Start order was backwards. Workers were started first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to and spends its startup budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own.

None of this is verified against real engines. It is tested with stub processes; the Ray sequencing and which launch mode a given vLLM version wants can only be confirmed on hardware.

Still missing for phase 2

• mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted network.

Phase 3: placement

• Cluster inventory (internal/control/inventory): nodes, their hardware, their fabric attachments, and what is allocated where. A re-registering agent cannot clear an operator's cordon, and a heartbeat revives an unreachable node without doing so either. A GPU with an ECC fault or another process on it is not counted as free. Replicas are released whole, because they fail whole.
• Placement (internal/control/placement): picks the highest-priority profile that fits and explains every candidate it rejected. Two constraints are enforced structurally rather than checked afterwards — there is no code path that can combine nodes from two fabrics, and only GPUs of the profile's class are ever considered.
• Bin packing fills partly used nodes first, so whole nodes stay free for large models that cannot be split. Multi-node replicas get the opposite treatment: they are the least movable thing in the cluster, so they take the emptiest machines.
• NVLink-aware GPU selection keeps a tensor-parallel group inside one NVLink domain where the topology is known. Without the NVML build it falls back to the lowest free indices, which is usually right but only by accident of how vendors number slots.
• Property-based invariant tests (plan §13): 400 randomly generated clusters, checking on every placement that a multi-node replica lives on one fabric, that each shard receives only that fabric's adapters, and that no placement mixes GPU classes or uses an unhealthy or already-allocated GPU. The table-driven tests cover the cases someone thought of; this covers the ones nobody did.
• Catalog (internal/control/catalog): models, profiles, aliases, fallbacks and scaling bounds. Aliases and fallbacks are resolved at load rather than at request time — a typo in a fallback should stop a deployment, not surface during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load instead of minutes into model loading.
• Reconciler (internal/control/reconciler): the desired-versus-actual loop. It places one replica per pass, because placement depends on what is already allocated and starting several from one snapshot would hand them the same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves.
• Replica failure is whole-replica failure. One dead shard makes the others useless — they wait on collectives that will never complete while holding GPUs — so the reconciler stops every shard and rebuilds, with a jittered exponential backoff carried on the model rather than the replica it just destroyed.
• Node expiry in two stages. Unreachable first, Lost later. A brief network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading.
• Scale-down drains before stopping, so in-flight requests finish instead of being cut off mid-stream.

v0.3.0 — 2026-09-16

Flow control. A batch workload no longer starves interactive traffic.

Added

• Admission control (internal/router/admission), inserted between quota checks and scheduling. It answers when a request may proceed; the scheduler still answers where it goes. Keeping them apart is the point: an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no amount of endpoint selection rescues the request behind it.
• Saturation detector. One number per pool, the worse of queue depth and KV occupancy. Either signal alone misleads — a pool can have an empty queue and a nearly full cache, or the reverse.
• Priority bands. Each band has a saturation ceiling; above it, that band waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means "flow up to the healthy limit"; above 1 means never wait.
• Tenant fairness. Within a band, dispatch turns rotate between tenants, so one caller sending a thousand requests occupies one slot in the rotation rather than a thousand.
• Bounded queues. Capacity per band (immediate 503 when full) and a TTL (504), because sending the GPU work the client has already abandoned helps nobody. Cancelled requests leave the queue instead of holding capacity.
• Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total, llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry x-llmgw-queue-ms when a request waited.

Phase 1.5 exit criterion, measured in test/e2e

While 500 concurrent bronze requests run continuously, interactive p99 in the platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget. With admission switched off the same load pushes it to 55–60x. Both directions are asserted, so the test fails if admission regresses and if the load generator ever stops being a real load.

Two lags, two different fixes

Getting this right took longer than the queueing itself:

• Engine metrics are scraped, so they lag by an interval. The router's inflight count does not — but it includes running requests, and treating those as queue depth makes a healthy engine look overloaded every interval. What is genuinely queued is inflight − batch capacity, and batch capacity is not in the metric contract, so it is learned as the high-water mark of observed running.
• Between admission granting a request and the router dispatching it, the request is invisible to every signal. A simultaneous burst therefore passed the check en masse — exactly the batch arrival pattern this layer exists to survive. The controller now counts its own outstanding leases against dispatched inflight.

Known limitations

• The band comes from a trusted header until authentication arrives in v1.0.0. Anyone who can reach the port can promote themselves to platinum.
• The tenant key is the session id, for the same reason.
• Displacement (dropping queued low-band work to make room) is not implemented; the plan defers it, and bounded queues plus TTL cover most of the need.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

• Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
• Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
• OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
• Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
• Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
• Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

• 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
• Every client disconnecting frees the KV blocks on every engine.
• Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

• No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
• The endpoint list is static; the control plane arrives in v0.5.0.
• No authentication, rate limiting or budgets yet (v1.0.0).
• Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

• continuous batching with a bounded concurrent sequence count
• KV cache block accounting, queueing when blocks run out, including head-of-line blocking
• automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
• per-token latency that grows with batch size
• dropping a request and freeing its blocks when the client disconnects
• /metrics under vLLM's real metric names
• failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

• router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
• router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
• router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
• router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

• llmgw server, llmgw agent and llmgw ctl are stubs.
• The scheduler has no HTTP layer yet; nothing routes real requests.
• No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
• Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

• The project language is English throughout: code, comments, tests, commits and the design document.
• Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.