llmgw/docs/install.md

Installing llmgw from nothing

Written for whoever is standing up the cluster: you have servers, you have root on them, and nothing is installed yet. For only the commands, read quick-install.md first; this page explains them. By the end there is a control plane, a node in the fleet, and a model answering requests.

It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond the package names. Everything here has been done on a real cluster except where this page says otherwise, and where it says otherwise it says so plainly.

What llmgw does not install

Read this first, because it is the difference between an afternoon and a week.

llmgw installs itself, its engines and its store. It does not install the GPU driver or the InfiniBand stack: those are kernel modules with version matrices attached, and a platform decision. A node missing them still registers and reports ready, and fails every replica that needs the GPU or the fabric — so make a node able to serve before onboarding it. llmgw doctor on the node says what is missing and the command that fixes it.

You provide	llmgw provides
NVIDIA driver and nvidia-smi	GPU discovery, classes, health
InfiniBand stack (rdma-core/OFED)	Fabric discovery and the invariant that a replica never spans one
Nothing — the nodes install them	llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: Engines
Which models to run	The mirror that fetches them once, placement, routing, accounting

Weights are handled too. llmgw includes a mirror — step 3 — which holds the files on your own network and hands them to the nodes, so getting weights onto every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you.

0. Before you begin

Four machines is a reasonable first cluster: one control plane and three nodes. One of each also works.

The control plane needs no GPU: one machine, holding its store in SQLite or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable from every node on the agent port, and from your users on the client port.

Nodes need whatever the model needs, plus about 8 GB for the operating system and the agent. They dial the control plane; the control plane never dials them, so nodes need no inbound access except your own SSH.

Ports

Port	Who connects	Expose to
8080	clients, the web UI	your users, usually via a reverse proxy
7443	node agents	the node network only
9090	llmgw ctl, the management API, pprof	loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap
9091	Prometheus, if you run it	your monitoring host; metrics only, nothing else is served here
8081	node agents fetching weights	the node network only (the mirror machine)
8082	the control plane administering the mirror	the control plane only — this port can delete models and start downloads

Decide now which address nodes will use to reach the control plane. It goes into a certificate: the machine's own addresses are found for you, and any other name — a DNS alias, a NAT or floating address — has to be given with --host.

1. The binary

One static binary, no runtime dependencies, the same on every machine.

VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL  $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version

arm64 and darwin builds are published alongside. They are built with CGO_ENABLED=0 deliberately, so a binary built today runs on a node whose libc is older than the builder's.

To build instead of download, you need Go 1.26.6 or later and nothing else (an older Go fetches it itself):

git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw

2. The control plane

The short way is the screens. On the machine that will be the control plane, run sudo llmgw with nothing after it. With nothing installed it asks whether this is one machine or three for high availability; for three, it asks for the other two — address, user, and an ssh key, the ssh agent, or a password, and the sudo password if sudo asks for one; or, for a machine nothing may log into, a join code to run there with sudo llmgw join <code>, which dials out to this one on port 7444 — checks all three before installing anything (system, room, clock, ports, internet, whether they reach one another), shows the plan, and installs. Passwords are held in memory for the run only; a sudo password lets the user sudo without one until the install finishes, and is then taken back. At the end it shows the administrator's password and the command that does the same from a script.

Run again later, sudo llmgw is the console: the cluster's state, refreshed, and what to do next — add a node, replicate the store. A setup that stopped halfway opens the setup screens again, which finish it.

The rest of this section is the same by command, as root:

sudo llmgw server init --admin you@example.com

That is the whole step. In order, it:

1. creates the llmgw system user, /etc/llmgw and /var/lib/llmgw
2. creates the cluster's certificate authority in /etc/llmgw/pki
3. issues the control plane's certificate for this machine's hostname and every address it has, and copies ca.pem beside it
4. writes /etc/llmgw/server.yaml: mutual TLS, authentication, default limits and a price, keys that expire after at most 90 days, the baseline guardrails policy, and a control plane with an empty catalog
5. creates the store, the default project, and your account, already approved and promoted to administrator — the password is generated and printed once
6. installs the systemd unit, starts it, and waits until /health answers
7. runs a model mirror beside it (/srv/models, bytes on :8081, its management API on loopback) and points the control plane at it
8. makes this machine a node of its own control plane — on its GPUs when nvidia-smi finds any, on CPU otherwise — and waits for it to install its engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs two replicas on the CPUs of nodes with room — this one, or the first that join
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the first thing anyone opens /chat to find is a model that answers. A machine without room for it is told so within two minutes, and why.

--no-mirror and --no-node leave out 7 and 8, --no-firewall 9, --no-guard-model 10 and --no-model 11. An NVIDIA card with no working driver is reported rather than silently ignored; install the driver and run init again to serve on it.

Secure by default. What init writes is the hardened configuration, not a demonstration one:

Agents	mutual TLS with the cluster's own authority; the agent port refuses plaintext
Admin API	loopback only, behind a token; pprof off
Store	between machines, TLS everywhere (PostgreSQL, etcd, Patroni)
Firewall	llmgw's internal ports answer the cluster's networks only
Accounts	signup closed; administrators create accounts or approve them
Keys	at most 90 days (auth.max_key_ttl), shorter per part of the tree with llmgw keys ttl
Requests	every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind
Guardrails	the baseline policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers

The judge runs as two replicas, each on a CPU — about half a second for a short text on any CPU with AVX2 — and reads a streamed answer every 600 characters. The policy fails closed: a request the judge cannot read within 5 seconds is refused with 503 rather than served unjudged, and judge_unavailable goes to the notification webhooks. Two replicas are what make that bearable; on one small machine, or a VM without AVX2 (see step 6), the judge is the first thing to fail, so fail: open — serve it, recorded as flagged — or --no-guard-model is the choice there. Defamation, specialized advice, intellectual property and elections are left out of what it refuses: they catch too much ordinary work. guardrails.md has the rest.

Two things are left to you because they need something only you have: a certificate for the client port — port 8080 is plain HTTP; put the proxy in deploy/proxy/ in front — and the keys backups are encrypted to (backup_recipients, see Backing up). An NVIDIA card with no working driver is reported rather than silently ignored; install the driver and run init again to serve on it.

Add --postgres to keep accounts and keys in PostgreSQL rather than SQLite: init installs it and sets it up, and on a machine already running on SQLite it moves everything across. See PostgreSQL.

Or --patroni, given two machines (or more), to keep it in PostgreSQL replicated across them, so losing one loses nothing:

sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni ubuntu@10.0.0.149 --patroni ubuntu@10.0.0.150

Over ssh with passwordless sudo, as node add works, it installs PostgreSQL, Patroni and etcd there — this machine is etcd's third vote — and moves the store already in use, SQLite or PostgreSQL, into the cluster. A standby is promoted within about thirty seconds of the primary being lost, and the control plane follows it by itself. llmgw ctl status says who leads; see deploy/ha for what it sets up.

Everything between the machines is TLS, with certificates from the cluster's own authority: PostgreSQL (every remote rule is hostssl, and the control plane verifies the member it reaches), etcd (peers and clients, and a client must present a certificate), and Patroni's API, which also asks for a password before a change. A cluster set up by an earlier llmgw is moved to this by running the same command again, on the control plane that holds the authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so PostgreSQL serves throughout and no failover happens.

Every step looks before it acts. Running it again finishes an install that stopped halfway and changes nothing on one that did not; it never overwrites server.yaml, the authority, or an existing account's password.

The addresses in the certificate matter. Agents check that the address they dial is named in it, and the certificate names only what this machine could see of itself. If nodes will reach it by another name — a DNS alias, a NAT address, a floating IP — give it:

sudo llmgw server init --host llmgw-ctl.internal --host 203.0.113.10

Run that at any time: the certificate is reissued with the new names added to the old ones, and the service restarted.

ca-key.pem signs every node's identity. init leaves it in /etc/llmgw/pki, because onboarding a node needs it and the control plane is where most people run onboarding from. Back it up; there is no recovering it. If this machine is exposed, move the directory elsewhere once the nodes are in and pass --pki to onboarding.

This covers the fleet-facing port only. The client-facing API is public and belongs behind an ordinary publicly trusted certificate, terminated by a reverse proxy in front of port 8080. deploy/proxy/ has a Caddyfile and an nginx configuration that get streaming, timeouts and the forwarded headers right; both are attached to every release.

Check it

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200
sudo llmgw ctl status

ctl status will report no nodes and no models, which is correct — nothing has been added yet. Sign in at http://<control-plane>:8080/login with the printed password, change it, and issue yourself a key from the profile page.

The configuration

/etc/llmgw/server.yaml is yours from here on. Three things in it are worth knowing before you edit it:

• admin_listen is on loopback on purpose, and the server refuses any other address unless admin_remote: true. Its API under /api adds models and upgrades nodes, so it also wants a token: the server writes one, 0600, to /var/lib/llmgw/admin-token the first time it starts, and llmgw ctl reads it from there — which is why llmgw ctl runs with sudo. LLMGW_ADMIN_TOKEN gives it from elsewhere. pprof, whose heap holds the API keys requests presented, is off unless debug_pprof: true, and then behind the token too. To see the dashboard from elsewhere use /admin on the client port — do not move this listener. The API itself is described in admin-api.md.
• default_rpm, default_tpm and default_price are what make limits and budgets mean anything. Without a price every request costs zero and a budget never binds; without the defaults, a project created without limits is metered by nothing at all.
• control.enabled: true is what makes this a control plane while its catalog is empty. Without it, a file with no models is refused as having nothing to serve.

Every key is in reference.md.

Keeping the authority off the control plane

Create it on another machine, issue the control plane's certificate there, and bring only the results:

# on the authority's machine
llmgw ctl cert init-ca --dir /etc/llmgw/pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.140 --host llmgw-ctl.internal \
  --dir /etc/llmgw/pki --out ./ctl

# copy ca.pem, ctl/server.pem and ctl/server-key.pem into /etc/llmgw on the
# control plane, then there:
sudo llmgw server init --admin you@example.com

init finds certificates in /etc/llmgw and no authority beside them, and keeps them as they are. Adding an address later is then done where the authority is; init --host says so rather than minting a second authority no node trusts.

3. The mirror

Do this before the nodes, not after. Every node fetches its own copy of a model's weights, and where they fetch from decides whether bringing up a cluster takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging Face is a day's work, and on a good number of corporate networks it is not possible at all — the nodes have no route out, which is usually the point.

The mirror is one machine holding the files on your own network. It is the same binary again, and it is the highest-value piece of infrastructure around llmgw for the least work.

Running it

sudo install -d -o llmgw -g llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081

In production, the unit:

sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror

Two listeners, on purpose. :8081 serves bytes and nothing else — that is what every agent talks to, and it is plain HTTP because a weights fetch is a public file transfer that is checksummed anyway. :8082 is the management API: it can start downloads and delete models, and only the control plane needs it. The split is what lets you lock the second one down without re-certifying every agent that fetches from the first.

Give the management port the cluster's own CA and it takes client certificates — the control plane's, named with --api-client, and no other. The same CA signs every node's certificate, so the name is required: without it any node, a removed one's included, could delete models.

llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
  --tls-ca /etc/llmgw/ca.pem \
  --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem \
  --api-client control-plane

Without those flags the management port answers this machine only, and the addresses given with --api-allow.

The layout

/srv/models/
  qwen2.5-14b-instruct/
    2026-03/
      q4_k_m.gguf

Model, then version, then files. The version directory is the point of the layout rather than decoration: a repository that changes upstream changes what you are running, and the first sign is a prompt that worked last week answering differently this week with nobody able to say why. Pin it here and that cannot happen.

A version holding several files is a sharded model, and is served as a set: the agent asks the directory what it consists of and fetches all of it into one place, marking it complete only when the last file lands. A catalogue entry for one names the directory, trailing slash included.

Telling the control plane about it

# /etc/llmgw/server.yaml
mirror:
  api: https://models.internal:8082
  tls:
    ca: /etc/llmgw/ca.pem
    cert: /etc/llmgw/server.pem
    key: /etc/llmgw/server-key.pem

With this set, /admin/mirror appears: what the mirror holds, what it is downloading and how far along, a search of Hugging Face, and a form that puts a mirrored model into service without anyone writing YAML. Without it that page stays unmounted and the mirror is still perfectly usable — a model's source is just a URL, and the URL can be yours.

If the control plane cannot reach the internet, mirror.registry points the search at an internal registry mirror instead.

Filling it

llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
  hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf

or from the admin page, or by copying files into the directory by hand — the mirror is a directory, and anything that puts files in it works.

Files already on the mirror machine go in with import, which copies them the way a download is kept — under a temporary name until whole, the checksum beside them for nodes to check against, a version never replaced — and checks that a .gguf is one:

llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
  --version 2026-10 --sha256 <the checksum its publisher gave>

From Ollama

Models you already run in Ollama are GGUFs llama.cpp serves as they are. On the mirror machine, with Ollama's models readable there:

llmgw mirror import-ollama                 # lists what Ollama has
llmgw mirror import-ollama llama3.2:3b

It finds Ollama's models directory where Ollama keeps it — OLLAMA_MODELS, ~/.ollama/models, or /usr/share/ollama/.ollama/models for Ollama's Linux service, which needs sudo to read — or takes --ollama-dir. It copies the model's GGUF, and a vision model's projector beside it, each checked against the sha256 Ollama named its file by; a file that does not match is not mirrored, and nothing of the model is. The version is the tag and the start of the digest (llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf), because a tag like latest moves in Ollama and a mirror version must not; --model and --version choose otherwise. It ends with a model definition to save and llmgw ctl model add -f, with --ctx-size set to the num_ctx Ollama ran it with.

Ollama's prompt template is not carried: llama.cpp renders the chat template inside the GGUF, which for any recent model is the one it was trained on. Its system prompt and adapters are not carried either, and the import says so. Models Ollama keeps as separate tensors rather than one GGUF, or as several weight layers, are refused with a sentence saying so — mirror that model's GGUF from its publisher instead. When the mirror runs beside the control plane, the console's Mirror view does the same with I.

4. Preparing a node

Do this before onboarding. A node missing its driver or engine registers healthy and fails on first placement, and the reason is three logs away.

The driver

sudo apt-get update
sudo apt-get install -y nvidia-driver-550   # or whatever your cards need
sudo reboot
nvidia-smi                                   # must print your GPUs

nvidia-smi must be on PATH for the user the agent runs as, which is root. Containerised driver installs often put it elsewhere; if so, note the path and set discovery.nvidia_smi in the agent configuration rather than moving it.

A node with no GPU at all is fine and is not an error. It reports zero cards and can host CPU profiles, which is a good way to try llmgw before committing hardware to it.

NVLink, if your GPUs have it

The agent reads the topology from nvidia-smi topo -m: which cards NVLink joins, each card's NUMA node, and which network adapter is nearest each. A tensor-parallel group is then the set of cards with the most NVLink between them — on a PCIe box where bridges pair specific cards, a group straddling two bridges falls back to PCIe and runs several times slower, and nothing else would report it.

On an NVSwitch machine, nvidia-fabricmanager has to be running, at the driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an NVLink Switch system, the GPUs of several nodes form one domain, and a replica can run tensor parallelism across its nodes — which needs nvidia-imex and an IMEX channel as well. fabrics.md has the list, and llmgw doctor checks it:

sudo llmgw doctor

A -nvml build is still published beside the portable one; it reads NVLink from NVML directly, and needs a libc no older than the builder's.

MIG, if you want to cut a card into pieces

An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance GPU) instances, each with its own memory, its own share of the card's compute and its own UUID. llmgw places a model on an instance as it would on a card, which makes MIG the way to run several small models on one large card without them competing for it.

llmgw does not create the instances. How a card is cut is your decision, made once with nvidia-smi on the node; the agent reads what is there when it starts:

sudo nvidia-smi -i 0 -mig 1                 # MIG mode on card 0; may need a reset
sudo nvidia-smi mig -i 0 -lgip              # the profiles this card offers
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C   # two instances, each with its compute instance
nvidia-smi -L                               # the instances, with their MIG- UUIDs
sudo systemctl restart llmgw-agent          # so it registers the new devices

-C matters: a GPU instance with no compute instance inside it is not a device yet. Turning MIG on takes effect at the card's next reset (nvidia-smi -i 0 -r with nothing running on it, or a reboot), and instances do not survive a reboot unless something recreates them — NVIDIA's mig-parted or a systemd unit of your own.

Classes. A card in MIG mode is reported as its instances instead of as the whole card, and each instance's class is the card's class and the profile: two 3g.40gb instances of an A100 80GB are two devices of class a100-80g-mig-3g.40gb, an H100's 1g.10gb+me is h100-80g-mig-1g.10gb-me. llmgw ctl nodes lists the classes, and a profile names one like any other:

profiles:
  - name: mig
    engine: vllm
    gpu_class: a100-80g-mig-3g.40gb

One instance a replica. A process can open only one MIG device, and instances have no NVLink or peer-to-peer access between them, so a replica on MIG runs on exactly one instance: tp: 1, one GPU, one node. A profile on a MIG class asking for more is refused when the model is added, with that reason. A model too large for one instance wants a larger profile or a whole card. llmgw ctl model plan and ctl model serve plan the same way: an instance is offered for a model it holds, and never several together.

How they are numbered and shown. Placement numbers card 0's instance 2 as device 1002 (1000 + 100 × card + instance), and every view shows it as 0/mig2. The engine is given the instance by its UUID — CUDA_VISIBLE_DEVICES=MIG-… — which works for vLLM and for llama.cpp's CUDA build. llama.cpp's Vulkan build cannot see MIG instances at all.

What is measured. Each instance's memory is its own; temperature, power and wear are its card's, shown on each of its instances. Utilisation is not reported for an instance, because no driver measures it per instance: the console says n/a on MIG, and no llmgw_gpu_utilization_percent series is published for it rather than a zero.

sudo llmgw doctor lists each card's MIG mode and instances, and warns of a card in MIG mode with none created (nothing can run on it: CUDA will not open the whole card while MIG is on), of GPU instances with no compute instance, and of a mode change waiting for a reset. Such a card is also kept out of placement, with the reason, in llmgw ctl nodes.

InfiniBand, if you have it

sudo apt-get install -y rdma-core ibverbs-utils infiniband-diags
ibstat            # must list your HCAs with State: Active
ls /sys/class/infiniband

Each fabric needs a subnet manager, and should have a standby; nvidia_peermem should be loaded for GPUDirect RDMA. You name each fabric when you onboard the node, and llmgw checks the name against what the adapters report: every port on one fabric sees the same subnet managers, so two independent fabrics given one name are placed as two, and the cluster page says so. An adapter that is not there, or whose port goes down, takes that fabric off the node within seconds, and gives it back when the port returns.

fabrics.md covers declaring several fabrics, what the cluster and fabric pages show, and what to look at when a replica spanning nodes will not run. Run llmgw doctor on the node before it joins.

Engines

Nothing to do: the agent installs them once the node joins, under /opt/llmgw/engines, and checks each one runs before the node offers it.

Engine	On	How
llama.cpp	every node	the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU
vLLM	nodes with an NVIDIA GPU it can run on	a Python environment built by uv from PyPI, about 8 GB
SGLang	only when you ask for it	the same way as vLLM
TensorRT-LLM	only nodes where you installed it	not installed by llmgw: see TensorRT-LLM, which you bring

vLLM is resolved as PyPI stood a day after that vLLM was published, so every node gets the same dependencies whenever it is built, and installed with --require-hashes against the sha256 PyPI publishes. The resolved list stays in the environment as llmgw-requirements.txt. It needs:

• the build its hardware runs, chosen by the node. Driver 580 and newer: the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing (7.5) and newer, 8.0 and newer on aarch64. A V100 (Volta, 7.0) gets vLLM 0.19.1, the newest built for it — no build from 0.20 on has its kernels — and the cluster page shows it running that version; run llmgw ctl engine-check vllm there once, since llmgw's flags are checked against the current one. Pascal and older run no vLLM. Whatever is chosen, the install ends by computing something on the GPU; a build that cannot is passed over for the next;
• about 25 GB free under the engines directory while it installs;
• gcc, which Triton builds its kernel launcher with. The agent installs it.

A node with no GPU skips vLLM and says nothing about it. A node that has a GPU but cannot run it — an old driver, a V100, a small disk — says why on the cluster page. llmgw doctor checks the same before the node joins.

The versions are the control plane's, in server.yaml:

engines:
  vllm: "0.30.0"      # the default; "" leaves vLLM to you
  sglang: "0.5.20"    # SGLang only when named

A changed version reaches every node when it next connects. To use an engine you installed yourself on a node, name its path in that node's agent.yaml (engines.llamacpp.bin, engines.vllm.bin, engines.sglang.python, engines.trtllm.python): the agent then installs none of its own beside it.

Choosing a node's vLLM build

Which build a GPU node installs is chosen on the cluster page, in the vLLM build column — auto (what its hardware runs, the default), CUDA 13, CUDA 12.9, or none — or from the command line:

llmgw ctl engine-build gpu07 vllm cu129

The choice is kept by the control plane and the node is told at once; it installs the build in the background, and the column shows how that stands.

Different GPUs, different engines

A cluster of two GPU generations may need two builds of an engine, or one on only some nodes. engine_overrides picks nodes — by a GPU class they have, by id, or by labels, every one given having to match — and lays engines over the cluster's for them, in order:

engines:
  vllm: "0.30.0"              # CUDA 13: Turing and newer
engine_overrides:
  - gpu_class: a10g           # pin a class to a version you have tested
    engines: { vllm: "0.29.0" }
  - labels: { role: gguf }    # nodes meant for llama.cpp alone
    engines: { vllm: "" }     # none

A node told nothing it can run of an engine joins without it: llmgw doctor and node add warn, and the node serves what it does run. To keep an engine off a node for good, whatever the control plane says, add the node with llmgw node add --disable-engine vllm, which writes engines.disabled: [vllm] into its agent.yaml and leaves vLLM out of doctor's checks. That is narrower than --ignore-checks, which passes over every other problem as well.

A llama.cpp CUDA build is checked for the GPU it is meant for: installing it asks it which devices it sees, and a replica placed on GPUs is not ready until its process holds memory on each of them. A build missing a CUDA library fails there, naming the library, rather than serving on the CPU.

After changing a version, check it still takes the flags llmgw builds for it, which drift between releases:

llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm

TensorRT-LLM, which you bring

llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its container; an installer guessing at all of that would fail on most machines it met, a long way into a large download. Install it on the node yourself — a virtual environment, or NVIDIA's TensorRT-LLM container — and name its interpreter in that node's agent.yaml:

engines:
  trtllm:
    python: /opt/trtllm/venv/bin/python   # python3 inside NVIDIA's container

The node then offers the engine trtllm whenever that interpreter has the tensorrt_llm package; without the line it offers nothing, whatever is on PATH. It needs TensorRT-LLM 1.3 (1.3.0rc13 or later): 1.2 has no --served_model_name, and publishes none of the queue figures the router reads. Check a node's against the flags llmgw builds:

llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python

A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve downloads a repository itself, or loads one from the mirror:

name: llama-3.1-70b-instruct
source: meta-llama/Llama-3.1-70B-Instruct
profiles:
  - name: h100x4
    engine: trtllm
    gpu_class: h100-80g
    tp: 4
    args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]

The agent starts trtllm-serve serve <model> --host … --port … --tp_size … --served_model_name …, through a few lines of Python of its own that do what trtllm-serve has no flag for:

• the key. trtllm-serve checks none; every path but /health is made to demand the one the gateway gives the replica, so the port on the node is not a way around the gateway.
• the metrics. Its options file turns on return_perf_metrics and enable_iter_perf_stats, without which /prometheus/metrics does not exist and its queue figures do not move. The router reads trtllm_num_requests_running, trtllm_num_requests_waiting and trtllm_kv_cache_utilization there — not /metrics, which on trtllm-serve is JSON. They are set as the engine iterates, so an engine gone idle shows its last batch until the next request.
• no usage reports to NVIDIA: TRTLLM_NO_USAGE_STATS and DO_NOT_TRACK are set, after the profile's own env.

What it does not do:

• span nodes. A TensorRT-LLM replica larger than one machine is launched under MPI or Slurm from one of them, which the agents, each starting its own node's part, do not do. A profile with nodes above 1 is refused; use vLLM or SGLang for one.
• run before Ampere. Compute capability 8.0 or newer, which is what current TensorRT-LLM builds kernels for.
• take an options file from a profile. --config and --extra_llm_api_options are llmgw's; change a setting with --set PATH=VALUE instead (--set kv_cache_config.enable_block_reuse=false). Also refused, like the other engines' equivalents: --trust_remote_code, --tokenizer, --middleware, --chat_template, --grpc and --num_serve_frontends (both would serve without the key), and the flags the agent sets.

This engine was written against TensorRT-LLM's documented command line, server and metrics, and tested with stand-ins for them; it has not yet run on a datacenter GPU. Run engine-check on the first node before placing a model on it.

Weights

Nothing to do here if you built the mirror in step 3 — the agent fetches what a model's source names, and that source is your mirror. This section is for the arrangements where the files are already on the machine.

Put them where the node can read them. scp, a shared filesystem, or the engine's own downloader are all valid, and which you want depends on how many nodes you have.

sudo install -d -o root -g root /var/lib/llmgw-models
# then place your .gguf or model directory there

For a first model, something small: a 3B at q4_k_m is about 2 GB, runs on CPU, and answers fast enough to prove the whole path works.

Letting llmgw fetch them

The simple arrangement, and the default: every node keeps its own copy. Point a model's source at a file to download and the agent fetches it once, before the engine is launched:

# your mirror, which is what step 3 was for
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
# a whole version, for a model of several files — the trailing slash is the signal
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
# or straight from Hugging Face, where the nodes can reach it
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf

A fetch reports its progress to the control plane, so llmgw ctl replicas and the dashboard both show how far along a node is. A replica pulling a hundred gigabytes and one that is wedged do not look alike.

Eight shards of one replica starting together produce one download, not eight. An interrupted fetch resumes rather than starting over, and a partial file is never visible under the model's name — so a half-downloaded model cannot be picked up by the next scan and handed to an engine.

Three kinds of source, and only one of them is fetched:

source	What happens
/var/lib/llmgw-models/x.gguf	Checked, never copied. However it got there.
hf:org/repo/file.gguf, https://…	Fetched once into the cache, then reused
Qwen/Qwen2.5-7B-Instruct	Left alone — vLLM and SGLang download repositories themselves

Which means the exotic arrangements need no configuration at all. Weights on a Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are simply present, and present means nothing is fetched.

Telling llmgw where they are

Optional, and unset by default. Naming the directories in the agent's model_dirs lets a node report which models it can see, and placement then prefers a node that already holds one over a node that must first move tens of gigabytes:

model_dirs:
  - /var/lib/llmgw-models
  - /mnt/lustre/models

llmgw does not manage that storage and does not care how the files arrived. A Lustre or NFS mount, an image exported over the fabric with SRP or NVMe-oF, local NVMe, or an scp all look identical to it. A node with nothing configured reports nothing and is treated exactly as every node was before this existed.

It matters most when your fabrics are not connected to each other. Weights in ib-a are not weights in ib-b: the only path between them is the management network, which is the slow one. llmgw ctl status shows what each fabric holds, so a model seeded into one fabric does not quietly leave the rest of the cluster unable to serve it.

How much disk a node needs follows from the models rather than a rule of thumb. llmgw ctl model plan <repo> prints the sizes; as a starting point, a 70B at bf16 is 141 GB and the same model at q4 is 43 GB.

5. Adding the node

On the control plane, which holds the certificate authority:

sudo llmgw node add ubuntu@10.0.0.141 --fabric ib-a:mlx5_0,mlx5_1

It logs in to the machine over SSH as that user, who needs passwordless sudo there, and:

1. installs llmgw: the release for the machine's architecture, downloaded by the machine and checked against the release's checksums, or this machine's copy when it cannot download;
2. runs llmgw doctor there and stops on any problem, printing the fix. Run it again once fixed, or add --ignore-checks;
3. issues the node's certificate and installs it, with the cluster's root, trusted system-wide;
4. writes agent.yaml — only what the node cannot find for itself — and the systemd unit, and starts the agent;
5. waits for the node to join and to install llama.cpp (--wait also waits for vLLM, --no-wait for neither).

In a terminal it draws this as a checklist and asks what a script cannot: which fabric each group of the node's active adapters is on, when --fabric was not given, and whether to go on past a problem doctor found.

The node id is the machine's hostname, and the address its agent dials is this machine's on the route to it; --node and --server override them. The VM name, the hostname, the node id and the certificate's common name being the same string saves an afternoon each time they would differ.

SSH runs as root under sudo, with root's keys. To use yours, log in to the control plane with ssh -A and run sudo --preserve-env=SSH_AUTH_SOCK llmgw node add …. It waits for SSH rather than failing on it, and learns the host key of a machine it has not seen — a new machine spends its first minute regenerating host keys and restarting sshd.

Running it again on a node is safe: a fresh certificate and configuration, and the engines it runs are adopted by the restarted agent.

Within a few seconds:

llmgw ctl nodes

NODE          STATE  GPUS      MEM   CLASSES   FABRICS  REPLICAS  LAST SEEN
llmgw-node01  ready  8/8 free  503G  h100-80g  ib-a     0         2s

If it does not appear, the control plane's log says why. A certificate whose common name does not match the claimed node id is refused by design — that check is what makes a node's identity something it proves rather than something it picks.

Repeat for each node. Nodes in the same fabric must be given the same --fabric name; nodes with no InfiniBand take no --fabric at all and are each their own island, which is fine for single-node models.

6. Setting the node's CPU model, if it is a VM

Proxmox defaults a VM to kvm64, which hides AVX, AVX2, FMA and even SSE4.2 from the guest. Inference then falls back to scalar code and is not slightly slower but unusable. Measured on one 8-core VM, same weights, same threads, the only change being --cpu host:

Qwen2.5-Coder	kvm64	host
1.5B q4_K_M	5.80	39.09 tok/s
3B q4_K_M	2.90	20.51
7B q4_K_M	1.33	9.88

llmgw-provision-node sets --cpu host for you. A VM you built yourself probably did not, and nothing about it looks broken — every number is simply small. Check with grep -qw avx2 /proc/cpuinfo on the node.

7. Adding a model

Write the definition. This is the same shape a model takes in server.yaml, so an entry moves between the two by being copied:

# qwen-coder-3b.yaml
name: qwen2.5-coder-3b-instruct
aliases: [coder]
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
routing_profile: chat
# What a client may use in one request. Worth stating: llama.cpp divides
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
# 32768 each, and nothing else will tell you that.
context_length: 32768
max_output: 4096
scaling: { min: 1, max: 1 }
profiles:
  - name: cpu
    engine: llamacpp
    weight: 1
    priority: 100
    startup_timeout: 5m
    min_memory_gb: 8
    args: ["--ctx-size", "65536"]

llmgw ctl model add -f qwen-coder-3b.yaml
llmgw ctl models

Adding a model checks that the source is actually there before accepting it, so a typo or a mirror that is down is a sentence now rather than a replica failing on a node in five minutes with an error naming the node. --no-source-check skips it, which is right when the nodes reach the mirror by an address this machine does not.

With a mirror configured you can skip the file entirely: /admin/mirror lists what the mirror holds and offers a form that writes the same definition, with the engine, the source and the memory figure already filled in from the file itself.

MODEL                      SERVING  REPLICAS  STATUS   ALIASES
qwen2.5-coder-3b-instruct  yes      1/1       ready    coder

No restart, and it survives one: models added this way are stored, while anything in server.yaml is declarative and returns on every start.

Models that load when someone asks

A cluster with more models than GPUs can hold every model in its catalog and only the wanted ones in its cards:

scaling: { min: 0, max: 1, idle_timeout: 15m }

min: 0 with a max above it means load on demand. The first request for the model starts it and waits briefly; if it is not serving by then the client gets a 503 with Retry-After and an honest "the model is loading", rather than being told nothing serves it — a client told to retry succeeds where one told the model does not exist gives up. Once nobody has asked for it in idle_timeout, the replicas are stopped and the cards go back to the pool.

min: 0, max: 0 is different and equally useful: configured, and deliberately not running. A model with no scaling stanza at all gets that, because making every unscaled model loadable on demand would let any client request start a 70B nobody intended to run.

Two things to weigh. A cold start is visible to whoever triggers it, so this suits the long tail people are experimenting with rather than the two or three models used daily — keep those at min: 1. And stopping a replica throws away the KV cache prefix routing spent its effort filling, so an idle timeout wants to be minutes rather than seconds.

When the cluster is full, a model somebody asks for will stop an idle one to make room. Only idle replicas of other on-demand models: a model at min: 1 is a promise that it stays up, and breaking that to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, because evicting a multi-node one frees several machines and costs tens of minutes to rebuild — a decision worth making deliberately. One at a time. And never a replica that has just started, which is what stops two models on a cluster with room for one from taking turns evicting each other.

For a GPU model, name the class and the width instead:

profiles:
  - name: tp2
    engine: vllm
    gpu_class: h100-80g    # as `llmgw ctl nodes` reports it
    tp: 2
    priority: 100

If it does not place, ask:

llmgw ctl explain-placement qwen2.5-coder-3b-instruct

which names every candidate and why each was rejected, rather than leaving you to infer it.

LoRA adapters

A llama.cpp model can carry LoRA adapters, each asked for by its own name as though it were a model:

- name: qwen2.5-coder-7b-instruct
  source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
  adapters:
    - name: coder-sql
      source: http://mirror:8081/coder-sql/v1/adapter.gguf
      scale: 1.0          # optional; 1 when left out
  profiles: [...]

Every replica of the model fetches every adapter and loads it at start, applied to nothing. A request for coder-sql goes to the model's replicas with the adapter applied to it alone; a request for the model gets the model. Adapter names share the namespace of models and aliases, appear in /v1/models, and follow key allowlists on their own: a key allowed coder-sql reaches that adapter and not the model under it, while a key allowed the model reaches its adapters too. Adding or removing an adapter is a change to the model; running replicas load the new set when restarted (llmgw ctl restart --model M). Besides the mirror page's model card, adapters are attached from the command line, which reads the file first and refuses one that is not an adapter for this model's architecture:

llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
llmgw ctl model adapters coder --remove coder-sql

The adapter file must be a GGUF LoRA made for this model — llama.cpp's convert_lora_to_gguf.py makes one from a PEFT adapter — and not a model with the adapter already merged in, which is what most "LoRA GGUF" repositories hold.

Models served outside the cluster

A model can be an OpenAI-compatible API elsewhere — a hosted provider, or another gateway — rather than replicas here. Nothing is placed for it; the router sends its requests on, with the provider's key and the provider's name for the model:

- name: cloud-mini
  source: external
  external:
    url: https://api.openai.com          # without /v1
    model: gpt-4o-mini                   # what the provider calls it
    api_key_file: /etc/llmgw/secrets/openai.key

Named in another model's fallbacks, it is used only when the cluster has no capacity left for that model:

- name: qwen2.5-coder-7b-instruct
  fallbacks: [cloud-mini]
  ...

The key is read from its file on the control plane, never stored in the catalog, and a key rotated in the file is used within seconds. A model whose key file cannot be read is not offered. Everything else applies as to any model: prices and budgets, rate limits, usage records, and key allowlists.

Its prompts leave the cluster. Decide who may send them: a key or project whose allowlist leaves the external model out cannot reach it, as a fallback or by name. The cluster page and ctl models mark it external.

Trying new weights on some of the traffic first

When a model's repository publishes new weights, llmgw ctl model update swaps them in. To see how they serve first, run them as a canary:

llmgw ctl model canary qwen-7b --percent 10    # fetched, started beside, given 10%
llmgw ctl model canaries                       # how each side has served
llmgw ctl model canary qwen-7b --percent 50    # more
llmgw ctl model promote qwen-7b                # the model moves to them
llmgw ctl model canary qwen-7b --abort         # or not

• What runs: the new files are fetched into the mirror and checked as for an update, then run as qwen-7b-canary — the same model in everything but its weights, one replica. Clients cannot name it; it gets its share of qwen-7b's requests.
• Who goes where: a session (x-session-id, or the user field) stays on one side, so a conversation does not change weights halfway. x-llmgw-variant: canary or stable pins a request to one side; the response says which served it in the same header, and the usage record names qwen-7b-canary as the model that served it.
• Rolled back by itself: once the canary has served --min-requests (50), it is taken out if its error rate is --max-error-increase (0.05, five points) above the model's, or its mean time to the first token — or to the answer, unstreamed — is --max-ttft-ratio (1.5) times the model's. canary_rolled_back goes to the notification webhooks. The comparison is the router's own count since the canary began, kept per control plane.
• Promote is the update the canary was a trial of: the model's replicas are replaced one at a time, and llmgw ctl model rollback goes back.

Quality suites, and holding new weights to one

A canary says whether new weights serve as fast and as reliably; it does not say whether their answers are as good. A republished quantisation or a chat template that no longer matches can make a model faster and worse at once. A quality suite is a handful of questions with known answers, asked of a model and scored:

llmgw ctl eval --suites                                   # what the control plane can run
llmgw ctl eval qwen-7b --suite arithmetic                 # by name, on the control plane
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
llmgw ctl eval --history qwen-7b                          # the runs kept
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d        # one, case by case

It prints each case's score and why, then the suite's: the cases' scores weighed, 0 to 1, and whether it reached the suite's threshold (or --min-score); it exits non-zero below it, so a script can stop on it.

• How it is asked: on the control plane, through the router as the gateway's own requests, the way a replay is — past admission, quotas, accounting and the guardrails, so the score is the weights' and not a tenant's limits'. It reaches the model exactly as a client's request would, a canary or a model loading on demand included. Answers are asked at temperature 0 unless a case says otherwise, and a reasoning model's <think> block is not part of its answer.
• Where suites come from: --suite is a file on the machine running ctl, sent whole, or the name of one in the control plane's control.eval_dir (/etc/llmgw/eval). Two are built in and run on small models: arithmetic (sums and word problems, each scored by the number in the answer) and json-format (answers asked for as JSON of a given shape); their files are in deploy/eval/, to copy and change. A file in the directory with the same name replaces the built-in one.
• What is kept: every run, in the store, with the model, the repository and revision the answering weights came from (when known), the suite's name and sha256, when, the score, and each case's answer (up to 2,000 characters), score and reason. A run on a canary is filed under the model it is a canary of. Runs are not pruned.

A suite is YAML (or JSON lines: one case on each line, and optionally a line without a prompt that holds the suite's own settings):

name: support            # the file's name unless given
threshold: 0.8           # the score to pass at, 0 to 1; 0.8 unless given
judge: llama-70b         # grades the cases scored by a rubric
system: You are a support agent for Acme.   # every case's, unless it has one
max_tokens: 256          # every case's; 512 unless given
cases:
  - id: refund-window
    prompt: How many days do customers have to ask for a refund?
    number: 30                     # the last number in the answer, ±tolerance
    tolerance: 0
  - prompt: Which plan includes SSO?
    contains: [Enterprise]         # every string must appear
    ignore_case: true
  - prompt: Reply with only the word OK.
    exact: OK                      # the whole answer, white space around it aside
  - prompt: What is today's ISO date format?
    regex: '\d{4}-\d{2}-\d{2}'     # matches somewhere in the answer
  - prompt: Give the order as JSON with "id" and "total".
    json_schema:                   # the answer, read as JSON, satisfies it
      type: object
      required: [id, total]
      properties: { id: {type: string}, total: {type: number, minimum: 0} }
  - messages:                      # a conversation instead of a prompt
      - { role: user, content: My card was charged twice. }
    judge: Apologises, and says the duplicate charge is refunded within 5 days.
    weight: 3                      # counts three times; 1 unless given

Each case is scored by exactly one of exact, contains, regex, number, json_schema or judge. A JSON answer in a code fence or inside a sentence is still read. The schema supports type, properties, required, additionalProperties, items, enum, const, the numeric and length bounds, pattern, anyOf and allOf; a suite using anything else is refused when it is read rather than passed unchecked. A judge case asks another model on the cluster (judge_model: on the case, else the suite's judge:, else --judge) to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that got no answer scores 0 and says why.

Holding updates to a suite. model update and model promote take --eval SUITE --min-score N (and --judge); or a model carries one — set with ctl model gate, or eval: { suite: arithmetic, min_score: 0.8 } in a definition added with ctl model add — so every update and promotion of it is held to it, from ctl, the console or anywhere:

llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
llmgw ctl model update qwen-7b              # asked the suite first
llmgw ctl model update qwen-7b --skip-eval  # once, without it; the job says so
llmgw ctl model gate qwen-7b --off

• Promote asks the canary the suite before anything moves; below the score the job fails with the score, and the canary stays, with its share, until you take it out.
• Update runs the new weights first as MODEL-canary given none of the model's requests (hidden from clients like any canary), asks it the suite, then either promotes it — the model's replicas are replaced one at a time, as for any update — or takes it out again, leaving the model on its old weights. That needs room for one more replica while it runs; a model that already has a canary is promoted or aborted first. Without a suite, an update swaps the weights in as it always has.
• The job's eval step says the score (llmgw ctl jobs JOB), and the run is kept with the job's id beside it.

The console has it on a model's menu: Ask it a quality suite… (a suite the control plane has, the score as it comes in), Its quality runs, and Hold its updates to a quality suite….

How many replicas, and how much each one carries

Replica count is changed while the cluster runs, without editing the definition:

llmgw ctl scale qwen2.5-coder-3b-instruct --min 3

Models added at runtime only. One declared in server.yaml is read again on every start, so a number set here would hold until the next restart and then revert — change it in the file instead.

Between min and max a model scales with its load: requests waiting at every replica add one, a quiet spell takes it back. For load that comes by the clock, a schedule says so ahead of time:

scaling:
  min: 1
  max: 4
  prewarm: 30m                 # start a window's replicas this long before it opens
  schedule:
    - {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
    - {from: "22:00", to: "02:00", min: 1}     # past midnight

Times are in control.timezone (Europe/Istanbul; empty is the control plane's own). The first window that holds wins; outside every window, min and max apply. An on-demand model (min: 0) with a window is kept loaded through it, and loaded again on the first request after. Days are mon … sun, weekdays or weekends.

How much traffic each replica is given is the profile's weight, and guessing it is how a cluster ends up with one saturated replica beside two idle ones. Measure it instead:

llmgw ctl bench qwen2.5-coder-3b-instruct --save

Concurrency climbs until the latency target stops being met, and the last level that held it is the capacity — peak throughput arrives long after latency has become unusable, so throughput alone would overstate what the hardware serves. It prints the weight to put in the profile, and --save records the figure against the model so the dashboard can show what the cluster is measured to carry rather than what somebody typed.

This drives real traffic and competes with real requests. Run it against a cordoned replica, or in a window where that is acceptable.

An engine that publishes no metrics endpoint is measured from the outside only. The run says so, and queue depth and KV occupancy are then not part of the answer.

8. The first request

curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "coder", "messages": [{"role":"user","content":"hello"}]}'

Or open /chat in a browser and use it without a key at all — the page carries your session, and the request still goes through admission, rate limits, budget and accounting exactly as an API call does.

What the client port answers

Path	What
POST /v1/chat/completions, /v1/completions, /v1/embeddings	OpenAI's, served by the models in the catalog
POST /v1/responses	OpenAI's Responses API, translated to chat completions: tools, text.format (structured output), input_image (by URL or data: URL), and previous_response_id — the conversation is kept as a thread, so the project must keep conversations (llmgw keys threads); store: false keeps none
POST /v1/messages, /v1/messages/count_tokens	Anthropic's Messages API, translated to chat completions: text, images, tools and their results, the system prompt, streaming; the key in x-api-key or Authorization. Extended thinking, prompt caching and server tools are ignored
POST /v1/rerank, /v1/score	passed through to an engine that serves them: vLLM with a reranker or a cross-encoder
POST /v1/audio/transcriptions, /v1/audio/translations	the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper
POST /v1/moderations	answered by the guardrails' judge model (role: guard): flagged, and the judge's own categories
/v1/files, /v1/batches	batches; see Batches
GET /v1/models	the models this key may use

Watching it

The control plane exports both halves on one endpoint: what the cluster is serving — requests, latency, time to first token, tokens, admission, per endpoint — and what it is made of, per node and per GPU, including memory, temperature, ECC faults and utilisation.

The dashboard is the answer for most clusters. /admin shows capacity per GPU class, free whole nodes per fabric, which replicas are holding hardware nothing is asking for, who is using the cluster and what they are running, and the same node and model tables ctl prints — with no monitoring stack to stand up beside it. One binary, no dependencies, which is the point.

Prometheus is worth adding when you want history, alerts, or one view across several clusters. The dashboard shows what is true now; it keeps nothing.

Metrics then get their own listener, because the admin one also carries pprof and pprof can dump the process heap, where API keys sit in plaintext:

admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof — keep local
metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach

One scrape target for the whole fleet. The node and GPU figures come from the control plane, which already holds them, so there is nothing to install on the nodes and no Prometheus configuration per machine.

A starting configuration, with the alerts worth having and a provisioned dashboard, is in deploy/monitoring/ — including where to run it, which is not the control plane: monitoring that dies with the thing it monitors tells you nothing at the moment you need it.

Two figures repay knowing about. llmgw_gpu_utilization_percent is absent rather than zero when the driver does not report it — a flat zero across a fleet means the measurement is missing, not that the fleet is idle. And llmgw_gpu_nvlink_peers at zero everywhere means the topology is unknown, which is what a binary built without the nvml tag reports.

Token by token, and SLOs

Beside time to the first token, every streamed answer is timed per token: llmgw_tpot_seconds (each output token after the first), llmgw_inter_token_seconds (the gaps between streamed events, which show the stalls a mean hides) and llmgw_output_tokens_per_second, by model and pool, and llmgw_tenant_tpot_seconds and llmgw_tenant_output_tokens_per_second by tenant.

A model can say what it promises:

slo: {ttft_p99: 2s, tpot_p99: 80ms}

Then its recent p99 — the last five minutes, once there are twenty streamed requests — counts in admission: a model past its SLO is treated as saturated by as much as it is past, so the lower bands wait while the queue and the cache still look healthy. llmgw_model_slo_seconds publishes the SLOs, and the llmgw-slo alerts in deploy/monitoring fire when a p99 stays past one for fifteen minutes.

Being told

Prometheus alerts travel once alertmanager.yml names a receiver; it ships templates for Slack, e-mail, PagerDuty and a webhook of your own, commented out until you choose.

A few things are worth a message even without a monitoring stack, and the control plane sends those itself:

Event	When
model_parked	A model failed every start in a row and is no longer retried (ctl retry tries again).
node_lost	A node was silent past control.lost_after; its replicas are rebuilt elsewhere.
budget_spent	A key, project, team or organization reached its budget; priced models are refused.
update_available	A model's registry repository published different weights. Once per revision.
canary_rolled_back	A model's canary served worse than the model and was taken out.
judge_unavailable	A guardrails judge gave no verdict: with fail: closed requests are being refused, with open served unjudged.

notify:
  webhooks:
    - url: https://hooks.slack.com/services/…      # Slack, Mattermost, Rocket.Chat
      format: slack
      events: [model_parked, node_lost]
    - url: https://ops.example.com/llmgw            # your own receiver
      secret_file: /etc/llmgw/notify-secret          # signs the body

A json webhook (the default) gets the event as JSON — kind, at, subject, message, detail — with X-Llmgw-Event naming the kind and, when secret_file is set, X-Llmgw-Signature: sha256=<hex>, the HMAC-SHA256 of the body under the file's contents; check it before believing the body. A webhook that does not answer is tried three times and then given up on, and the same event about the same thing is sent at most once an hour, so a budget that refuses a thousand requests is one message. llmgw ctl notify test sends a test event to each webhook and says how each answered.

Tracing

Metrics say that a model took four minutes to start; a trace says where they went. Give the control plane and every agent the same OpenTelemetry collector, over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:

# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
  endpoint: http://otel.internal:4318
  sample_ratio: 1          # of requests that arrive without a decision of their own

OTEL_EXPORTER_OTLP_ENDPOINT does the same when the file does not say. With neither, nothing is traced and nothing is sent: every span is a no-op, and a command to a node carries no trace context.

What is traced:

Trace	Spans
An inference request (llmgw-server)	inference /v1/… with admission, schedule and an upstream per attempt beneath it; a caller's traceparent is continued and the engine is sent one.
A reconcile pass that did something	reconcile, and beneath it what it did: scale up with place (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and replica start; scale down; drain replica and stop replica, each with its reason; evict, preempt with its victims; reap replica with the error and when it is tried again; restart replica, restart refused; node unreachable, node lost. A pass that found nothing to do is not traced.
A replica's start	replica start stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A send start per shard beneath it is the command to the node, and the node's spans (llmgw-agent) continue the same trace: start shard, with fetch weights (local, cached, downloaded or fetched by another shard, and a download with its bytes), launch engine (and wait for rendezvous on a worker of a multi-node replica) and wait for engine (how often the engine was asked, and its last answer if it never came up).
A stop on a node	stop shard and stop engine in the trace of the control plane's stop replica, saying whether the engine had to be killed.
An adoption	adopt, when a node that registers reports engines already running, with how many nobody claimed.
A model job	serve model or update model, with a span per step — registry, mirror, check, catalog, placement, smoke test, replicas — and from the placement step a link to the replica's own start.
An engine install	install engine on the node, a trace of its own.

So "why did qwen take four minutes to start" is one trace: find replica start with llmgw.model = qwen, and the node's spans beneath it say whether the time went to the download, the launch or the engine loading.

A node follows the control plane's decision: a start the control plane kept is traced on the node whatever the agent's own sample_ratio, and one it sampled out is not; the agent's ratio decides only for what arrives with no trace, as from a control plane that is not tracing. The agent needs its own tracing block either way — without one its spans go nowhere, and the control plane's trace simply ends at send start.

PostgreSQL, when SQLite is no longer the right shape

SQLite is the default and stays the right answer for most installations: one file, no server, and a request path that barely touches it — usage is written in batches, budgets are counted in memory, authentication is cached.

Two things change that. Conversations and agent notes read and write on every turn, which is the first per-request load this store has had; and SQLite takes one writer at a time, so those turns serialise. And a second control plane is impossible while the store is a file on one machine's disk.

On the control plane, one command does all of it — installs PostgreSQL if the machine has none, creates its role and database with a generated password kept in /etc/llmgw/secrets/store-dsn, stops the gateway, copies the SQLite store into it so every account and key survives, points server.yaml at it, and starts the gateway again:

sudo llmgw server init --postgres

It can be given on the first run too, and then there is never a SQLite store. The rest of this section is what that command does, for a PostgreSQL somewhere else.

The setting is the same one, and the address decides:

auth:
  store: "@/etc/llmgw/secrets/store-dsn"

A path is SQLite, a postgres:// URL is PostgreSQL, and @ followed by a path reads the address out of that file — which is how a DSN carrying a password stays out of a configuration that gets copied, diffed and pasted into issues. Keep that file at 0600 and owned by the user llmgw runs as.

Moving an existing installation is a copy, not a fresh start. Keys are stored hashed, so starting fresh means reissuing every one of them and being unable to say which was which:

sudo systemctl stop llmgw-server
llmgw keys move-store --from /var/lib/llmgw/llmgw.db --to "@/etc/llmgw/secrets/store-dsn"
# point auth.store at the new address, then
sudo systemctl start llmgw-server

Stop the gateway first. The copy makes no attempt to follow a store being written to: rows landing after the table they belong to has been passed would simply be absent, and nothing would say so. The destination must be empty, and the source is left untouched — so the way back is to point the configuration at the file again.

On a machine that also runs the control plane, PostgreSQL's defaults assume it owns the box. A gateway swapped out to give the database cache is a worse outcome than a smaller cache, so cap it: shared_buffers = 96MB and max_connections = 40 are enough for a store whose working set is measured in megabytes.

Tagging requests

A client can say what a request was for — which feature, which customer, which environment — and the spend can then be found by it inside a project:

curl http://gateway:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-tags: feature=search, env=prod" \
  -d '{"model": "qwen-1.5b", "user": "customer-42",
       "metadata": {"feature": "search"},
       "messages": [{"role": "user", "content": "hello"}]}'

• metadata is OpenAI's: an object of strings, up to 16 keys. x-llmgw-tags is the same as a header, for clients that cannot change the body; for a key both give, the header wins. A key is 1 to 64 letters, digits, _, . or -, a value at most 512 bytes; anything else is refused with 400 rather than dropped, since a tag lost silently is spend nobody finds.

• user is OpenAI's end-user field: the client's own name for whoever it serves, not an account here. It is passed on to the engine, as OpenAI does; tags are not.

• Where they show: every usage record keeps them, and llmgw ctl request <id> prints them. For Prometheus, name the tags worth a series:

  router:
    tag_labels: [feature, env]
  

  llmgw_tagged_requests_total{tag,value,outcome} and llmgw_tagged_tokens_total{tag,value,direction} count those, the first 100 values of each apart and the rest as other: a tag is the client's to set, and every value is a series.

Keeping requests, and replaying them

To follow a request up — what exactly was sent, what came back — or to try real traffic on another model, a project can keep its requests:

llmgw keys requests --project Acme/ml/train --retain-hours 72            # every one
llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05
llmgw ctl request req_3f2a… --body                   # what was sent and answered
llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct   # the same, to another model
llmgw keys requests --project Acme/ml/train --off

• What is kept: the request as the model was sent it — after the guardrails' redaction, with any stored conversation spliced in — and the answer as the client got it, up to 128 KB of it. A share (--sample) is the same requests every time for the same ids.
• Replay sends it as the gateway itself: past the limits, the budget and the guardrails, which judged it the first time, and charged to nobody.
• Never for a project under a guardrails policy with no_store. Kept requests past their hours, or of a project turned off, are removed hourly. The console has it as Keep requests… on a project.

Reusing answers

A project that asks the same thing again and again — an evaluation run over the same prompts, embeddings of documents that do not change, an agent that retries — can have the same request answered from memory rather than by the model:

llmgw keys cache --project Acme/ml/train --ttl 1h          # reproducible answers
llmgw keys cache --project Acme/ml/eval --ttl 24h --any    # every answer
llmgw keys cache --project Acme/ml/train                   # how it stands
llmgw keys cache --project Acme/ml/train --off

• What is the same request: the same project, the same endpoint, the same model (by its catalog name, whichever alias was used) and the same body, with stream, stream_options, user, metadata, store and service_tier left out and the keys in any order. Keys of another project never see the answer, whatever they send.
• Which answers are reused: a chat or completion asked at temperature 0 or with a seed, and every embedding, rerank and score, which sample nothing. A request sampled above temperature 0 with no seed asks for a new answer, and is asked anew — unless the project says --any. Only a whole 200 from the model that was asked for is kept: not an error, not a fallback model's or a canary's answer, not one a guardrail changed or stopped, not one longer than router.cache.max_entry_bytes (1 MB).
• Streams: an answer given whole is streamed to a client that asks for a stream, as one chunk per choice with the usage chunk if it asked for that; a streamed answer is kept put back together, and served whole to a client that does not stream. Through /v1/messages and /v1/responses too.
• The client decides too: Cache-Control: no-cache (or no-store), or x-llmgw-cache: no, is always asked anew. Every answer of a project that reuses answers says x-llmgw-cache: hit, miss or bypass (not one that can be reused), and a hit says its Age in seconds.
• Policies still apply: the limits are checked and the guardrails judge the request before the cache is looked in, and the output policy judges a reused answer as it would a new one. A request continuing a stored conversation, or under a no_store policy, is never answered from the cache nor kept in it.
• What a hit costs: nothing. It is recorded as a request — status 200, the model, the key, the rate limit's count — with no tokens and no cost, marked as a cache hit (llmgw ctl request <id> says so), so it spends no budget and no token limit. The answer itself carries the usage the model reported when it was first given.
• Where it lives: in each gateway process's memory, at most router.cache.max_bytes (128 MB) of answers for all projects together, those used longest ago dropped first; nothing is written to the store, and a restart begins it again. With two control planes behind a balancer each keeps its own. A change of setting is seen within 30 seconds; a shorter TTL applies to what is already kept.

router:
  cache:
    max_bytes: 268435456     # 256 MB
    max_entry_bytes: 2097152 # 2 MB

llmgw_cache_requests_total{model,result} counts hits, misses and bypasses, llmgw_cache_bytes and llmgw_cache_entries what is held. The console has it as Cache answers… on a project.

Batches

Work that can wait — a night's classification, embeddings for a document store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.

curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
  -d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"

Each line of the file is {"custom_id": …, "method": "POST", "url": "/v1/chat/completions", "body": {…}}, at most 50 000 of them (router.batch.max_lines) in at most 100 MB (router.batch.max_file_bytes).

• How it is served: each line as a request of the key that made the batch — its limits, its budget, its guardrails — in the lowest admission band, behind every interactive request, router.batch.concurrency (4) at a time. A rate limit or a model still loading is waited out; a spent budget fails the lines left.
• What comes back: answers in an output file, failures in an error file, both OpenAI's form, kept router.batch.retain_days (30). Each request is a usage record tagged batch=<id>, so llmgw ctl request finds it.
• Who sees it: files and batches belong to the key's project; every key of the project sees them, no other does.
• Surviving a restart: every answered line is kept as it is answered, and whichever gateway holds the batch's lease carries on from there; a control plane that stops hands its batches to another within two minutes.
• What it keeps: a batch is prompts and answers stored until deleted or past their retention — a project under a guardrails policy with no_store is refused it.

Conversations across models

Off everywhere until a project asks. llmgw otherwise stores no message content at all — usage rows carry token counts, costs and latencies, never text, and even its own chat page keeps the conversation in the browser. A gateway that keeps nothing anyone typed is a much easier thing to have in front of a team, so keeping it is a decision someone makes rather than a default they discover.

llmgw keys threads --project default --retain-hours 24

The retention is required. A conversation store with no expiry is a transcript of everything a team ever typed, kept forever because nobody chose a number. The window is measured from the last turn, so a conversation still in progress does not expire underneath it.

A client then sends a thread id and only its new turn:

curl $BASE/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-thread: my-conversation" \
  -d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'

A thread belongs to the project, not to the key: any key of the project that knows the id reads the conversation. Use ids nobody can guess — a UUID, not my-conversation — of at most 128 letters, digits and - _ . :.

llmgw puts the stored history in front of that turn and sends it to whichever model is serving — including a different one from the one the conversation started on. That is the point: a thread begun on a large model can continue on a small one, and a caller dropped to a cheaper model by a spent budget keeps their conversation.

When the history does not fit the model now serving, the oldest turns are dropped until it does, and the reply carries x-llmgw-thread-trimmed with how many. System messages are never dropped, and neither is the turn the caller just sent.

DELETE /v1/threads/{id} forgets one. It is a client endpoint on purpose: the person who said the thing should be able to unsay it without asking an operator to read it first. A thread belonging to another project answers the same as one that never existed.

Turning it off again stops new storage, and existing conversations go at the next hourly prune:

llmgw keys threads --project default --off

Memory notes for agents

Also off until a project asks, for the same reason. Where conversations are what was said, notes are what an agent chose to keep, and they persist across conversations:

llmgw keys memory --project default --retain-hours 720

The retention is required and is measured from the last write to a note. An agent then saves, searches and deletes notes through /v1/memory or, as tools, through the MCP server at /mcp (remember, recall, forget). Both need the API key of the project, and notes are private to the project and to the agent id sent in x-llmgw-agent. A note in another project or agent answers the same as one that does not exist. llmgw does not add these tools to requests itself.

Turning it off stops new storage, and existing notes go at the next hourly prune:

llmgw keys memory --project default --off

Without authentication, threads.store and threads.retain_hours in server.yaml keep both conversations and notes in one shared pool. That is for one trusted machine only; see development.md.

Two control planes

llmgw handles the hard half itself: only one places work, and agents find whichever is leading. What it does not do is move client traffic, which happens in front of both.

deploy/ha/ has the pattern and a keepalived example. The one thing to get right is the health check: point it at /ready, not /health. /health is 200 on a standby, which is alive and holds no endpoints at all, so a load balancer using it would send half the traffic to something that refuses every request.

The portal's language

The web portal — the landing page, sign-up and sign-in, the account and chat pages, and the administrators' pages — speaks English and Turkish. Each person gets, in this order:

1. the language they chose with the switcher at the foot of every page, which is remembered in a cookie (llmgw_lang) for a year;
2. otherwise the first language their browser asks for that the portal speaks;
3. otherwise the server's own default:

# /etc/llmgw/server.yaml
landing:
  language: tr   # en or tr; empty is English

A code the portal does not speak stops the server starting, rather than quietly serving English. Figures and dates follow the page's language: a Turkish page writes 1.234,5 and 04.10.2026. Messages that come from the control plane itself — a node's warnings, an engine's errors — are shown as they were written, in English. The API, the CLI and the console are English.

Adding a language is adding a directory: internal/web/locales/<code>/ with the same files and keys as locales/en/, and its _meta.* entries saying how it writes numbers and dates. The tests refuse a catalogue that lacks a key a page uses.

Single sign-on

People can sign in through your identity provider — Keycloak, Entra ID, Google, Okta, Authentik, anything that speaks OpenID Connect.

1. Register a client there with the redirect address https://<gateway>/login/oidc/callback.
2. Put its secret in a file only root reads.
3. Add to server.yaml:

auth:
  oidc:
    issuer: https://login.example.com/realms/acme
    client_id: llmgw
    client_secret_file: /etc/llmgw/secrets/oidc-client-secret
    label: Acme SSO                 # the button says "Sign in with Acme SSO"
    create_accounts: true           # make an active account for someone new
    roles:                          # staff roles from groups, set at each sign-in
      admin: [llmgw-admins]
      operator: [llmgw-ops]
    memberships:                    # groups to places in the tree, joined at sign-in
      ml-engineers: "team:Acme/ml"
    no_passwords: true              # the provider only; administrators keep a password way in

How the sign-in works:

• Request: it uses PKCE, a state and a nonce.
• Identity: the ID token is verified against the provider's keys, and the account is found by the verified email address.
• Unknown addresses: without create_accounts, only people who already have an account can sign in.
• Roles: a person taken out of a group loses the role the next time they sign in.
• Memberships: they are added, never taken away.
• Audit: each sign-in is recorded in the audit log.

Provisioning (SCIM)

The provider can also keep accounts in step: made when someone is assigned the application, suspended when they are removed or leave. Give it a token:

auth:
  scim:
    token_file: /etc/llmgw/secrets/scim-token     # any long random string, root-only

and point it at https://<gateway>/scim/v2 with that token as the bearer.

• Users: the userName is the email address. Deleting a user suspends the account, and its keys and history stay.
• Groups: auth.oidc.roles and memberships are applied as people join and leave them. Membership in a group is removed when someone leaves it.

Secrets in Vault

Every setting that names a file holding a secret — the store's @address, api_key_file, auth.oidc.client_secret_file, auth.scim.token_file, a webhook's secret_file — also takes env:VARIABLE or a field of a secret in HashiCorp Vault or OpenBao:

auth:
  store: "@vault:secret/llmgw/store#dsn"
secrets:
  vault:
    addr: https://vault.internal:8200
    ca_file: /etc/llmgw/vault-ca.pem
    role_id_file: /etc/llmgw/secrets/vault-role-id      # AppRole; or token_file,
    secret_id_file: /etc/llmgw/secrets/vault-secret-id  # or VAULT_TOKEN
  master_key: vault-transit:llmgw

These are read when the gateway starts (an external model's key, each time the catalog changes): a secret rotated in Vault is taken up at the next restart.

The store's own secrets — what every engine's key is derived from — are kept in the store, and so in its backups. secrets.master_key seals them:

• vault-transit:KEY seals with Vault's transit engine. The key never leaves Vault, so neither the store nor any backup of it opens without Vault.
• Anything else is a reference to 32 random bytes (openssl rand -hex 32), used here: better than nothing, but a key in /etc/llmgw is in every backup beside what it seals. Keep it in Vault or the environment.

Values kept plain from before are sealed the first time they are read after the key is set. The key cannot be changed once set without losing them: a new one cannot open what the old one sealed (the engines are then restarted with new keys: llmgw ctl restart --all).

The firewall

server init, node add, server add and mirror init each keep that machine's internal llmgw ports to the cluster's own networks, with an nftables table of llmgw's own (inet llmgw, in /etc/llmgw/firewall.nft, loaded at boot by llmgw-firewall.service):

Machine	Ports, for the cluster only
control plane	7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote
node	engine ports (control.port_range, 20000–20999), 9100
store member	5432, 8008, 2379, 2380, 9100
mirror	8081, 8082, 9100

"The cluster's networks" are, on each machine, the network of the interface it reaches the control planes (or the other store members) on — or just their address, when the route goes through a gateway. Anything else that must reach those ports, a Prometheus on another subnet or nodes behind a router, goes in server.yaml before the machines are set up:

firewall_networks: [10.0.5.0/24, 192.168.1.20]

Only those ports are touched. The table's policy is to accept, so whatever else the machine serves — and whatever firewall it already has — is left as it was; the client API (:8080) is not closed, being meant to be reached. Running a command again adds to what is there rather than replacing it, so a machine that is both a store member and a node keeps both sets. llmgw doctor on a node says when the table is missing. --no-firewall on any of the four commands leaves the firewall alone; sudo systemctl disable --now llmgw-firewall removes it.

Upgrading

On the control plane:

sudo llmgw ctl upgrade v0.10.4

It upgrades this machine first — fetching the release for its platform, checking it against the release's checksums, keeping the old binary as /usr/local/bin/llmgw.bak-pre-<version>, restarting llmgw-server and waiting for every node to reconnect — and then each node in turn. A node's agent is told the version over the stream it already holds, fetches and checks the release itself, swaps it in and restarts; no ssh into the nodes is needed. Release candidates (v0.10.4-rc.2) work the same way.

A node counts as done when it is back on the new version, ready, with as many ready replicas as before. One that is not within --timeout (three minutes) stops the upgrade there, so a bad release reaches one node rather than all of them; llmgw ctl logs <node> says why, and the node keeps its previous binary beside the new one. Nodes already on the version are skipped, so running the command again resumes where it stopped. --nodes a,b limits it to some nodes, and --skip-control-plane leaves this machine alone.

Nothing stops serving. Engines are started with KillMode=process, so they keep running when the agent restarts, and the agent reports them back on its next connection. The control plane holds no replica state across its own restart either — it rebuilds the fleet from what the agents report.

A mixed fleet is fine while you roll. The agent protocol only ever gains fields, and both sides treat an absent one as unknown rather than as a default. Agents older than v0.10.4 cannot upgrade themselves: move those by hand once, installing the release binary and restarting llmgw-agent, and every upgrade after that is the command above.

The mirror is upgraded on its own host, since it is not an agent and its service cannot replace its own binary:

sudo llmgw upgrade v0.10.5 --service llmgw-mirror

The same checks — the release's checksums, the old binary kept beside the new — then a restart of the unit, waiting for it to stay up. A mirror older than v0.10.5 does not have the command: install the release binary by hand once.

Then confirm what you have, rather than assuming:

llmgw ctl nodes       # every agent on the new version, LAST SEEN in seconds
llmgw ctl replicas    # same replicas, adopted rather than replaced

A replica may come back marked ready (stale profile). That means it is running a profile that has since been edited — the reconciler leaves a serving replica alone on purpose, because churning one costs more than the difference. llmgw ctl restart <replica> replaces it with one on the new profile, starting the new one before stopping the old. Where the old one is all that stands in the new one's way — a single GPU — it is stopped first, at once, and the model is unserved while the new one loads; where the new profile fits nowhere even then, the restart is refused and the old one keeps serving, which ctl replicas shows as restart refused: with the reason. --in-place asks for stop-then-start whatever room there is.

Signed releases

Every install above checks a release against its published checksums, which proves the download is what the server holds — not that the server holds what was built. Signing closes that: CI signs the checksums, and a binary that trusts a key installs only releases signed with it. Off until a key exists; turning it on, once:

llmgw release keygen

1. Put the private key in the repository's Actions secret RELEASE_SIGNING_KEY. From the next tag, releases carry checksums.txt.sig.
2. Put the public key in internal/release/trusted.pub and release. Every binary built from then on carries it and refuses releases that are not signed with it — ctl upgrade, agents upgrading themselves, and llmgw upgrade alike.

In that order: a build that trusts a key refuses unsigned releases, so the secret has to be signing before the key is trusted. The first trusting release is itself installed by a binary that trusts nothing; every one after is checked by the one before. LLMGW_RELEASE_KEY overrides the built-in key, for rotating it. The onboarding script fetches with curl and checks checksums only; a node is checked from its first upgrade on.

Backing up

llmgw server backup writes one archive holding everything a control plane cannot be rebuilt without:

• the store — users, API keys, the tenancy tree, limits and budgets, usage history, models added at runtime, measured capacity, and the secret every engine key is derived from (pg_dump in its custom format for PostgreSQL, a consistent VACUUM INTO snapshot for SQLite; both are safe against a running gateway);
• /etc/llmgw — the configuration, the store's address, and the certificate authority's key (pki/). Nothing in llmgw can reconstruct that key; losing it means reissuing every node's identity.

Archives go to /var/backups/llmgw-server, readable by root only because of the CA key, and the newest 14 are kept. Run it nightly with the timer:

sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service     # one now, to see it work
ls -l /var/backups/llmgw-server

The control plane publishes when the newest archive was written as llmgw_backup_last_success_timestamp_seconds, and BackupTooOld fires when it is more than 26 hours old — including when there has never been one.

Encrypt them. Name who may open them in /etc/llmgw/server.yaml, and every archive is encrypted to those keys with age and named .tar.gz.age:

backup_recipients:
  - age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p   # age-keygen's public key
  - ssh-ed25519 AAAAC3Nza... ops@example                              # or an SSH public key

Keep the matching private keys off this machine — an archive whose key sits beside it is not protected by it. Encrypted, a copy is safe wherever it lands.

Copy the archives off the machine. They sit on the disk they back up and do not survive losing it. Anything that copies files will do — rsync from the mirror, a NAS mount, an offsite bucket — as long as it keeps the archives readable by root only.

Keys are stored hashed and cannot be recovered from a backup or from anywhere else. Restoring an old store restores which keys exist and who owns them, not their secrets. A key created after the backup is gone and has to be reissued.

Copying archives off the machine

backup_recipients: [age1…]          # required: an archive leaves only encrypted
backup_copy_to:
  - /mnt/backups/llmgw              # a mounted share or another disk
  - ssh://backup@10.0.0.20/srv/llmgw-backups
backup_copy_key: /etc/llmgw/secrets/backup-ssh-key

Each archive is copied there as soon as it is written, and the same number is kept there as here (--keep, 14). An ssh host's key is learnt the first time and checked every time after, in /etc/llmgw/backup_known_hosts. A copy that fails makes the backup fail, so the timer's unit shows it.

Restoring

On a machine with llmgw installed — the same version as the archive's, or newer — and, for PostgreSQL, the database and role the store's address names:

sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz --dry-run
sudo llmgw server restore llmgw-backup-<host>-<time>.tar.gz.age --identity key.txt

It checks before it touches anything — that the archive is whole and opens, that this llmgw is not older than the one that wrote it, where the store goes and that pg_restore is there for a PostgreSQL dump — and --dry-run stops there. Then it stops llmgw-server, moves /etc/llmgw aside to /etc/llmgw.before-restore-<time>, puts the archive's in its place with each file's mode and owner, restores the store (a SQLite file replaced, the old one kept beside it; PostgreSQL by pg_restore --clean in one transaction), starts the server and waits for it to answer. --store restores the store somewhere else than the archive's server.yaml says.

Running engines keep serving throughout: agents reconnect to the restored control plane and their replicas are adopted, and because the engine-key secret came back with the store, the router can still reach them.

A disaster recovery drill

Worth doing once before it is needed, on a spare machine, and again after a major upgrade:

1. Install the same llmgw version on the spare machine (step 1).
2. Copy the newest archive there from wherever backup_copy_to put it, and the private key that opens it.
3. sudo llmgw server restore <archive> --identity <key> --dry-run, and read what it says. For PostgreSQL, create the database and role first, or point --store at a scratch database.
4. Without --dry-run. Then llmgw ctl status, llmgw keys list --all, and a request with a key that existed before.
5. Note how long it took: that is the number to plan an outage around.

The drill machine now holds the cluster's CA key and secrets: take it down afterwards, or keep it as the standby it has become.

The mirror's directory is the one thing you can rebuild, by downloading it all again. A pinned version may no longer be upstream to download: that is exactly what pinning it protected you from, and a copy of the directory is what makes the protection outlive the disk.

Reclaiming disk

llmgw ctl nodes has a DISK column: free space against the size it is free out of, for the filesystems holding model weights. A dash means the node was not asked — an agent with no model_dirs, or one older than this — and not that it has no space. Shared filesystems are excluded from a node's figure, because space on a filer belongs to everyone mounting it.

Weights llmgw downloaded for models that are no longer in the catalog can be listed and removed:

llmgw ctl weights             # what could be freed, per node
llmgw ctl weights --collect   # delete it

Only files llmgw fetched itself are ever candidates. They are recognisable because llmgw names them after a hash of their source, so the set still wanted is computed from the catalog exactly rather than guessed. Anything you put on a node — by scp, by a shared mount, by hand — is left alone however unreferenced it looks, and no llmgw command will remove it. The nodes check every path again before deleting it, so a control plane that is wrong cannot talk one into losing a model that took a week to copy.

Which also means llmgw will not reclaim the storage most likely to be filling your disk. Weights you placed yourself are yours to remove.

A profile may declare min_disk_gb, and a node without that much free is then not a candidate for it — checked only on nodes that do not already hold the weights, since those need no room for them. Without the field, disk is not considered, which is how every profile written before it keeps behaving.

When something does not work

The node is ready but every replica fails. Almost always a missing engine or weights. llmgw ctl replicas shows the shard's error and the engine's last output, which is the actual diagnosis.

A model will not place. llmgw ctl explain-placement <model> names each node and the reason: wrong GPU class, not enough free cards, too little memory, no fabric, a label it does not carry.

Everything is slow. On a VM, check AVX2 (step 6). On CPU, check the quantisation — the difference between q8_0 and q4_k_m is roughly a factor of two. llmgw ctl status shows how full the cluster is.

Tool calls come back as text. The model made the call and the engine did not parse it. llmgw ctl tool-check --endpoint http://<node>:<port> distinguishes that from a model ignoring its tools, which has a different fix.

Requests fail at exactly sixty seconds. timeouts.response_header is a liveness check for streaming and a total generation budget for non-streaming requests. On a slow model, set timeouts.generation higher.

What to read next

• docs/choosing-models.md — which models to run and how many, if the answer to "we have GPUs, now what" is not already obvious
• docs/data.md — what is stored and what leaves the machine, for the review that happens before this is allowed near anyone's documents
• docs/reference.md — every configuration key, generated from the source, so it describes the version you are running rather than the one somebody last wrote about
• deploy/mirror/README.md — the mirror in more detail, including the layout and what belongs on which port
• deploy/monitoring/README.md — Prometheus, Grafana and the alerts, with what each one assumes about this system
• docs/fabrics.md — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node, and what to look at when a replica spanning nodes will not run
• docs/node-onboarding.md — adding nodes in more detail, and what onboarding deliberately does not do
• docs/onboarding.md — giving people accounts and keys
• llmgw-plan.md — why the system is shaped the way it is

Known gaps, so you find them here rather than at 2am

• Every node fetches its own copy. There is no seeding between nodes, so bringing up eight nodes on a new model is eight downloads. The mirror makes those eight local, which is most of the problem; on a fabric where the nodes can see each other, seeding between them is the obvious next thing to build.
• A budget without a price never binds. Set default_price or pricing (step 2) or every request costs zero, and a budget is something the dashboard displays rather than something that stops anything. The Limits tab says so where it can, which is not the same as being told before you rely on it.
• A V100 runs vLLM 0.19.1, not the version the rest of the cluster runs: no build from 0.20 on has Volta kernels, and the node picks the newest that has. llmgw's flags are checked against the current version, so run llmgw ctl engine-check vllm on a V100 node once.
• Two control planes have not been run for long. The leader election and agent failover work and there is a pattern in deploy/ha/, but every cluster this has been run on in anger has had one. A single control plane is the well-trodden path.
• Multi-node replicas have not been run on real InfiniBand or NVLink. The scheduling, the fabric identity, the maps and the launch are written against the real tools' formats and tested against them, not yet run on hardware that has them. Single-node placement on real GPUs is the well-trodden path; if you are the first to run a TP group across real cards, run llmgw doctor --json on each node first and keep the reports.
• One engine setup is weaker than the rest. vLLM's launch_mode: ray starts a Ray head without authentication, so anyone who can reach its port can run code on the node, and Ray's ports are not among those llmgw's firewall guards. The dashboard is off, but firewall the port to the other nodes in the replica, or use the default mp launch mode, which needs no Ray at all.