llmgw/docs/install.md

# Installing llmgw from nothing

Written for whoever is standing up the cluster: you have servers, you have root
on them, and nothing is installed yet. For only the commands, read
[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
node in the fleet, and a model answering requests.

It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
the package names. Everything here has been done on a real cluster except where
this page says otherwise, and where it says otherwise it says so plainly.

## What llmgw does not install

Read this first, because it is the difference between an afternoon and a week.

llmgw installs itself, its engines and its store. It does **not** install the
GPU driver or the InfiniBand stack: those are kernel modules with version
matrices attached, and a platform decision. A node missing them still
registers and reports `ready`, and fails every replica that needs the GPU or
the fabric — so make a node able to serve *before* onboarding it.
`llmgw doctor` on the node says what is missing and the command that fixes it.

| You provide | llmgw provides |
| --- | --- |
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) |
| Which models to run | The mirror that fetches them once, placement, routing, accounting |

Weights are handled too. llmgw includes a mirror — step 3 — which holds the
files on your own network and hands them to the nodes, so getting weights onto
every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you.

## 0. Before you begin

Four machines is a reasonable first cluster: one control plane and three nodes.
One of each also works.

**The control plane** needs no GPU: one machine, holding its store in SQLite
or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under
Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable
from every node on the agent port, and from your users on the client port.

**Nodes** need whatever the model needs, plus about 8 GB for the operating
system and the agent. They dial the control plane; the control plane never dials
them, so nodes need no inbound access except your own SSH.

**Ports**

| Port | Who connects | Expose to |
| --- | --- | --- |
| 8080 | clients, the web UI | your users, usually via a reverse proxy |
| 7443 | node agents | the node network only |
| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
| 8081 | node agents fetching weights | the node network only (the mirror machine) |
| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |

Decide now which address nodes will use to reach the control plane. It goes into
a certificate: the machine's own addresses are found for you, and any other name
— a DNS alias, a NAT or floating address — has to be given with `--host`.

## 1. The binary

One static binary, no runtime dependencies, the same on every machine.

```
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
  | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL  $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```

`arm64` and `darwin` builds are published alongside. They are built with
`CGO_ENABLED=0` deliberately, so a binary built today runs on a node whose libc
is older than the builder's.

To build instead of download, you need Go 1.26.6 or later and nothing else (an older Go fetches it itself):

```
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
```

## 2. The control plane

**The short way is the screens.** On the machine that will be the control
plane, run `sudo llmgw` with nothing after it. With nothing installed it asks
whether this is one machine or three for high availability, then lists what
an install is made of, each part with what it is for and what is missing
without it:

| Part | | Its settings (Enter) |
|---|---|---|
| Control plane | always | the administrator, other names for the certificate, the portal's language, the sign-up form |
| Store | always | SQLite or PostgreSQL on this machine; with three machines, PostgreSQL under Patroni on the other two |
| Control plane on all three | three machines, on | a floating address for whichever leads |
| Model mirror | on | here, and in which directory, or on another machine |
| This machine as a node | on for one machine | |
| Starter model | on for one machine | which: Qwen2.5 0.5B, 1.5B or 3B, Qwen3 4B, 8B or 14B, or any GGUF on Hugging Face; its context |
| Pictures and 3D | off | |
| Guardrails' judge | on | refuse or serve flagged when it does not answer; copies; how long it is waited for; what it refuses |
| Firewall | on | |
| Notifications | off | the webhooks |
| Limits and keys | always | requests and tokens a minute; the longest a key lives |
| Taking turns on a card | always | how long an on-demand model goes unused, and is safe after loading |
| Offline bundle | off | the bundle and its sha256 |

Space turns a part on or off; Enter opens its settings, which say more. What
cannot go together is said before anything is done — a starter model with no
mirror to fetch it into — and closing the message opens the setting that is
wrong. For three machines it then asks for the other two — address, user, and
an ssh key, the ssh agent, or a password, and the sudo password if sudo asks
for one; or, for a machine nothing may log into, a join code to run there
with `sudo llmgw join <code>`, which dials out to this one on port 7444 —
checks all three before installing anything (system, room, clock, ports,
internet, whether they reach one another). Last it shows the plan, part by
part, what is left out and what that costs, and installs. Passwords are held in memory for
the run only; a sudo password lets the user sudo without one until the install
finishes, and is then taken back. At the end it shows the administrator's
password and the command that does the same from a script.

Run again later, `sudo llmgw` is the console: the cluster's state, refreshed,
and what to do next — add a node, replicate the store. A setup that stopped
halfway opens the setup screens again, which finish it.

The rest of this section is the same by command, as root:

```
sudo llmgw server init --admin you@example.com
```

That is the whole step. In order, it:

1. creates the `llmgw` system user, `/etc/llmgw` and `/var/lib/llmgw`
2. creates the cluster's certificate authority in `/etc/llmgw/pki`
3. issues the control plane's certificate for this machine's hostname and
   every address it has, and copies `ca.pem` beside it
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
   and a price, keys that expire after at most 90 days, the baseline
   guardrails policy, and a control plane with an empty catalog
5. creates the store, the `default` project, and your account, already approved
   and promoted to administrator — the password is generated and **printed
   once**
6. installs the systemd unit, starts it, and waits until `/health` answers
7. runs a model mirror beside it (`/srv/models`, bytes on `:8081`, its
   management API on loopback) and points the control plane at it
8. makes this machine a node of its own control plane — on its GPUs when
   `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
   engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
    CPUs: one replica on up to four cores when this machine is the only
    node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
    first thing anyone opens `/chat` to find is a model that answers. A
    machine without room for it is told so within two minutes, and why.

`--no-mirror` and `--no-node` leave out 7 and 8, `--no-firewall` 9,
`--no-guard-model` 10 and `--no-model` 11. Every setting the screens ask for
is a flag too, and the plan shows the command with them: `--starter qwen3-8b`
(or `hf:<owner>/<repo>/<file>.gguf`) and `--starter-context`, `--mirror-dir`,
`--judge-fail open`, `--judge-categories`, `--judge-replicas`,
`--judge-budget`, `--language tr`, `--signup-open`, `--default-rpm`,
`--default-tpm`, `--max-key-ttl`, `--media` (pictures and 3D: ComfyUI on the
nodes, and `presets.auto`), `--evict-after-idle` and `--min-lifetime`. They
go into `server.yaml` as init writes it, and become the cluster's first
settings when it starts; a `server.yaml` already there is never overwritten,
so run again they change nothing — the settings page does. An NVIDIA card with no working
driver is reported rather than silently ignored; install the driver and run
init again to serve on it.

**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:

| | |
|---|---|
| Agents | mutual TLS with the cluster's own authority; the agent port refuses plaintext |
| Admin API | loopback only, behind a token; pprof off |
| Store | between machines, TLS everywhere (PostgreSQL, etcd, Patroni) |
| Firewall | llmgw's internal ports answer the cluster's networks only |
| Accounts | signup closed; administrators create accounts or approve them |
| Keys | at most 90 days (`auth.max_key_ttl`), shorter per part of the tree with `llmgw keys ttl` |
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |

The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
streamed answer every 600 characters. The policy fails **closed**: a request
the judge cannot read within 5 seconds is refused with 503 rather than
served unjudged, and `judge_unavailable` goes to the notification webhooks.
So a judge gets cores of its own: on one machine one replica on up to four,
and on a larger cluster two replicas of two cores each, spread over its nodes
so that losing one leaves the other. On a machine of fewer than four cores,
a busy one, or a VM without AVX2 (see
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
first thing to fail. The choices there are a longer `latency_budget`,
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
or `--no-guard-model`. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.

Two things are left to you because they need something only you have: a
certificate for the client port — port 8080 is plain HTTP; put the proxy in
[deploy/proxy/](../deploy/proxy/) in front — and the keys backups are
encrypted to (`backup_recipients`, see [Backing up](#backing-up)). An NVIDIA
card with no working driver is reported rather than silently ignored; install
the driver and run init again to serve on it.

Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
init installs it and sets it up, and on a machine already running on SQLite it
moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).

Or `--patroni`, given two machines (or more), to keep it in PostgreSQL
replicated across them, so losing one loses nothing:

```
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --patroni ubuntu@10.0.0.149 --patroni ubuntu@10.0.0.150
```

Over ssh with passwordless sudo, as `node add` works, it installs PostgreSQL,
Patroni and etcd there — this machine is etcd's third vote — and moves the
store already in use, SQLite or PostgreSQL, into the cluster. A standby is
promoted within about thirty seconds of the primary being lost, and the
control plane follows it by itself. `llmgw ctl status` says who leads; see
[deploy/ha](../deploy/ha/README.md) for what it sets up.

Everything between the machines is TLS, with certificates from the cluster's
own authority: PostgreSQL (every remote rule is `hostssl`, and the control
plane verifies the member it reaches), etcd (peers and clients, and a client
must present a certificate), and Patroni's API, which also asks for a
password before a change. A cluster set up by an earlier llmgw is moved to
this by running the same command again, on the control plane that holds the
authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
PostgreSQL serves throughout and no failover happens.

Each member's PostgreSQL is sized to the machine's memory, read each time
the command runs: a quarter of it as cache (`shared_buffers`), three quarters
counted on for the planner, and room to sort and build indexes, all capped. A
member that also runs a control plane or an agent gets less: about 15% as
cache, so the gateway is not pushed into swap. A knowledge base's vector
index is searched well only from memory, so give a store machine more RAM and
run the command again. The new cache takes a PostgreSQL restart, which Patroni
does one member at a time, the leader last. The leader's restart costs a few
seconds without writes, and there is no failover.

Every step looks before it acts. Running it again finishes an install that
stopped halfway and changes nothing on one that did not; it never overwrites
`server.yaml`, the authority, or an existing account's password.

**The addresses in the certificate matter.** Agents check that the address
they dial is named in it, and the certificate names only what this machine
could see of itself. If nodes will reach it by another name — a DNS alias, a
NAT address, a floating IP — give it:

```
sudo llmgw server init --host llmgw-ctl.internal --host 203.0.113.10
```

Run that at any time: the certificate is reissued with the new names added to
the old ones, and the service restarted.

**`ca-key.pem` signs every node's identity.** `init` leaves it in
`/etc/llmgw/pki`, because onboarding a node needs it and the control plane is
where most people run onboarding from. Back it up; there is no recovering it.
If this machine is exposed, move the directory elsewhere once the nodes are in
and pass `--pki` to onboarding.

This covers the fleet-facing port only. The client-facing API is public and
belongs behind an ordinary publicly trusted certificate, terminated by a
reverse proxy in front of port 8080. [deploy/proxy/](../deploy/proxy/) has a
Caddyfile and an nginx configuration that get streaming, timeouts and the
forwarded headers right; both are attached to every release.

### Check it

```
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200
sudo llmgw ctl status
```

`ctl status` will report no nodes and no models, which is correct — nothing has
been added yet. Sign in at `http://<control-plane>:8080/login` with the printed
password, change it, and issue yourself a key from the profile page.

### The configuration

There are two halves, and each change to either is in force within seconds,
on every machine it concerns, without anyone restarting anything.

**The cluster's settings** — guardrails, limits and prices, routing, the
reconciler, engines, notifications, the mirror's policy, models declared by
hand, offline, federation, facility budgets, backups' recipients — are kept in
the store, so every control plane runs on the same ones. Change them with

```
llmgw ctl config                         # what is in force, and its version
llmgw ctl config set router.retries 3 --note "flaky upstream"
llmgw ctl config edit                    # the whole document in $EDITOR
llmgw ctl config history                 # who changed what, when, and why
llmgw ctl config rollback 4              # back to version 4, as a new version
```

or on the console's **Settings** screen, or in a browser on the portal's
**Settings** page (`/admin/settings`, for administrators only: the document
can hold webhook secrets). The page edits the same document, lists its
history, and puts an earlier version back. A change is checked before it is
kept, so one that would not load is refused. A change made from a version
someone else has replaced since is refused too, rather than overwriting
theirs. Every change is recorded in the audit log, with the document's size
but not its contents.
Guardrails change live; anything else has every control plane start again on
the new settings, the standbys first and the leader half a minute later, so
one of them is always serving. Engines keep running through it.

The first control plane to start against a store brings in its server.yaml's
cluster keys — with a `baseline` guard policy, if the file had none — and
from then on those keys in any server.yaml are not read (the server says so
when it starts).

**The nodes' agents** are set the same way, under `agents`. `agents.default`
is for every node, and `agents.nodes.<node-id>` is for one node, laid over
the default:

```
llmgw ctl config set agents.default.heartbeat_interval 5s
llmgw ctl config set agents.nodes.cuda01.labels.gpu rtx6000
llmgw ctl config edit                    # fabrics, engines, supervisor, discovery…
```

Each node's `agent.yaml` then needs only what its agent must know before it
can be told anything: `node_id`, `server` or `servers`, `hostname`,
`mgmt_addr` and `tls`. Those keys are refused under `agents`.

- **Precedence.** Anything else in the file is still read, with the
  cluster's settings over it.
- **Applying a change.** The control plane sends each node its settings, a
  second apart. An agent whose settings changed keeps them beside its
  `agent.yaml` (in `/var/lib/llmgw-agent/agent-settings.yaml`) and starts
  again on them, its engines serving throughout.
- **Bad settings.** Settings that would not load are refused when they are
  saved. If they somehow do not load beside one node's own file, that agent
  keeps running on what it has and logs why.
- **Editing `agent.yaml` itself.** The agent notices the change within ten
  seconds and starts again on it. If the edited file does not load, the
  agent reports it and keeps running on what it had.

**This machine's own keys** stay in `/etc/llmgw/server.yaml`: `listen`,
`admin_listen`, `metrics_listen`, `agent_listen`, `agent_tls`, `auth.store`,
`secrets`, `tracing`, `monitoring`, `mirror.api` and `mirror.tls`, the backup
directory and key. The server notices the file change and starts again on it
by itself. Three things are worth knowing before you edit it:

- `admin_listen` is on loopback on purpose, and the server refuses any other
  address unless `admin_remote: true`. Its API under `/api` adds models and
  upgrades nodes, so it also wants a token: the server writes one, 0600, to
  `/var/lib/llmgw/admin-token` the first time it starts, and `llmgw ctl`
  reads it from there — which is why `llmgw ctl` runs with sudo.
  `LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
  keys requests presented, is off unless `debug_pprof: true`, and then behind
  the token too. To see the dashboard from elsewhere use `/admin` on the
  client port — do not move this listener. The API itself is described in
  [admin-api.md](admin-api.md).
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
  budgets mean anything. Without a price every request costs zero and a budget
  never binds; without the defaults, a project created without limits is
  metered by nothing at all.
- `control.enabled: true` is what makes this a control plane while its catalog
  is empty. Without it, a file with no models is refused as having nothing to
  serve.

Every key is in [reference.md](reference.md).

### Keeping the authority off the control plane

Create it on another machine, issue the control plane's certificate there, and
bring only the results:

```
# on the authority's machine
llmgw ctl cert init-ca --dir /etc/llmgw/pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.140 --host llmgw-ctl.internal \
  --dir /etc/llmgw/pki --out ./ctl

# copy ca.pem, ctl/server.pem and ctl/server-key.pem into /etc/llmgw on the
# control plane, then there:
sudo llmgw server init --admin you@example.com
```

`init` finds certificates in `/etc/llmgw` and no authority beside them, and
keeps them as they are. Adding an address later is then done where the
authority is; `init --host` says so rather than minting a second authority no
node trusts.

## 3. The mirror

Do this before the nodes, not after. Every node fetches its own copy of a
model's weights, and where they fetch from decides whether bringing up a cluster
takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
Face is a day's work, and on a good number of corporate networks it is not
possible at all — the nodes have no route out, which is usually the point.

The mirror is one machine holding the files on your own network. It is the same
binary again, and it is the highest-value piece of infrastructure around llmgw
for the least work.

### Running it

```
sudo install -d -o llmgw -g llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
```

In production, the unit:

```
sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
```

**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
what every agent talks to, and it is plain HTTP because a weights fetch is a
public file transfer that is checksummed anyway. `:8082` is the management API:
it can start downloads and delete models, and only the control plane needs it.
The split is what lets you lock the second one down without re-certifying every
agent that fetches from the first.

Give the management port the cluster's own CA and it takes client certificates —
the control plane's, named with `--api-client`, and no other. The same CA signs
every node's certificate, so the name is required: without it any node, a
removed one's included, could delete models.

```
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
  --tls-ca /etc/llmgw/ca.pem \
  --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem \
  --api-client control-plane
```

Without those flags the management port answers this machine only, and the
addresses given with `--api-allow`.

### The layout

```
/srv/models/
  qwen2.5-14b-instruct/
    2026-03/
      q4_k_m.gguf
```

Model, then version, then files. The version directory is the point of the
layout rather than decoration: a repository that changes upstream changes what
you are running, and the first sign is a prompt that worked last week answering
differently this week with nobody able to say why. Pin it here and that cannot
happen.

A version holding several files is a sharded model, and is served as a set: the
agent asks the directory what it consists of and fetches all of it into one
place, marking it complete only when the last file lands. A catalogue entry for
one names the directory, trailing slash included.

### Telling the control plane about it

```yaml
# /etc/llmgw/server.yaml
mirror:
  api: https://models.internal:8082
  tls:
    ca: /etc/llmgw/ca.pem
    cert: /etc/llmgw/server.pem
    key: /etc/llmgw/server-key.pem
```

With this set, `/admin/mirror` appears: what the mirror holds, what it is
downloading and how far along, a search of Hugging Face, and a form that puts a
mirrored model into service without anyone writing YAML. Without it that page
stays unmounted and the mirror is still perfectly usable — a model's `source` is
just a URL, and the URL can be yours.

If the control plane cannot reach the internet, `mirror.registry` points the
search at an internal registry mirror instead.

### Filling it

```
llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
  hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
```

or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.

Files already on the mirror machine go in with `import`, which copies them the
way a download is kept — under a temporary name until whole, the checksum
beside them for nodes to check against, a version never replaced — and checks
that a `.gguf` is one:

```
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
  --version 2026-10 --sha256 <the checksum its publisher gave>
```

#### From Ollama

Models you already run in Ollama are GGUFs llama.cpp serves as they are.
On the mirror machine, with Ollama's models readable there:

```
llmgw mirror import-ollama                 # lists what Ollama has
llmgw mirror import-ollama llama3.2:3b
```

It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
model's GGUF, and a vision model's projector beside it, each checked against
the sha256 Ollama named its file by; a file that does not match is not
mirrored, and nothing of the model is. The version is the tag and the start of
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
`latest` moves in Ollama and a mirror version must not; `--model` and
`--version` choose otherwise. It ends with a model definition to save and
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
with.

Ollama's prompt template is not carried: llama.cpp renders the chat template
inside the GGUF, which for any recent model is the one it was trained on. Its
system prompt and adapters are not carried either, and the import says so.
Models Ollama keeps as separate tensors rather than one GGUF, or as several
weight layers, are refused with a sentence saying so — mirror that model's GGUF
from its publisher instead. When the mirror runs beside the control plane, the
console's Mirror view does the same with **I**.

## 4. Preparing a node

**Do this before onboarding.** A node missing its driver or engine registers
healthy and fails on first placement, and the reason is three logs away.

### The driver

```
sudo apt-get update
sudo apt-get install -y nvidia-driver-550   # or whatever your cards need
sudo reboot
nvidia-smi                                   # must print your GPUs
```

`nvidia-smi` must be on `PATH` for the user the agent runs as, which is root.
Containerised driver installs often put it elsewhere; if so, note the path and
set `discovery.nvidia_smi` in the agent configuration rather than moving it.

A node with no GPU at all is fine and is not an error. It reports zero cards and
can host CPU profiles, which is a good way to try llmgw before committing
hardware to it.

### NVLink, if your GPUs have it

The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
joins, each card's NUMA node, and which network adapter is nearest each. A
tensor-parallel group is then the set of cards with the most NVLink between
them — on a PCIe box where bridges pair specific cards, a group straddling two
bridges falls back to PCIe and runs several times slower, and nothing else
would report it.

On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
checks it:

```
sudo llmgw doctor
```

A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.

### MIG, if you want to cut a card into pieces

An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
GPU) instances, each with its own memory, its own share of the card's compute
and its own UUID. llmgw places a model on an instance as it would on a card,
which makes MIG the way to run several small models on one large card without
them competing for it.

**llmgw does not create the instances.** How a card is cut is your decision,
made once with `nvidia-smi` on the node; the agent reads what is there when
it starts:

```
sudo nvidia-smi -i 0 -mig 1                 # MIG mode on card 0; may need a reset
sudo nvidia-smi mig -i 0 -lgip              # the profiles this card offers
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C   # two instances, each with its compute instance
nvidia-smi -L                               # the instances, with their MIG- UUIDs
sudo systemctl restart llmgw-agent          # so it registers the new devices
```

`-C` matters: a GPU instance with no compute instance inside it is not a
device yet. Turning MIG on takes effect at the card's next reset
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
do not survive a reboot unless something recreates them — NVIDIA's
`mig-parted` or a systemd unit of your own.

**Classes.** A card in MIG mode is reported as its instances instead of as
the whole card, and each instance's class is the card's class and the
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
`llmgw ctl nodes` lists the classes, and a profile names one like any other:

```yaml
profiles:
  - name: mig
    engine: vllm
    gpu_class: a100-80g-mig-3g.40gb
```

**One instance a replica.** A process can open only one MIG device, and
instances have no NVLink or peer-to-peer access between them, so a replica on
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
MIG class asking for more is refused when the model is added, with that
reason. A model too large for one instance wants a larger profile or a whole
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
instance is offered for a model it holds, and never several together.

**How they are numbered and shown.** Placement numbers card 0's instance 2 as
device 1002 (1000 + 100 × card + instance), and every view shows it as
`0/mig2`. The engine is given the instance by its UUID —
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
build. llama.cpp's Vulkan build cannot see MIG instances at all.

**What is measured.** Each instance's memory is its own; temperature, power
and wear are its card's, shown on each of its instances. Utilisation is not
reported for an instance, because no driver measures it per instance: the
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
published for it rather than a zero.

`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
card in MIG mode with none created (nothing can run on it: CUDA will not open
the whole card while MIG is on), of GPU instances with no compute instance,
and of a mode change waiting for a reset. Such a card is also kept out of
placement, with the reason, in `llmgw ctl nodes`.

### InfiniBand, if you have it

```
sudo apt-get install -y rdma-core ibverbs-utils infiniband-diags
ibstat            # must list your HCAs with State: Active
ls /sys/class/infiniband
```

Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
node, and llmgw checks the name against what the adapters report: every port on
one fabric sees the same subnet managers, so two independent fabrics given one
name are placed as two, and the cluster page says so. An adapter that is not
there, or whose port goes down, takes that fabric off the node within seconds,
and gives it back when the port returns.

[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
fabric pages show, and what to look at when a replica spanning nodes will not
run. Run `llmgw doctor` on the node before it joins.

### Engines

Nothing to do: the agent installs them once the node joins, under
`/opt/llmgw/engines`, and checks each one runs before the node offers it.

| Engine | On | How |
|---|---|---|
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |

vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:

- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
  the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
  CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
  publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
  (7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
  **vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
  kernels — and the cluster page shows it running that version; run `llmgw
  ctl engine-check vllm` there once, since llmgw's flags are checked against
  the current one. Pascal and older run no vLLM. Whatever is chosen, the
  install ends by computing something on the GPU; a build that cannot is
  passed over for the next;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.

A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
but cannot run it — an old driver, a V100, a small disk — says why on the
cluster page. `llmgw doctor` checks the same before the node joins.

The versions are the control plane's, in `server.yaml`:

```yaml
engines:
  vllm: "0.30.0"      # the default; "" leaves vLLM to you
  sglang: "0.5.20"    # SGLang only when named
```

SGLang is started with `--enable-metrics`, without which it serves no
`/metrics`; the router reads `sglang:num_running_reqs`, `sglang:num_queue_reqs`
and `sglang:token_usage` there. Check an SGLang against the flags llmgw builds
with `llmgw ctl engine-check sglang --bin <its environment>/bin/python`.

A changed version reaches every node when it next connects. To use an engine
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.

#### Choosing a node's vLLM build

Which build a GPU node installs is chosen on the cluster page, in the **vLLM
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
*CUDA 12.9*, or *none* — or from the command line:

```
llmgw ctl engine-build gpu07 vllm cu129
```

The choice is kept by the control plane and the node is told at once; it
installs the build in the background, and the column shows how that stands.

#### Different GPUs, different engines

A cluster of two GPU generations may need two builds of an engine, or one on
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
id, or by labels, every one given having to match — and lays engines over the
cluster's for them, in order:

```yaml
engines:
  vllm: "0.30.0"              # CUDA 13: Turing and newer
engine_overrides:
  - gpu_class: a10g           # pin a class to a version you have tested
    engines: { vllm: "0.29.0" }
  - labels: { role: gguf }    # nodes meant for llama.cpp alone
    engines: { vllm: "" }     # none
```

A node told nothing it can run of an engine joins without it: `llmgw doctor`
and `node add` warn, and the node serves what it does run. To keep an engine
off a node for good, whatever the control plane says, add the node with
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
than `--ignore-checks`, which passes over every other problem as well.

A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
asks it which devices it sees, and a replica placed on GPUs is not ready until
its process holds memory on each of them. A build missing a CUDA library fails
there, naming the library, rather than serving on the CPU.

After changing a version, check it still takes the flags llmgw builds for it,
which drift between releases:

```
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```

#### TensorRT-LLM, which you bring

llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
container; an installer guessing at all of that would fail on most machines it
met, a long way into a large download. Install it on the node yourself — a
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
interpreter in that node's `agent.yaml`:

```yaml
engines:
  trtllm:
    python: /opt/trtllm/venv/bin/python   # python3 inside NVIDIA's container
```

The node then offers the engine `trtllm` whenever that interpreter has the
`tensorrt_llm` package; without the line it offers nothing, whatever is on
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
`--served_model_name`, and publishes none of the queue figures the router
reads. Check a node's against the flags llmgw builds:

```
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
```

A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
downloads a repository itself, or loads one from the mirror:

```yaml
name: llama-3.1-70b-instruct
source: meta-llama/Llama-3.1-70B-Instruct
profiles:
  - name: h100x4
    engine: trtllm
    gpu_class: h100-80g
    tp: 4
    args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
```

The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
--served_model_name …`, through a few lines of Python of its own that do what
trtllm-serve has no flag for:

- **the key.** trtllm-serve checks none; every path but `/health` is made to
  demand the one the gateway gives the replica, so the port on the node is not
  a way around the gateway.
- **the metrics.** Its options file turns on `return_perf_metrics` and
  `enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
  and its queue figures do not move. The router reads
  `trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
  `trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
  is JSON. They are set as the engine iterates, so an engine gone idle shows
  its last batch until the next request.
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
  are set, after the profile's own `env`.

What it does not do:

- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
  under MPI or Slurm from one of them, which the agents, each starting its own
  node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
  or SGLang for one.
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
  TensorRT-LLM builds kernels for.
- **take an options file from a profile.** `--config` and
  `--extra_llm_api_options` are llmgw's; change a setting with `--set
  PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
  refused, like the other engines' equivalents: `--trust_remote_code`,
  `--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
  `--num_serve_frontends` (both would serve without the key), and the flags the
  agent sets.

This engine was written against TensorRT-LLM's documented command line,
server and metrics, and tested with stand-ins for them; it has not yet run on a
datacenter GPU. Run `engine-check` on the first node before placing a model on it.

### Weights

Nothing to do here if you built the mirror in step 3 — the agent fetches what a
model's `source` names, and that source is your mirror. This section is for the
arrangements where the files are already on the machine.

Put them where the node can read them. `scp`, a shared filesystem, or the
engine's own downloader are all valid, and which you want depends on how many
nodes you have.

```
sudo install -d -o root -g root /var/lib/llmgw-models
# then place your .gguf or model directory there
```

For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on
CPU, and answers fast enough to prove the whole path works.

### Letting llmgw fetch them

The simple arrangement, and the default: every node keeps its own copy. Point a
model's `source` at a file to download and the agent fetches it once, before the
engine is launched:

```yaml
# your mirror, which is what step 3 was for
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
# a whole version, for a model of several files — the trailing slash is the signal
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
# or straight from Hugging Face, where the nodes can reach it
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
```

A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
the dashboard both show how far along a node is. A replica pulling a hundred
gigabytes and one that is wedged do not look alike.

Eight shards of one replica starting together produce one download, not eight.
An interrupted fetch resumes rather than starting over, and a partial file is
never visible under the model's name — so a half-downloaded model cannot be
picked up by the next scan and handed to an engine.

Three kinds of source, and only one of them is fetched:

| `source` | What happens |
| --- | --- |
| `/var/lib/llmgw-models/x.gguf` | Checked, never copied. However it got there. |
| `hf:org/repo/file.gguf`, `https://…` | Fetched once into the cache, then reused |
| `Qwen/Qwen2.5-7B-Instruct` | Left alone — vLLM and SGLang download repositories themselves |

Which means the exotic arrangements need no configuration at all. Weights on a
Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are
simply *present*, and present means nothing is fetched.

### Telling llmgw where they are

Optional, and unset by default. Naming the directories in the agent's
`model_dirs` lets a node report which models it can see, and placement then
prefers a node that already holds one over a node that must first move tens of
gigabytes:

```yaml
model_dirs:
  - /var/lib/llmgw-models
  - /mnt/lustre/models
```

llmgw does not manage that storage and does not care how the files arrived. A
Lustre or NFS mount, an image exported over the fabric with SRP or NVMe-oF,
local NVMe, or an `scp` all look identical to it. A node with nothing configured
reports nothing and is treated exactly as every node was before this existed.

**It matters most when your fabrics are not connected to each other.** Weights
in `ib-a` are not weights in `ib-b`: the only path between them is the
management network, which is the slow one. `llmgw ctl status` shows what each
fabric holds, so a model seeded into one fabric does not quietly leave the rest
of the cluster unable to serve it.

How much disk a node needs follows from the models rather than a rule of thumb.
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
bf16 is 141 GB and the same model at q4 is 43 GB.

## 5. Adding the node

On the control plane, which holds the certificate authority:

```
sudo llmgw node add ubuntu@10.0.0.141 --fabric ib-a:mlx5_0,mlx5_1
```

It logs in to the machine over SSH as that user, who needs passwordless sudo
there, and:

1. installs llmgw: the release for the machine's architecture, downloaded by
   the machine and checked against the release's checksums, or this machine's
   copy when it cannot download;
2. runs `llmgw doctor` there and stops on any problem, printing the fix. Run it
   again once fixed, or add `--ignore-checks`;
3. issues the node's certificate and installs it, with the cluster's root,
   trusted system-wide;
4. writes `agent.yaml` — only what the node cannot find for itself — and the
   systemd unit, and starts the agent;
5. waits for the node to join and to install llama.cpp (`--wait` also waits
   for vLLM, `--no-wait` for neither).

In a terminal it draws this as a checklist and asks what a script cannot: which
fabric each group of the node's active adapters is on, when `--fabric` was not
given, and whether to go on past a problem doctor found.

**The node id is the machine's hostname**, and the address its agent dials is
this machine's on the route to it; `--node` and `--server` override them. The
VM name, the hostname, the node id and the certificate's common name being the
same string saves an afternoon each time they would differ.

**SSH runs as root under sudo**, with root's keys. To use yours, log in to the
control plane with `ssh -A` and run `sudo --preserve-env=SSH_AUTH_SOCK llmgw node
add …`. It waits for SSH rather than failing on it, and learns the host key of
a machine it has not seen — a new machine spends its first minute regenerating
host keys and restarting sshd.

Running it again on a node is safe: a fresh certificate and configuration, and
the engines it runs are adopted by the restarted agent.

Within a few seconds:

```
llmgw ctl nodes
```

```
NODE          STATE  LOCATION  GPUS      MEM   CLASSES   FABRICS  REPLICAS  LAST SEEN
llmgw-node01  ready  -         8/8 free  503G  h100-80g  ib-a     0         2s
```

If it does not appear, the control plane's log says why. A certificate whose
common name does not match the claimed node id is refused by design — that check
is what makes a node's identity something it proves rather than something it
picks.

Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
name; nodes with no InfiniBand take no `--fabric` at all and are each their own
island, which is fine for single-node models.

### Saying where the node is

On more than one rack, tell llmgw which rack, and which PDUs, each node is in.
Placement then keeps a model's replicas in different racks and on different
PDUs, places nothing in a rack held for maintenance, and keeps each rack under
a power cap if you give it one:

```
llmgw ctl label llmgw-node01 llmgw.io/rack=r12-07 llmgw.io/pdu=r12-07-a,r12-07-b
llmgw ctl label llmgw-node01 llmgw.io/row=12 llmgw.io/room=hall-b llmgw.io/site=sto1
```

Labels set this way are kept by the control plane and win over the node's
`agent.yaml`, where the same keys can go under `labels:` instead; a node need
not have joined yet. `llmgw ctl nodes` shows the LOCATION, `--domain
rack:r12-07` lists one rack, and `llmgw ctl facility` lists every rack, row and
PDU with what it draws. In the console it is the node's menu, **Location…**.
A node with no location is placed as before, as a rack of its own.
See [facility-aware-placement.md](facility-aware-placement.md) for power caps
and holds.

## 6. Setting the node's CPU model, if it is a VM

Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. Inference then falls back to scalar code and is not slightly
slower but unusable. Measured on one 8-core VM, same weights, same threads, the
only change being `--cpu host`:

| Qwen2.5-Coder | kvm64 | host |
| --- | ---: | ---: |
| 1.5B q4_K_M | 5.80 | **39.09** tok/s |
| 3B q4_K_M | 2.90 | **20.51** |
| 7B q4_K_M | 1.33 | **9.88** |

`llmgw-provision-node` sets `--cpu host` for you. A VM you built yourself
probably did not, and nothing about it looks broken — every number is simply
small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.

## 7. Adding a model

Write the definition. This is the same shape a model takes in `server.yaml`, so
an entry moves between the two by being copied:

```yaml
# qwen-coder-3b.yaml
name: qwen2.5-coder-3b-instruct
aliases: [coder]
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
routing_profile: chat
# What a client may use in one request. Worth stating: llama.cpp divides
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
# 32768 each, and nothing else will tell you that.
context_length: 32768
max_output: 4096
scaling: { min: 1, max: 1 }
profiles:
  - name: cpu
    engine: llamacpp
    weight: 1
    priority: 100
    startup_timeout: 5m
    min_memory_gb: 8
    args: ["--ctx-size", "65536"]
```

```
llmgw ctl model add -f qwen-coder-3b.yaml
llmgw ctl models
```

Adding a model checks that the source is actually there before accepting it, so
a typo or a mirror that is down is a sentence now rather than a replica failing
on a node in five minutes with an error naming the node. `--no-source-check`
skips it, which is right when the nodes reach the mirror by an address this
machine does not.

With a mirror configured you can skip the file entirely: `/admin/mirror` lists
what the mirror holds and offers a form that writes the same definition, with
the engine, the source and the memory figure already filled in from the file
itself.

```
MODEL                      SERVING  REPLICAS  STATUS   ALIASES
qwen2.5-coder-3b-instruct  yes      1/1       ready    coder
```

No restart, and it survives one: models added this way are stored, while
anything in `server.yaml` is declarative and returns on every start.

A profile may load weights of its own. An embedding model, for example, can
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
on CPUs, as one model:

```yaml
name: qwen3-embed-4b
source: Qwen/Qwen3-Embedding-4B          # what a profile loads unless it says
capabilities: { embeddings: true }
profiles:
  - name: gpu
    engine: vllm
    gpu_class: 6000-24g
    args: ["--runner", "pooling"]
  - name: cpu
    engine: llamacpp
    source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
```

Updates and canaries move the model's `source`; a profile with its own keeps
it.

### Models that load when someone asks

A cluster with more models than GPUs can hold every model in its catalog and
only the wanted ones in its cards:

```yaml
scaling: { min: 0, max: 1, idle_timeout: 15m }
```

`min: 0` with a `max` above it means load on demand. The first request for the
model starts it and waits briefly; if it is not serving by then the client gets
a 503 with `Retry-After` and an honest "the model is loading", rather than being
told nothing serves it — a client told to retry succeeds where one told the
model does not exist gives up. Once nobody has asked for it in `idle_timeout`,
the replicas are stopped and the cards go back to the pool.

`min: 0, max: 0` is different and equally useful: configured, and deliberately
not running. A model with no `scaling` stanza at all gets that, because making
every unscaled model loadable on demand would let any client request start a
70B nobody intended to run.

Two things to weigh. A cold start is visible to whoever triggers it, so this
suits the long tail people are experimenting with rather than the two or three
models used daily — keep those at `min: 1`. And stopping a replica throws away
the KV cache prefix routing spent its effort filling, so an idle timeout wants
to be minutes rather than seconds.

When the cluster is full, a model somebody asks for will stop an idle one to
make room. Only idle replicas of other on-demand models: a model at `min: 1` is
a promise that it stays up, and breaking that to serve a different request would
make every guarantee in the catalog conditional on load. Only single-node
replicas, because evicting a multi-node one frees several machines and costs
tens of minutes to rebuild — a decision worth making deliberately. One at a
time. And never a replica that has just started, which is what stops two models
on a cluster with room for one from taking turns evicting each other.

For a GPU model, name the class and the width instead:

```yaml
profiles:
  - name: tp2
    engine: vllm
    gpu_class: h100-80g    # as `llmgw ctl nodes` reports it
    tp: 2
    priority: 100
```

If it does not place, ask:

```
llmgw ctl explain-placement qwen2.5-coder-3b-instruct
```

which names every candidate and why each was rejected, rather than leaving you
to infer it.

### LoRA adapters

A llama.cpp model can carry LoRA adapters, each asked for by its own name as
though it were a model:

```yaml
- name: qwen2.5-coder-7b-instruct
  source: http://mirror:8081/qwen2.5-coder-7b-instruct/5d567d26/model.gguf
  adapters:
    - name: coder-sql
      source: http://mirror:8081/coder-sql/v1/adapter.gguf
      scale: 1.0          # optional; 1 when left out
  profiles: [...]
```

Every replica of the model fetches every adapter and loads it at start, applied
to nothing. A request for `coder-sql` goes to the model's replicas with the
adapter applied to it alone; a request for the model gets the model. Adapter
names share the namespace of models and aliases, appear in `/v1/models`, and
follow key allowlists on their own: a key allowed `coder-sql` reaches that
adapter and not the model under it, while a key allowed the model reaches its
adapters too. Adding or removing an adapter is a change to the model; running
replicas load the new set when restarted (`llmgw ctl restart --model M`).
Besides the mirror page's model card, adapters are attached from the command
line, which reads the file first and refuses one that is not an adapter for
this model's architecture:

    llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
    llmgw ctl model adapters coder --remove coder-sql

The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
the adapter already merged in, which is what most "LoRA GGUF" repositories hold.

#### Adapters on vLLM, and loading them on demand

A vLLM model carries adapters the same way. There an adapter is a PEFT adapter
directory (`adapter_config.json` and `adapter_model.safetensors`): a mirror
version ending in `/`, a path on the nodes, or a Hugging Face repository id,
which vLLM downloads itself. A request names the adapter as its model, as on
llama.cpp; `scale` applies on llama.cpp only. Without anything more, every
replica registers every adapter at start (`--enable-lora --lora-modules`).
vLLM mixes one adapter per batch by default; raise that with `--max-loras` in
the profile's `args`, and `--max-lora-rank` (16 by default) for adapters of a
higher rank.

With many adapters, have each replica hold only some of them:

```yaml
- name: qwen2.5-7b-instruct
  source: Qwen/Qwen2.5-7B-Instruct
  max_loaded_adapters: 4      # per replica; vllm profiles only
  adapters:
    - name: support-de
      source: http://mirror:8081/support-de/v3/
    - name: support-fr
      source: someone/support-fr-lora
  profiles: [...]
```

Replicas then start holding none (vLLM runs with
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` and `--max-loras 4`). The first
request for an adapter has the router load it on a replica through vLLM's
`/v1/load_lora_adapter`, and the requests after it go to that replica while
it is not saturated; when it is, the next request goes where load sends it and
the adapter is loaded there too. Requests that arrive while an adapter is being
loaded wait for that load rather than starting another. A replica that holds
`max_loaded_adapters` already unloads the one least recently used, never one a
request is running on. Each node still fetches every adapter when a replica
starts, and links it as `llmgw-lora/<name>` in the directory engines run in,
which is the path the router has vLLM load it from — so adding an adapter
still needs the replicas restarted, as above. An adapter vLLM refuses (not
found, a rank above `--max-lora-rank`, not for this model) fails the request
with vLLM's reason.

What each replica holds is what the router loaded there, checked against each
replica's `/v1/models` when the router first meets it and every 30 seconds.
`llmgw_adapter_affinity_total` counts where requests found their adapter (hit,
joined a load, loaded it), `llmgw_adapter_loads_total` and
`llmgw_adapter_evictions_total` what that cost. llama.cpp keeps loading every
adapter at start: llama-server can change the scale of an adapter it started
with, not load a new one.

### Routing by what each replica's cache actually holds

With several replicas of a vLLM model, a request is fastest on the replica
that already holds its prompt's prefix in KV cache. The router guesses which
that is from what it has sent where (`router.prefix`). It can also be told:
vLLM publishes an event every time it stores or evicts a cache block, and with
`kv_events` on a profile llmgw routes that model by those events.

```yaml
- name: llama-3.1-8b-instruct
  source: meta-llama/Llama-3.1-8B-Instruct
  routing_profile: chat           # chat or long-context; batch ignores prefixes
  profiles:
    - name: h100
      gpu_class: h100-80g
      kv_events: true
```

How it works, so you know what you are paying for:

- The node agent starts vLLM with `--kv-events-config` publishing to a port
  on 127.0.0.1 that the agent listens on, `--prefix-caching-hash-algo
  sha256_cbor`, and `PYTHONHASHSEED=1729`. The events never leave the node:
  the agent keeps the set of blocks each engine holds and sends the changes,
  four times a second, up the stream it already has to the control plane,
  which hands them to the router. A new stream — a control plane restart, a
  failover — starts with a full copy. Nothing new is opened in the firewall.
- For each request to such a model, the router asks one of its replicas to
  tokenize the prompt (vLLM's `/tokenize`, with the request's messages and
  tools), hashes the token ids into blocks the way vLLM does, and prefers the
  replica holding the most leading blocks. Tokenizing at the engine is what
  makes the hashes agree: the chat template is part of the tokens, and only
  the engine runs it exactly. It costs a round trip before the request is
  placed and the template run twice; `router.kv_index.tokenize_timeout`
  (250ms) bounds it.
- When the index has nothing to say — no events have arrived yet, the
  request has an image or a `cache_salt` or names a LoRA adapter, the
  tokenize call failed — the router routes as it would without it: by the
  approximate tree if `router.prefix.enabled`, else by load. The imbalance
  guard of `router.prefix` applies to both.

Turning `kv_events` on changes how the engine is started, so replicas already
running keep running without it (and show as stale) until they are replaced.
Leave `--prefix-caching-hash-algo` out of the profile's `args`: an engine
hashing another way publishes blocks the router can never match. The agent
checks the first block of prompts against its own hashing, and if they never
agree it says so in its log and on `llmgw_kv_index_problem`, and the replica
is routed as though it published nothing.

Watch `llmgw_kv_index_decisions_total` beside `llmgw_prefix_decisions_total`:
`kv_hit` is the index deciding, `kv_miss_min_load` is it knowing no replica
holds enough (`router.kv_index.min_blocks`, 4 blocks), and the `fallback_*`
values say why it could not answer. `llmgw_kv_index_tokenize_seconds` is what
the tokenize call costs, and `llmgw_kv_index_blocks` what each replica holds.

This has been built and tested against vLLM's event format and its hashing as
its source defines them, with libzmq and msgspec publishing — not yet against
a running vLLM. Treat it as experimental until it has been.

### Models served outside the cluster

A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
another gateway — rather than replicas here. Nothing is placed for it; the
router sends its requests on, with the provider's key and the provider's name
for the model:

```yaml
- name: cloud-mini
  source: external
  external:
    url: https://api.openai.com          # without /v1
    model: gpt-4o-mini                   # what the provider calls it
    api_key_file: /etc/llmgw/secrets/openai.key
```

Named in another model's `fallbacks`, it is used only when the cluster has no
capacity left for that model:

```yaml
- name: qwen2.5-coder-7b-instruct
  fallbacks: [cloud-mini]
  ...
```

The key is read from its file on the control plane, never stored in the
catalog, and a key rotated in the file is used within seconds. A model whose key
file cannot be read is not offered. Everything else applies as to any model:
prices and budgets, rate limits, usage records, and key allowlists.

A provider's limits are per key. With several keys, one answering 429 need not
hold up the model: give them all, each with an optional weight (its share
against the others'), in place of `api_key_file`:

```yaml
  external:
    url: https://api.openai.com
    model: gpt-4o-mini
    api_key_files:
      - {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
      - {file: /etc/llmgw/secrets/openai-2.key}
```

Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
answers 429 is passed over while its breaker is open, and the request goes to
another. One that answers 401 (revoked, expired, mistyped) is passed over too,
and `external_key_refused` goes to the notification webhooks, naming the
endpoint and never the key. The usage records say which key answered by the
same name. A key whose file cannot be read is left out, and the others stay.

**Its prompts leave the cluster.** Decide who may send them: a key or project
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.

### Pictures from ComfyUI

Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
waits, and sends the pictures back as `b64_json`.

**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
GPU node like any other, started when asked for, and stopped when unused.

**The short way: a preset.** Some models come with llmgw, with their
workflow, custom nodes, packages and licence already found and tried:

```
llmgw ctl model presets
llmgw ctl model add flux-schnell     # pictures, Apache 2.0, a card of 24 GB, Volta or newer
llmgw ctl model add triposr          # 3D models from a picture, MIT, a card of 8 GB
```

A preset's profile says the card it needs, not a class, so it places on
whatever cards the site has. When the cluster's settings name no ComfyUI,
adding one sets `engines.comfyui` to the version this release installs, and
every node with an NVIDIA GPU installs it; a cluster that names a version,
or `""` for none, keeps it; the control planes then start again on that
setting, standbys first.

They also come by themselves. Once a node has a card that fits one, and the
catalog has nothing that makes what it makes, the leader adds it, on demand —
where the cluster already installs ComfyUI; it never sets `engines.comfyui`
by itself. A preset somebody removes stays out until it is added again.
`llmgw ctl model presets auto off` (the setting `presets.auto`) stops it.

The cluster page on the portal has a **Pictures and 3D models** card under
the models: each preset, the classes of card here it fits, whether it is in
the catalog and who added it, why the leader has not, an **Add** button (one
that would also install ComfyUI says so first) and the switch for adding
them by themselves. In the console, the models screen shows the same: **A**
adds one, **P** turns adding them by themselves on or off.
`llmgw ctl model presets` prints it. The rest of this section is for a model
of your own.

The node must offer the engine. Ask for it in the cluster's settings, and every
node with an NVIDIA GPU installs it:

```
llmgw ctl config set engines.comfyui 0.39.2
```

The agent fetches ComfyUI's source from GitHub and checks it against the
digest llmgw holds for that version. GitHub publishes none for a tag's archive,
so a version llmgw does not know is refused. The agent then builds a Python
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
dependencies are resolved as PyPI stood the day after the release. torch comes
from PyPI (CUDA 13), from PyTorch's CUDA 12.9 build where the driver is older
than 580, or from its CUDA 12.6 build on cards older than Turing (a V100):
torch's CUDA 12.8 and newer builds have no kernels for them. The install is
checked on the node's GPU, and takes about 10 GB.
`engine_overrides` can keep it off nodes that should not have it.

Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
its environment in `.venv`, readable by the user engines run as
(`llmgw-engine`). Name it in that node's `agent.yaml`; a node with one named
there is not given the installed one:

```yaml
engines:
  comfyui:
    dir: /opt/comfyui            # python: defaults to /opt/comfyui/.venv/bin/python
```

Then add the model:

```yaml
- name: flux-schnell
  source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
  comfyui:
    workflow: flux-schnell       # or flux-dev, or a JSON file
  scaling: {min: 0, max: 1, idle_timeout: 15m}
  profiles:
    - name: gpu
      engine: comfyui
      gpu_class: 6000-24g
      tp: 1
```

The source is fetched as for any model. It can be one file, such as the
single-file checkpoints the built-in workflows load. It can also be a mirror
set (a URL ending in `/`) for a model in several files, such as a diffusion
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
own name, so a workflow of your own can load each one. A set must name the
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
is named by its own file name.

ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
listens on the replica's port, demands the replica's key, and lets through only
what the router asks: queueing, waiting, fetching a picture and cancelling.
Uploads and node installs are refused. Each replica has its own directory,
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
`--disable-smart-memory` frees the card between pictures, at the cost of loading
the model again for each one.

**On a ComfyUI server outside the cluster.** Make it an external model with
`api: comfyui`:

```yaml
- name: flux-schnell
  source: external
  external:
    url: http://10.0.0.147:8188                  # ComfyUI, started with --listen
    model: flux1-schnell-fp8.safetensors         # the checkpoint, as ComfyUI lists it
    api: comfyui
    workflow: flux-schnell                       # or flux-dev, or a JSON file
```

Like any external model, its requests count as leaving the cluster.

`flux-schnell` (the default) and `flux-dev` are built in. They suit the
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
For any other model, export the workflow from ComfyUI with **Save (API)** and
name the file. In the file, put placeholders where the request's values go:

- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
- `{{width}}` and `{{height}}` are the size.
- `{{n}}` is the batch size.
- `{{seed}}` is the seed.
- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
  request leaves them out.
- `{{checkpoint}}` is the `model` above.

A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
before the picture is sent.

Requests for pictures pass the same checks as any other request: keys,
allowlists, limits, admission, and the guardrails on the prompt. A client that
leaves takes its workflow off the ComfyUI queue.

Each request's usage row records how many pictures it made (`llmgw ctl request
ID` shows them). A picture's price is set with `per_image`, in micro-units like
the other prices, and is charged on top of the prompt's tokens:

```yaml
auth:
  pricing:
    flux-schnell: {prompt_per_1k: 0, per_image: 20000}
```

An external ComfyUI on a card that llmgw also places models on should be
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
the card after each picture, and llmgw sees that memory as somebody else's.

### Federating clusters

Another llmgw cluster — another site, another team's GPUs — can be a provider
of this one, and llmgw knows it is one. Ask the other cluster's operator for a
key in one of their projects (`llmgw keys create`), put it in a file on this
control plane (on every control plane, with HA), and add the peer:

```
llmgw ctl federation add east --url https://llm.east.example --key-file /etc/llmgw/secrets/east.key
llmgw ctl federation list
llmgw ctl model fallbacks qwen-7b east/qwen-7b   # overflow goes to east
```

- **Its models, named PEER/MODEL.** What the peer's `/v1/models` offers that
  key is offered here as `east/qwen-7b` and so on, always with the prefix —
  never under the peer's own name, even when nothing here uses it, so
  whether a request leaves the cluster never depends on what else is in the
  catalog, and every usage record and `x-llmgw-served-model` header says which
  site answered. A model here called `east/...` wins over the peer's of the
  same name. The peer's aliases, and models the peer has from peers of its
  own, are not taken: federation is one hop.
- **Kept in step.** Each peer is checked every `federation.interval` (10s):
  its `/ready`, timed — the latency `ctl federation list` shows — and its
  `/v1/models`. A model it adds appears here; one it drops goes, unless a
  model here still falls back to it, when it is kept *withdrawn* (in the
  catalog, sent nothing) until the fallback is changed. `ctl federation sync
  east` checks now.
- **Taken out when down.** A peer whose `/ready` fails, which refuses the key,
  or which does not answer is sent nothing until it answers again; its models
  stay in the catalog. A peer's `/ready` counts only what it serves itself, so
  two clusters federating each other are not each ready because of the other.
- **Overflow.** Named in a model's fallbacks (`ctl model fallbacks`, for
  models added at runtime), a peer's model is used only when the model has
  no capacity here — as for any external model.
- **What stays here, stays here.** A peer's model is an external model: a
  project with `keys residency --in-cluster`, or under a guardrails policy with
  `no_store`, is refused it by name and never sent to it as a fallback.
- **One key, and who is asking.** The peer sees this cluster's key, so its
  usage, limits and budgets are one tenant's. The request's `X-Request-Id`
  goes with it, so one id finds the request on both sides. With
  `--send-tags` the request also carries `x-llmgw-tags:
  federated_from=<federation.name>,org=…,team=…,project=…` (ids), for the
  peer's usage records to attribute it; off unless given, because it tells
  the other site who here is asking.
- **No loops.** Each cluster has an id (`ctl federation list` shows it), and a
  request sent to a peer carries `x-llmgw-hop` with the ids it has passed
  through. A cluster refuses a request that has already passed through it
  (508, `federation_loop`), and does not send one to a peer it came from —
  so two clusters falling back to each other try each once.
- `llmgw_federation_peer_up`, `_latency_seconds` and `_models` are on
  `/metrics`; the console has a Federation view. `ctl federation remove east`
  takes the peer and its models away, refused while a fallback names one.

Not yet: models declared in server.yaml cannot name a peer's model as a
fallback (the catalog in the file is checked before the peers are read), and
a peer behind a private CA needs that CA in the system's trust store.

### Trying new weights on some of the traffic first

When a model's repository publishes new weights, `llmgw ctl model update`
swaps them in. To see how they serve first, run them as a canary:

```
llmgw ctl model canary qwen-7b --percent 10    # fetched, started beside, given 10%
llmgw ctl model canaries                       # how each side has served
llmgw ctl model canary qwen-7b --percent 50    # more
llmgw ctl model promote qwen-7b                # the model moves to them
llmgw ctl model canary qwen-7b --abort         # or not
```

- **What runs:** the new files are fetched into the mirror and checked as
  for an update, then run as `qwen-7b-canary` — the same model in everything
  but its weights, one replica. Clients cannot name it; it gets its share
  of `qwen-7b`'s requests.
- **Who goes where:** a session (`x-session-id`, or the `user` field)
  stays on one side, so a conversation does not change weights halfway.
  `x-llmgw-variant: canary` or `stable` pins a request to one side; the
  response says which served it in the same header, and the usage record
  names `qwen-7b-canary` as the model that served it.
- **Rolled back by itself:** once the canary has served `--min-requests`
  (50), it is taken out if its error rate is `--max-error-increase` (0.05,
  five points) above the model's, or its mean time to the first token — or
  to the answer, unstreamed — is `--max-ttft-ratio` (1.5) times the
  model's. `canary_rolled_back` goes to the notification webhooks. The
  comparison is the router's own count since the canary began, kept per
  control plane.
- **Promote** is the update the canary was a trial of: the model's replicas
  are replaced one at a time, and `llmgw ctl model rollback` goes back.

### Quality suites, and holding new weights to one

A canary says whether new weights serve as fast and as reliably; it does not
say whether their answers are as good. A republished quantisation or a chat
template that no longer matches can make a model faster and worse at once. A
quality suite is a handful of questions with known answers, asked of a model
and scored:

```
llmgw ctl eval --suites                                   # what the control plane can run
llmgw ctl eval qwen-7b --suite arithmetic                 # by name, on the control plane
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
llmgw ctl eval --history qwen-7b                          # the runs kept
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d        # one, case by case
```

It prints each case's score and why, then the suite's: the cases' scores
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
`--min-score`); it exits non-zero below it, so a script can stop on it.

- **How it is asked:** on the control plane, through the router as the
  gateway's own requests, the way a replay is — past admission, quotas,
  accounting and the guardrails, so the score is the weights' and not a
  tenant's limits'. It reaches the model exactly as a client's request
  would, a canary or a model loading on demand included. Answers are asked
  at temperature 0 unless a case says otherwise, and a reasoning model's
  `<think>` block is not part of its answer.
- **Where suites come from:** `--suite` is a file on the machine running
  `ctl`, sent whole, or the name of one in the control plane's
  `control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
  models: `arithmetic` (sums and word problems, each scored by the number in
  the answer) and `json-format` (answers asked for as JSON of a given shape);
  their files are in `deploy/eval/`, to copy and change. A file in the
  directory with the same name replaces the built-in one.
- **What is kept:** every run, in the store, with the model, the repository
  and revision the answering weights came from (when known), the suite's name
  and sha256, when, the score, and each case's answer (up to 2,000
  characters), score and reason. A run on a canary is filed under the model
  it is a canary of. Runs are not pruned.

A suite is YAML (or JSON lines: one case on each line, and optionally a line
without a prompt that holds the suite's own settings):

```yaml
name: support            # the file's name unless given
threshold: 0.8           # the score to pass at, 0 to 1; 0.8 unless given
judge: llama-70b         # grades the cases scored by a rubric
system: You are a support agent for Acme.   # every case's, unless it has one
max_tokens: 256          # every case's; 512 unless given
cases:
  - id: refund-window
    prompt: How many days do customers have to ask for a refund?
    number: 30                     # the last number in the answer, ±tolerance
    tolerance: 0
  - prompt: Which plan includes SSO?
    contains: [Enterprise]         # every string must appear
    ignore_case: true
  - prompt: Reply with only the word OK.
    exact: OK                      # the whole answer, white space around it aside
  - prompt: What is today's ISO date format?
    regex: '\d{4}-\d{2}-\d{2}'     # matches somewhere in the answer
  - prompt: Give the order as JSON with "id" and "total".
    json_schema:                   # the answer, read as JSON, satisfies it
      type: object
      required: [id, total]
      properties: { id: {type: string}, total: {type: number, minimum: 0} }
  - messages:                      # a conversation instead of a prompt
      - { role: user, content: My card was charged twice. }
    judge: Apologises, and says the duplicate charge is refunded within 5 days.
    weight: 3                      # counts three times; 1 unless given
```

Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
still read. The schema supports `type`, `properties`, `required`,
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
is read rather than passed unchecked. A `judge` case asks another model on the
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
got no answer scores 0 and says why.

**Holding updates to a suite.** `model update` and `model promote` take
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
definition added with `ctl model add` — so every update and promotion of it
is held to it, from `ctl`, the console or anywhere:

```
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
llmgw ctl model update qwen-7b              # asked the suite first
llmgw ctl model update qwen-7b --skip-eval  # once, without it; the job says so
llmgw ctl model gate qwen-7b --off
```

- **Promote** asks the canary the suite before anything moves; below the
  score the job fails with the score, and the canary stays, with its share,
  until you take it out.
- **Update** runs the new weights first as `MODEL-canary` given none of the
  model's requests (hidden from clients like any canary), asks it the suite,
  then either promotes it — the model's replicas are replaced one at a time,
  as for any update — or takes it out again, leaving the model on its old
  weights. That needs room for one more replica while it runs; a model that
  already has a canary is promoted or aborted first. Without a suite, an
  update swaps the weights in as it always has.
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
  kept with the job's id beside it.

The console has it on a model's menu: **Ask it a quality suite…** (a suite the
control plane has, the score as it comes in), **Its quality runs**, and **Hold
its updates to a quality suite…**.

### How many replicas, and how much each one carries

Replica count is changed while the cluster runs, without editing the
definition:

```
llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
```

Models added at runtime only. One declared in `server.yaml` is read again on
every start, so a number set here would hold until the next restart and then
revert — change it in the file instead.

Between `min` and `max` a model scales with its load: requests waiting at
every replica add one, a quiet spell takes it back. For load that comes by the
clock, a schedule says so ahead of time:

```yaml
scaling:
  min: 1
  max: 4
  prewarm: 30m                 # start a window's replicas this long before it opens
  schedule:
    - {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
    - {from: "22:00", to: "02:00", min: 1}     # past midnight
```

Times are in `control.timezone` (`Europe/Istanbul`; empty is the control
plane's own). The first window that holds wins; outside every window, `min`
and `max` apply. An on-demand model (`min: 0`) with a window is kept loaded
through it, and loaded again on the first request after. Days are `mon` …
`sun`, `weekdays` or `weekends`.

How much traffic each replica is given is the profile's `weight`, and guessing
it is how a cluster ends up with one saturated replica beside two idle ones.
Measure it instead:

```
llmgw ctl bench qwen2.5-coder-3b-instruct --save
```

Concurrency climbs until the latency target stops being met, and the last level
that held it is the capacity — peak throughput arrives long after latency has
become unusable, so throughput alone would overstate what the hardware serves.
It prints the `weight` to put in the profile, and `--save` records the figure
against the model so the dashboard can show what the cluster is measured to
carry rather than what somebody typed.

This drives real traffic and competes with real requests. Run it against a
cordoned replica, or in a window where that is acceptable.

An engine that publishes no metrics endpoint is measured from the outside only.
The run says so, and queue depth and KV occupancy are then not part of the
answer.

## 8. The first request

```
curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "coder", "messages": [{"role":"user","content":"hello"}]}'
```

Or open `/chat` in a browser and use it without a key at all — the page carries
your session, and the request still goes through admission, rate limits, budget
and accounting exactly as an API call does.

### What the client port answers

| Path | What |
|---|---|
| `POST /v1/chat/completions`, `/v1/completions`, `/v1/embeddings` | OpenAI's, served by the models in the catalog |
| `POST /v1/responses` | OpenAI's Responses API, translated to chat completions: tools, `text.format` (structured output), `input_image` (by URL or `data:` URL), and `previous_response_id` — the conversation is kept as a thread, so the project must keep conversations (`llmgw keys threads`); `store: false` keeps none |
| `POST /v1/messages`, `/v1/messages/count_tokens` | Anthropic's Messages API, translated to chat completions: text, images, tools and their results, the system prompt, streaming; the key in `x-api-key` or `Authorization`. Extended thinking, prompt caching and server tools are ignored |
| `POST /v1/rerank`, `/v1/score` | passed through to an engine that serves them: vLLM with a reranker or a cross-encoder |
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper |
| `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories |
| `/v1/files`, `/v1/batches` | batches; see [Batches](#batches) |
| `/v1/3d/generations` | 3D models, as jobs asked after; see [Jobs](#jobs-3d-models) |
| `GET /v1/models` | the models this key may use |

## Watching it

The control plane exports both halves on one endpoint: what the cluster is
serving — requests, latency, time to first token, tokens, admission, per
endpoint — and what it is made of, per node and per GPU, including memory,
temperature, ECC faults and utilisation.

**The dashboard is the answer for most clusters.** `/admin` shows capacity per
GPU class, free whole nodes per fabric, which replicas are holding hardware
nothing is asking for, who is using the cluster and what they are running, and
the same node and model tables `ctl` prints — with no monitoring stack to stand
up beside it. One binary, no dependencies, which is the point.

Prometheus is worth adding when you want history, alerts, or one view across
several clusters. The dashboard shows what is true now; it keeps nothing.

Metrics then get their own listener, because the admin one also carries pprof
and pprof can dump the process heap, where API keys sit in plaintext:

```yaml
admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof — keep local
metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach
```

One scrape target for the whole fleet. The node and GPU figures come from the
control plane, which already holds them, so there is nothing to install on the
nodes and no Prometheus configuration per machine.

A starting configuration, with the alerts worth having and a provisioned
dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
to run it, which is not the control plane: monitoring that dies with the thing
it monitors tells you nothing at the moment you need it.

Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
rather than zero when the driver does not report it — a flat zero across a
fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.

### Token by token, and SLOs

Beside time to the first token, every streamed answer is timed per token:
`llmgw_tpot_seconds` (each output token after the first),
`llmgw_inter_token_seconds` (the gaps between streamed events, which show the
stalls a mean hides) and `llmgw_output_tokens_per_second`, by model and pool,
and `llmgw_tenant_tpot_seconds` and `llmgw_tenant_output_tokens_per_second` by
tenant.

A model can say what it promises:

```yaml
slo: {ttft_p99: 2s, tpot_p99: 80ms}
```

Then its recent p99 — the last five minutes, once there are twenty streamed
requests — counts in admission: a model past its SLO is treated as saturated
by as much as it is past, so the lower bands wait while the queue and the
cache still look healthy. `llmgw_model_slo_seconds` publishes the SLOs, and
the `llmgw-slo` alerts in deploy/monitoring fire when a p99 stays past one for
fifteen minutes.

### Being told

Prometheus alerts travel once `alertmanager.yml` names a receiver; it ships
templates for Slack, e-mail, PagerDuty and a webhook of your own, commented
out until you choose.

A few things are worth a message even without a monitoring stack, and the
control plane sends those itself:

| Event | When |
|---|---|
| `model_parked` | A model failed every start in a row and is no longer retried (`ctl retry` tries again). |
| `node_lost` | A node was silent past `control.lost_after`; its replicas are rebuilt elsewhere. |
| `budget_spent` | A key, project, team or organization reached its budget; priced models are refused. |
| `update_available` | A model's registry repository published different weights. Once per revision. |
| `canary_rolled_back` | A model's canary served worse than the model and was taken out. |
| `judge_unavailable` | A guardrails judge gave no verdict: with `fail: closed` requests are being refused, with `open` served unjudged. |

```yaml
notify:
  webhooks:
    - url: https://hooks.slack.com/services/…      # Slack, Mattermost, Rocket.Chat
      format: slack
      events: [model_parked, node_lost]
    - url: https://ops.example.com/llmgw            # your own receiver
      secret_file: /etc/llmgw/notify-secret          # signs the body
```

A `json` webhook (the default) gets the event as JSON — `kind`, `at`,
`subject`, `message`, `detail` — with `X-Llmgw-Event` naming the kind and,
when `secret_file` is set, `X-Llmgw-Signature: sha256=<hex>`, the HMAC-SHA256
of the body under the file's contents; check it before believing the body. A
webhook that does not answer is tried three times and then given up on, and the
same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.

Setup asks for them: **Notification webhooks** on its first page, or
`--webhook` to `llmgw server init`, as often as there are webhooks:

```
sudo llmgw server init --admin ops@example.com \
  --webhook https://hooks.slack.com/services/… \
  --webhook json:https://ops.example.com/llmgw
```

An address of Slack's, or one with `/hooks/` in it (Mattermost, Rocket.Chat)
or ending in `/slack`, is sent Slack's `{"text": …}`; any other the event as
JSON; `slack:` or `json:` in front says which. They are added to the
cluster's settings beside any already there — run again, nothing is added
twice — and once the control plane runs on them, each is sent a test, whose
answer setup prints. Events to choose and a signing secret are set in the
settings afterwards (`llmgw ctl config edit`).

### Tracing

Metrics say that a model took four minutes to start; a trace says where they
went. Give the control plane and every agent the same OpenTelemetry collector,
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:

```yaml
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
  endpoint: http://otel.internal:4318
  sample_ratio: 1          # of requests that arrive without a decision of their own
```

`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
neither, nothing is traced and nothing is sent: every span is a no-op, and a
command to a node carries no trace context.

What is traced:

| Trace | Spans |
|---|---|
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
| An engine install | `install engine` on the node, a trace of its own. |

So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.

A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as
from a control plane that is not tracing. The agent needs its own `tracing`
block either way — without one its spans go nowhere, and the control plane's
trace simply ends at `send start`.

## PostgreSQL, when SQLite is no longer the right shape

SQLite is the default and stays the right answer for most installations: one
file, no server, and a request path that barely touches it — usage is written
in batches, budgets are counted in memory, authentication is cached.

Two things change that. Conversations and agent notes read and write on every
turn, which is the first per-request load this store has had; and SQLite takes
one writer at a time, so those turns serialise. And a second control plane is
impossible while the store is a file on one machine's disk.

On the control plane, one command does all of it — installs PostgreSQL if the
machine has none, creates its role and database with a generated password kept
in `/etc/llmgw/secrets/store-dsn`, stops the gateway, copies the SQLite store
into it so every account and key survives, points `server.yaml` at it, and
starts the gateway again:

```
sudo llmgw server init --postgres
```

It can be given on the first run too, and then there is never a SQLite store.
The rest of this section is what that command does, for a PostgreSQL somewhere
else.

The setting is the same one, and the address decides:

```yaml
auth:
  store: "@/etc/llmgw/secrets/store-dsn"
```

A path is SQLite, a `postgres://` URL is PostgreSQL, and `@` followed by a path
reads the address out of that file — which is how a DSN carrying a password
stays out of a configuration that gets copied, diffed and pasted into issues.
Keep that file at `0600` and owned by the user llmgw runs as.

Moving an existing installation is a copy, not a fresh start. Keys are stored
hashed, so starting fresh means reissuing every one of them and being unable
to say which was which:

```
sudo systemctl stop llmgw-server
llmgw keys move-store --from /var/lib/llmgw/llmgw.db --to "@/etc/llmgw/secrets/store-dsn"
# point auth.store at the new address, then
sudo systemctl start llmgw-server
```

Stop the gateway first. The copy makes no attempt to follow a store being
written to: rows landing after the table they belong to has been passed would
simply be absent, and nothing would say so. The destination must be empty, and
the source is left untouched — so the way back is to point the configuration at
the file again.

On a machine that also runs the control plane, PostgreSQL's defaults assume it
owns the box. A gateway swapped out to give the database cache is a worse
outcome than a smaller cache, so size it to the machine. `server init` does this
for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
scaled to it, and `max_connections = 40`. A store with a large knowledge base
wants a machine of its own, where its vector index fits in memory.

## Tagging requests

A client can say what a request was for — which feature, which customer,
which environment — and the spend can then be found by it inside a project:

```
curl http://gateway:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-tags: feature=search, env=prod" \
  -d '{"model": "qwen-1.5b", "user": "customer-42",
       "metadata": {"feature": "search"},
       "messages": [{"role": "user", "content": "hello"}]}'
```

- **`metadata`** is OpenAI's: an object of strings, up to 16 keys. **`x-llmgw-tags`**
  is the same as a header, for clients that cannot change the body; for a
  key both give, the header wins. A key is 1 to 64 letters, digits, `_`,
  `.` or `-`, a value at most 512 bytes; anything else is refused with 400
  rather than dropped, since a tag lost silently is spend nobody finds.
- **`user`** is OpenAI's end-user field: the client's own name for whoever
  it serves, not an account here. It is passed on to the engine, as OpenAI
  does; tags are not.
- **Where they show:** every usage record keeps them, and `llmgw ctl request
  <id>` prints them. For Prometheus, name the tags worth a series:

  ```yaml
  router:
    tag_labels: [feature, env]
  ```

  `llmgw_tagged_requests_total{tag,value,outcome}` and
  `llmgw_tagged_tokens_total{tag,value,direction}` count those, the first
  100 values of each apart and the rest as `other`: a tag is the client's
  to set, and every value is a series.

## Keeping requests, and replaying them

To follow a request up — what exactly was sent, what came back — or to try
real traffic on another model, a project can keep its requests:

```
llmgw keys requests --project Acme/ml/train --retain-hours 72            # every one
llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05
llmgw ctl request req_3f2a… --body                   # what was sent and answered
llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct   # the same, to another model
llmgw keys requests --project Acme/ml/train --off
```

- **What is kept:** the request as the model was sent it — after the
  guardrails' redaction, with any stored conversation spliced in — and the
  answer as the client got it, up to 128 KB of it. A share (`--sample`) is
  the same requests every time for the same ids.
- **Replay** sends it as the gateway itself: past the limits, the budget and
  the guardrails, which judged it the first time, and charged to nobody.
- **Never** for a project under a guardrails policy with `no_store`. Kept
  requests past their hours, or of a project turned off, are removed hourly.
  The console has it as **Keep requests…** on a project.

## Reusing answers

A project that asks the same thing again and again — an evaluation run over
the same prompts, embeddings of documents that do not change, an agent that
retries — can have the same request answered from memory rather than by the
model:

```
llmgw keys cache --project Acme/ml/train --ttl 1h          # reproducible answers
llmgw keys cache --project Acme/ml/eval --ttl 24h --any    # every answer
llmgw keys cache --project Acme/ml/train                   # how it stands
llmgw keys cache --project Acme/ml/train --off
```

- **What is the same request:** the same project, the same endpoint, the same
  model (by its catalog name, whichever alias was used) and the same body,
  with `stream`, `stream_options`, `user`, `metadata`, `store` and
  `service_tier` left out and the keys in any order. Keys of another project
  never see the answer, whatever they send.
- **Which answers are reused:** a chat or completion asked at `temperature`
  0 or with a `seed`, and every embedding, rerank and score, which sample
  nothing. A request sampled above temperature 0 with no seed asks for a new
  answer, and is asked anew — unless the project says `--any`. Only a whole
  200 from the model that was asked for is kept: not an error, not a
  fallback model's or a canary's answer, not one a guardrail changed or
  stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
- **Streams:** an answer given whole is streamed to a client that asks for a
  stream, as one chunk per choice with the usage chunk if it asked for that;
  a streamed answer is kept put back together, and served whole to a client
  that does not stream. Through `/v1/messages` and `/v1/responses` too.
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
  `x-llmgw-cache: no`, is always asked anew. Every answer of a project that
  reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
  can be reused), and a hit says its `Age` in seconds.
- **Policies still apply:** the limits are checked and the guardrails judge
  the request before the cache is looked in, and the output policy judges a
  reused answer as it would a new one. A request continuing a stored
  conversation, or under a `no_store` policy, is never answered from the
  cache nor kept in it.
- **What a hit costs:** nothing. It is recorded as a request — status 200,
  the model, the key, the rate limit's count — with no tokens and no cost,
  marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
  budget and no token limit. The answer itself carries the usage the model
  reported when it was first given.
- **Where it lives:** in each gateway process's memory, at most
  `router.cache.max_bytes` (128 MB) of answers for all projects together,
  those used longest ago dropped first; nothing is written to the store, and
  a restart begins it again. With two control planes behind a balancer each
  keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
  applies to what is already kept.

```yaml
router:
  cache:
    max_bytes: 268435456     # 256 MB
    max_entry_bytes: 2097152 # 2 MB
```

`llmgw_cache_requests_total{model,result}` counts hits, misses and
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
console has it as **Cache answers…** on a project.

## Batches

Work that can wait — a night's classification, embeddings for a document
store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.

```
curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
  -d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"
```

Each line of the file is `{"custom_id": …, "method": "POST", "url":
"/v1/chat/completions", "body": {…}}`, at most 50 000 of them
(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`).

- **How it is served:** each line as a request of the key that made the
  batch — its limits, its budget, its guardrails — in the lowest admission
  band, behind every interactive request, `router.batch.concurrency` (4) at
  a time. A rate limit or a model still loading is waited out; a spent
  budget fails the lines left.
- **What comes back:** answers in an output file, failures in an error
  file, both OpenAI's form, kept `router.batch.retain_days` (30). Each
  request is a usage record tagged `batch=<id>`, so `llmgw ctl request`
  finds it.
- **Who sees it:** files and batches belong to the key's project; every key
  of the project sees them, no other does.
- **Surviving a restart:** every answered line is kept as it is answered,
  and whichever gateway holds the batch's lease carries on from there; a
  control plane that stops hands its batches to another within two minutes.
- **What it keeps:** a batch is prompts and answers stored until deleted
  or past their retention — a project under a guardrails policy with
  `no_store` is refused it.

## Jobs: 3D models

A 3D model from a picture takes minutes on a card that first has to be
emptied of whatever else was on it. No client should hold a connection for
that, so it is a job: queued, answered with an id, asked after, its result
downloaded.

```
curl http://gateway:8080/v1/3d/generations -H "Authorization: Bearer $LLMGW_API_KEY" \
  -d '{"model": "trellis2", "image": "<base64>"}'
# → 202 {"id": "job_…", "status": "queued", …}
curl http://gateway:8080/v1/3d/generations/job_… -H "Authorization: Bearer $LLMGW_API_KEY"
# → {"status": "completed", "output_file_id": "file-…", "content_type": "model/gltf-binary", "usage": {…}}
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY" -o model.glb
curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorization: Bearer $LLMGW_API_KEY"
```

- **Status:** `queued`, `running`, `completed`, `failed` (with `error`),
  `cancelling`, `cancelled`, `expired` (not finished within 24 h).
  `progress` carries what the engine said of the work so far, once it says
  anything; `GET /v1/3d/generations` lists the caller's jobs. While the
  model waits for another on-demand model to give up the card,
  `progress` is `{"state":"waiting_gpu","held_by":"flux-schnell","eta_seconds":N}`:
  N is when the holder is stopped (idle and old enough to evict, or at its
  idle timeout) plus how long the model's last start took, and is left out
  when that is not known. A request made directly gets the same as a 503
  with `X-Llmgw-Waiting-For`, `X-Llmgw-Eta-Seconds` and `Retry-After`.
- **Refused at the door:** no key, a model that is not there or not in the
  key's allowlist, a request the model cannot serve. Refused when served,
  and said as the job's error: a spent budget, the guardrails, a model
  that will not come up.
- **How it is served:** the request is kept as it was sent and replayed by
  whichever gateway holds the job's lease, through the same path a
  client's request takes — limits, admission in the key's band, guardrails,
  accounting — by the leading control plane, since a standby has nothing
  to send it to. A model said to fit nowhere is asked for again for five
  minutes before that is the job's error: an agent re-registering says so
  for a few seconds. It is one usage record under the job's id, tagged
  `job=<id>`, charged like any request; `GET` shows the usage once it is
  done. A model still loading, a rate limit or a full queue is waited out
  for as long as the job has: nobody is holding a connection, which is the
  point.
- **Surviving a restart:** a gateway that stops hands its jobs to another
  within two minutes; a job it was running is run again from the start.
- **What it keeps:** the request as sent, and the result as a file owned
  by the key's project, kept `router.batch.retain_days` (30) like a batch's
  files; the job's record goes with it. A project under a `no_store`
  policy is refused jobs, as it is batches.

A request may also send `seed` and `resolution` (the model's own grid, a
multiple of 32 from 64 to 2048). The picture should show one object on a
plain background: the model makes a mesh of what it sees, background
included.

### A model that makes them

A 3D model is made by a ComfyUI model whose `comfyui.output` is `meshes`
(see [Pictures from ComfyUI](#pictures-from-comfyui) for engine `comfyui`).
Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the
uploaded picture, which a `LoadImage` node reads; `{{seed}}`,
`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that
saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB,
vertex colours kept.

TripoSR (MIT), on any card of 8 GB, comes with llmgw: `llmgw ctl model add
triposr`. What that adds is this entry, and its workflow is built in:

```yaml
- name: triposr
  source: hf:stabilityai/TripoSR/model.ckpt
  comfyui:
    workflow: triposr              # built in; a model of your own names a JSON file on every control plane
    output: meshes
    nodes:
      - repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR
        commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3
        requirements: [omegaconf, trimesh, scikit-image, imageio,
                       "transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers,
                       rembg, onnxruntime]
  license:
    id: MIT
    name: MIT
    commercial: true
    url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE
    notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..."
  scaling: {min: 0, max: 1, idle_timeout: 10m}
  profiles:
    - {name: gpu, engine: comfyui, tp: 1, gpu: {min_compute_capability: 7.0, min_memory_gb: 8}}
```

```json
{
  "1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
  "5": {"class_type": "LLMGWPrepareForMesh", "inputs": {"image": ["1", 0], "mask": ["1", 1],
        "foreground_ratio": 0.85, "background": 0.5, "remove_background": "auto", "rembg_model": "u2net"}},
  "2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
  "3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["5", 0],
        "geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
  "4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
}
```

`LLMGWPrepareForMesh` comes with the gate: it cuts the object out — by the
picture's alpha, or by rembg where it has none — and centres it on a plain
background, without which the background becomes geometry.

- **Custom nodes** are named by https repository and full commit (a branch
  is refused: it moves). The gate puts each in the replica's own
  `custom_nodes` and installs its `requirements` beside the replica, never
  into ComfyUI's environment. A package named there is installed at the
  range named; every other one at the version ComfyUI already runs. A
  package named is first on ComfyUI's path, so its range must suit ComfyUI
  as well. TripoSR's checkpoint loads with transformers 4, which ComfyUI's
  `>=4.50.3` allows.
- **Progress:** the gate listens to ComfyUI's websocket and the router
  writes what it hears into the job: `{"state": "queued", "ahead": 2}`,
  then `{"state": "running", "node_class": "KSampler", "step": 12,
  "steps": 30, "percent": 40}` for a node that counts its steps, and
  `done` at the end.
- **Licence:** a model's `license` is shown in `/v1/models`, in the job and
  on the portal, and written into the GLB's `asset.extras.license`, since
  the file travels without the API. Choose the model for its licence before
  its quality when what it makes is shipped. TRELLIS.2 needs Meta's gated
  DINOv3 and, as published, a background remover under a non-commercial
  licence; Hunyuan3D's licence excludes the EU, the UK and South Korea.
- **Price:** `per_mesh` in `auth.pricing`, beside `per_image`; each usage
  row says how many meshes it made (`meshes`).
- **Cards:** the strongest models today (TRELLIS.2, Direct3D-S2) need
  flash-attention, which runs on Ampere and newer. A Turing card such as
  the RTX 6000 runs TripoSR.

People make them at `/3d` on the portal: they choose a picture and watch
the job, and the mesh is shown in a viewer (model-viewer, served from the
gateway) with a download link and the licence. `llmgw ctl 3d list`, `get`
and `cancel` show and stop every owner's jobs.

## Conversations across models

Off everywhere until a project asks. llmgw otherwise stores no message content
at all — usage rows carry token counts, costs and latencies, never text, and
even its own chat page keeps the conversation in the browser. A gateway that
keeps nothing anyone typed is a much easier thing to have in front of a team,
so keeping it is a decision someone makes rather than a default they discover.

```
llmgw keys threads --project default --retain-hours 24
```

The retention is required. A conversation store with no expiry is a transcript
of everything a team ever typed, kept forever because nobody chose a number.
The window is measured from the last turn, so a conversation still in progress
does not expire underneath it.

A client then sends a thread id and only its new turn:

```
curl $BASE/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-thread: my-conversation" \
  -d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```

A thread belongs to the project, not to the key: **any key of the project that
knows the id reads the conversation.** Use ids nobody can guess — a UUID, not
`my-conversation` — of at most 128 letters, digits and `- _ . :`.

llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
their conversation.

When the history does not fit the model now serving, the oldest turns are
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
many. System messages are never dropped, and neither is the turn the caller
just sent.

`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
person who said the thing should be able to unsay it without asking an
operator to read it first. A thread belonging to another project answers the
same as one that never existed.

Turning it off again stops new storage, and existing conversations go at the
next hourly prune:

```
llmgw keys threads --project default --off
```

## Memory notes for agents

Also off until a project asks, for the same reason. Where conversations are what
was said, notes are what an agent chose to keep, and they persist across
conversations:

```
llmgw keys memory --project default --retain-hours 720
```

The retention is required and is measured from the last write to a note. An agent
then saves, searches and deletes notes through `/v1/memory` or, as tools, through
the MCP server at `/mcp` (`remember`, `recall`, `forget`). Both need the API key
of the project, and notes are private to the project and to the agent id sent in
`x-llmgw-agent`. A note in another project or agent answers the same as one that
does not exist. llmgw does not add these tools to requests itself.

Turning it off stops new storage, and existing notes go at the next hourly
prune:

```
llmgw keys memory --project default --off
```

Without authentication, `threads.store` and `threads.retain_hours` in
`server.yaml` keep both conversations and notes in one shared pool. That is for
one trusted machine only; see [development.md](development.md).

## Three machines, each a control plane

The screens' three machines are high availability for the gateway as well as
the store: the two database machines are made control planes too, standing
by while the first leads — what `server add`, below, does for one machine.
By command, `--standbys` with `--patroni`, and `--vip` for the floating
address:

```
sudo --preserve-env=SSH_AUTH_SOCK llmgw server init --admin you@example.com \
    --patroni ubuntu@10.0.0.6 --patroni ubuntu@10.0.0.7 \
    --standbys --vip 10.0.0.160/24 --no-node --no-model
```

| Machine | Runs |
|---|---|
| the first | control plane, leading at first; model mirror; etcd |
| the other two | control plane, standing by; PostgreSQL under Patroni; etcd |

etcd votes on all three and PostgreSQL is on two, so any one machine can be
lost: a standby leads within seconds, Patroni promotes the standby database
if it was the primary, and the floating address follows whichever leads. The
mirror's management API answers the three control planes' addresses, not
only its own machine's, so a standby that comes to lead can still fetch and
add models. The mirror itself stays on the first machine: while it is down,
models the nodes already have keep serving, and new ones wait for it.

## Two control planes

Once the store is replicated (`server init --patroni`, above), one command on
the control plane makes another machine a second one — or **Do → Add a
control plane** in the console, which asks the same two questions:

```
sudo --preserve-env=SSH_AUTH_SOCK llmgw server add ubuntu@10.0.0.150 --vip 10.0.0.161/24
```

Over ssh, as `node add` works, it gives the machine this llmgw, a certificate
from the cluster's authority, this control plane's `server.yaml` and secrets,
and lets it into the store; it starts there as a standby. Every node is told
of it within half a minute and fails over to it without being touched.
`--vip` keeps a floating address on whichever control plane leads, with
keepalived on both: that is the address to give clients, or a proxy in front.
Without it, put a proxy in front that sends to whichever answers `/leader`
with 200 — not `/health`, which is 200 on a standby too, and not `/ready`,
which is 503 on a leader with no model yet and would take the portal away
with it. The end of [deploy/proxy/Caddyfile](../deploy/proxy/Caddyfile) has
Caddy doing that for three control planes, and
[deploy/ha/](../deploy/ha/README.md#check-leader-not-health) why.

- **What a failover costs.** Measured on the reference cluster, stopping the
  leader: the standby led within five seconds, every node had moved to it
  and every replica was adopted — none restarted, the engines never stopped
  serving — and the floating address answered again with it.
- **Who leads.** `llmgw ctl status` and the console's Overview have a
  `CONTROL` line: which leads, which stand by, one that does not answer, and
  whether the floating address is on the leader. A standby with nothing to
  take over from is the thing to see before it is needed.
- **Either machine, the leader's cluster.** A standby passes its client port
  — the portal, its admin pages, the API — to the leader's, keeping only
  `/health`, `/ready` and `/leader` to describe itself, so a proxy pointed at
  the wrong machine still shows the cluster; the leader trusts a control
  plane's `X-Forwarded-For`, so clients keep their own address. Its admin
  API passes everything but `/debug/` and `/metrics` to the leader, over the port control planes already reach each
  other on and with the cluster's certificates both ways, so `sudo llmgw` and
  `llmgw ctl` show the same cluster on both — the leader's — and a change made
  on the standby is made by the leader. A node's certificate is refused
  there. Control plane certificates from before v0.11.3 cannot present
  themselves as clients: `server add` reissues them, and the running server
  picks the new one up when restarted.
- **The mirror** answers only the addresses it was given with `--api-allow`.
  `server add` checks, and says what to add to the mirror's unit if it
  refuses the new control plane — which would otherwise find out the first
  time it leads and fetches a model.
- **Limits are per process.** Rate limits are counted in memory, and clients
  reach the leader through the floating address, so a failover starts each
  key's minute afresh.

[deploy/ha/](../deploy/ha/) describes what each half does, for anyone putting
their own load balancer in front.

## The portal's language

The web portal — the landing page, sign-up and sign-in, the account and chat
pages, and the administrators' pages — speaks English and Turkish. Each person
gets, in this order:

1. the language they chose with the switcher at the foot of every page, which
   is remembered in a cookie (`llmgw_lang`) for a year;
2. otherwise the first language their browser asks for that the portal speaks;
3. otherwise the server's own default:

```yaml
# /etc/llmgw/server.yaml
landing:
  language: tr   # en or tr; empty is English
```

A code the portal does not speak stops the server starting, rather than
quietly serving English. Figures and dates follow the page's language: a
Turkish page writes 1.234,5 and 04.10.2026. Messages that come from the
control plane itself — a node's warnings, an engine's errors — are shown as
they were written, in English. The API, the CLI and the console are English.

Adding a language is adding a directory: `internal/web/locales/<code>/` with
the same files and keys as `locales/en/`, and its `_meta.*` entries saying how
it writes numbers and dates. The tests refuse a catalogue that lacks a key a
page uses.

## Single sign-on

People can sign in through your identity provider — Keycloak, Entra ID,
Google, Okta, Authentik, anything that speaks OpenID Connect.

1. Register a client there with the redirect address
   `https://<gateway>/login/oidc/callback`.
2. Put its secret in a file the llmgw user reads, on every control plane.
3. Add this to the cluster's settings with `llmgw ctl config edit`. On a
   cluster with a store, `auth` is the cluster's: the same block in
   `server.yaml` is not read, and the log says so.

```yaml
auth:
  oidc:
    issuer: https://login.example.com/realms/acme
    client_id: llmgw
    client_secret_file: /etc/llmgw/secrets/oidc-client-secret
    label: Acme SSO                 # the button says "Sign in with Acme SSO"
    create_accounts: true           # make an active account for someone new
    roles:                          # staff roles from groups, set at each sign-in
      admin: [llmgw-admins]
      operator: [llmgw-ops]
    memberships:                    # groups to places in the tree, joined at sign-in
      ml-engineers: "team:Acme/ml"
    no_passwords: true              # the provider only; administrators keep a password way in
```

How the sign-in works:

- **Request:** it uses PKCE, a state and a nonce.
- **Identity:** the ID token is verified against the provider's keys, and
  the account is found by the verified email address.
- **Unknown addresses:** without `create_accounts`, only people who already
  have an account can sign in.
- **Roles:** a person taken out of a group loses the role the next time
  they sign in.
- **Memberships:** they are added, never taken away.
- **Audit:** each sign-in is recorded in the audit log.

### Provisioning (SCIM)

The provider can also keep accounts in step: made when someone is assigned
the application, suspended when they are removed or leave. Give it a token:

```yaml
auth:
  scim:
    token_file: /etc/llmgw/secrets/scim-token     # any long random string, root-only
```

and point it at `https://<gateway>/scim/v2` with that token as the bearer.

- **Users:** the `userName` is the email address. Deleting a user
  suspends the account, and its keys and history stay.
- **Groups:** `auth.oidc.roles` and `memberships` are applied as people
  join and leave them. Membership in a group is removed when someone leaves
  it.

### Entra ID

Entra ID is an OpenID Connect provider like the others, with two things to
set in the app registration:

1. **Redirect URI:** under Authentication, add a Web platform with
   `https://<gateway>/login/oidc/callback`. The issuer is
   `https://login.microsoftonline.com/<tenant-id>/v2.0`.
2. **Groups in the token:** under Token configuration, add a groups claim.
   Choose **Groups assigned to the application**, and assign the groups you
   map in `roles` and `memberships` to the enterprise application. Entra
   sends groups by object id unless you choose to emit names
   (`sAMAccountName` for groups synced from AD), so write them the same way
   in the mapping.

Entra leaves the groups out of the token for someone in more than 200 of
them and sends a pointer to Graph instead. llmgw then keeps that person's
role and memberships as they were, rather than reading it as no groups, and
logs an error saying so. "Groups assigned to the application" avoids this.
So does SCIM below, which Entra also speaks: in the enterprise application
under Provisioning, set the tenant URL to `https://<gateway>/scim/v2` and
the secret token to the contents of `auth.scim.token_file`.

## Active Directory

With an on-premises Active Directory, people sign in on the gateway's own
form with their AD username (`ada`, `CORP\ada`, `ada@corp.example`) and
password. The gateway checks them with the domain controllers over LDAPS.
Put this in the cluster's settings with `llmgw ctl config edit` (or `llmgw
ctl config set auth.ldap '{…}'`); the files it names must be on every control
plane:

```yaml
auth:
  enabled: true
  ldap:
    urls: [ldaps://dc1.corp.example, ldaps://dc2.corp.example]   # tried in order
    ca_file: /etc/llmgw/corp-ca.pem         # the domain's CA, when not in the system's
    bind_dn: CN=svc-llmgw,OU=Service,DC=corp,DC=example
    bind_password_file: /etc/llmgw/secrets/ldap-bind   # root-only
    base_dn: OU=Staff,DC=corp,DC=example    # where people are
    user_filter: (memberOf:1.2.840.113556.1.4.1941:=CN=llmgw-users,OU=Groups,DC=corp,DC=example)
    label: Acme                             # "Sign in with your Acme account"
    create_accounts: true
    roles:                                  # by group CN or full DN; nested groups count
      admin: [llmgw-admins]
      operator: [llmgw-ops]
    memberships:
      ml-engineers: "team:Acme/ml"
    sync_interval: 15m                      # negative turns the sync off
```

- **Service account:** an ordinary domain user is enough: it only reads.
  It finds the person and reads their groups. The person's password is
  checked by binding as them, and is never stored.
- **Encryption:** `ldaps://`, or `ldap://` with `start_tls: true`.
  Plaintext LDAP is refused, since passwords would cross the network.
- **Certificate:** the domain controller's certificate must name it in
  its subject alternative names: a name in the CN only is refused, as are
  certificates with a negative serial number. Samba's own autogenerated
  certificate is both, so a Samba DC needs one from a CA (`tls certfile`
  in smb.conf). A CA file a control plane cannot read turns directory
  sign-in off on that control plane, said in its log. Local accounts still
  sign in.
- **Who may sign in:** enabled user objects under `base_dn` that match
  `user_filter`. The example admits members of `llmgw-users`, nested groups
  included.
- **Accounts:** an account is tied to the person's `objectGUID`, so a changed
  mail or name is still the same account. An account that already exists
  with the person's mail is taken over. Without `create_accounts`, only
  people who already have an account can sign in. Local accounts, such as
  the administrator `server init` made, keep signing in with their own
  password.
- **Sync:** every `sync_interval` the leader reads the directory.
  - Someone disabled or deleted in AD is suspended and their sessions
    end.
  - When they are enabled again they are reinstated. An account an
    administrator suspended by hand stays suspended.
  - Roles follow groups, and memberships the directory gave are taken away
    when the person leaves the group.
  - A sync that would suspend more than half of the directory's accounts is
    refused and logged, rather than locking everyone out over a wrong
    `base_dn` or filter.
- **Passwords:** a person's password is changed in AD. The gateway sends no
  reset link for these accounts.

## Secrets in Vault

Every setting that names a file holding a secret — the store's `@address`,
`api_key_file`, `auth.oidc.client_secret_file`, `auth.scim.token_file`, a
webhook's `secret_file` — also takes `env:VARIABLE` or a field of a secret in
HashiCorp Vault or OpenBao:

```yaml
auth:
  store: "@vault:secret/llmgw/store#dsn"
secrets:
  vault:
    addr: https://vault.internal:8200
    ca_file: /etc/llmgw/vault-ca.pem
    role_id_file: /etc/llmgw/secrets/vault-role-id      # AppRole; or token_file,
    secret_id_file: /etc/llmgw/secrets/vault-secret-id  # or VAULT_TOKEN
  master_key: vault-transit:llmgw
```

These are read when the gateway starts (an external model's key, each time
the catalog changes): a secret rotated in Vault is taken up at the next
restart.

**The store's own secrets** — what every engine's key is derived from — are
kept in the store, and so in its backups. `secrets.master_key` seals them:

- `vault-transit:KEY` seals with Vault's transit engine. The key never leaves
  Vault, so neither the store nor any backup of it opens without Vault.
- Anything else is a reference to 32 random bytes (`openssl rand -hex 32`),
  used here: better than nothing, but a key in `/etc/llmgw` is in every backup
  beside what it seals. Keep it in Vault or the environment.

Values kept plain from before are sealed the first time they are read after
the key is set. The key cannot be changed once set without losing them: a new
one cannot open what the old one sealed (the engines are then restarted with
new keys: `llmgw ctl restart --all`).

## The firewall

`server init`, `node add`, `server add` and `mirror init` each keep that
machine's **internal** llmgw ports to the cluster's own networks, with an
nftables table of llmgw's own (`inet llmgw`, in `/etc/llmgw/firewall.nft`,
loaded at boot by `llmgw-firewall.service`):

| Machine | Ports, for the cluster only |
| --- | --- |
| control plane | 7443 (agents), metrics when not on loopback, 8081 (its mirror), engines and 9100 when it is a node, 2379/2380 when it holds the store's third etcd vote |
| node | engine ports (`control.port_range`, 20000–20999), 9100 |
| store member | 5432, 8008, 2379, 2380, 9100 |
| mirror | 8081, 8082, 9100 |

"The cluster's networks" are, on each machine, the network of the interface
it reaches the control planes (or the other store members) on — or just
their address, when the route goes through a gateway. `node add` opens
the control plane's to each node it adds when the node is on a network not
already let in: its subnet when the control plane is on it, its address alone
when a router is between them. Anything else that must reach those ports, a
Prometheus on another subnet, a second control plane's or a separate mirror's
for nodes behind a router, goes in `server.yaml`:

```yaml
firewall_networks: [10.0.5.0/24, 192.168.1.20]
```

**Turning it off and on.** Whether llmgw keeps this firewall is one of the
cluster's settings:

```
llmgw ctl config set firewall.enabled false   # every machine's table taken out
llmgw ctl config set firewall.enabled true    # and put back as it was
```

It is in force on every machine within a minute, each in its own way:

- **nodes**: the agent takes its table out or puts it back;
- **control planes**: `llmgw-firewall-sync.path`, installed by `server
  init` and `server add` (`sudo llmgw firewall sync --install` on one set up
  before);
- **a separate mirror machine**: the leading control plane tells the mirror,
  which writes it to `.firewall.json` in its directory, and the same path
  unit applies it. `mirror init` installs it; on a mirror set up before, run
  `sudo llmgw mirror init` again with the same flags;
- **store members that are nothing else**: they run no llmgw, so
  `llmgw-firewall-follow.timer` reads what the leading control plane keeps in
  the store, from the member's own copy, once a minute. `server init
  --patroni` installs it; run it again on a cluster set up before.

The rules files stay, so turning it back on restores them. Nodes added while
it is off get none. Networks added to `firewall_networks` reach every machine
the same way. One taken out of it stays let in until it is taken out of
`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.

The leading control plane also tells the mirror machine where every node and
control plane is, so a node on a network the mirror had not let in can fetch
its weights without a `firewall_networks` entry.

Only those ports are touched. The table's policy is to accept, so whatever
else the machine serves — and whatever firewall it already has — is left as
it was; the client API (`:8080`) is not closed, being meant to be reached.
Running a command again adds to what is there rather than replacing it, so a
machine that is both a store member and a node keeps both sets. `llmgw
doctor` on a node says when the table is missing. `--no-firewall` on any of
the four commands leaves the firewall alone; `sudo systemctl disable --now
llmgw-firewall` removes it.

## Offline sites

Everything above downloads: the nodes fetch llama.cpp from GitHub and vLLM from
PyPI, `node add` and upgrades fetch llmgw's releases, and the mirror fetches
models from Hugging Face. A site with no route to the internet gets all of that
in a **bundle**, made somewhere that has one and carried in.

### Making the bundle

On any machine with internet and llmgw — a laptop will do:

```
llmgw bundle create --out llmgw-bundle --arch amd64 --tar
```

That carries:

- **llmgw's release** for each `--arch` (default: this llmgw's own version;
  `--version` for another), with `checksums.txt` and its signature, checked
  against the project's signing key as `ctl upgrade` checks it.
- **llama.cpp** at the version this llmgw installs, every build for each
  architecture — CPU, CUDA 13.3, CUDA 12.8 (amd64), Vulkan — each checked
  against the checksum GitHub publishes, the same check a node does online.
  `--variant cpu --variant cuda-13.3` carries only those, for a site whose
  hardware you know; `--engine llamacpp=b10991` names the version.
- **The starter model and the guardrails' judge** that `server init` serves
  (about 1.3 GB; `--no-starter-models` leaves them out), and any
  `--model name/version=hf:owner/repo/file` you add, pinned to the commit the
  registry had and checked against the checksum it publishes.

It ends by printing the manifest's **sha256**. Carry that apart from the bundle
— in the ticket, read out over the phone — because it is what says the bundle
that arrived is the one that was made. `llmgw bundle show llmgw-bundle.tar
--verify` reads every file back, worth doing once it is on the disk that
carries it. Run `create` again with the same `--out` (without `--tar`) to add
to a bundle; what is there already is not downloaded again.

What a bundle does **not** carry: vLLM and SGLang, which are Python
environments of several gigabytes from PyPI. Offline nodes serve with llama.cpp;
on a GPU node that needs vLLM, install it by hand and name it with `node add
--vllm-bin` (the node reports vLLM as unsupported until then). Nor the
operating system's packages — the OpenMP runtime llama.cpp needs, PostgreSQL
with `--postgres` — which come from each machine's own package sources: at an
offline site, its distribution mirror.

### Installing from it

On the control plane, with the bundle on a local disk:

```
sudo llmgw server init --bundle /media/usb/llmgw-bundle.tar --bundle-sha256 3f2a…
```

(The console's setup form has the same two fields.) Init then does what it
always does, with these differences:

1. Before the control plane starts, the bundle goes into the mirror beside it:
   every file checked against the manifest, the release against the signing
   key, and only then put in place. llmgw's releases and the engines go under
   `/srv/models/_llmgw/`, the models where the mirror keeps models.
2. The engines' checksums are written to `/etc/llmgw/offline.sha256`. That
   file, on the control plane, is what each node's engine files are held to.
3. `server.yaml` gets

   ```yaml
   offline:
     mirror: http://10.0.0.5:8081    # the mirror's file port, as nodes reach it
   ```

   and from then on the cluster asks nothing of the internet: nodes install
   llama.cpp from the mirror, `releases` defaults to the mirror's copy (so
   `node add` and `ctl upgrade` fetch from it), and the registry is not asked
   about newer weights.
4. The starter model and the judge are served from the mirror when the bundle
   carries them, and skipped — not waited for — when it does not. Without the
   judge, the guard policy is written without one.

With the mirror on another machine (`--mirror`), import the bundle there first
and then run init with `--bundle` as above: the control plane records only the
checksums.

```
# on the mirror machine
sudo llmgw bundle import /media/usb/llmgw-bundle.tar --sha256 3f2a…
```

Nodes join as usual with `sudo llmgw node add`: it fetches the node's binary
from the mirror and checks its signature, and the agent installs llama.cpp
from the mirror. Nothing on the node needs the internet.

### How it stays checked

Online, a node trusts GitHub, over TLS, for the checksum of the llama.cpp it
installs, and the project's signing key for llmgw. Offline it is the same
strength by a different route: the checksum GitHub published is recorded when
the bundle is made, the bundle is held to its sha256 when it is imported, and
the control plane keeps the checksums and tells each node, over its mutually
authenticated connection, what every file it fetches must be. The mirror serves
plain HTTP and is never asked for a checksum: a file changed on it, or on the
way from it, is refused by the node. Releases carry the project's signature,
which holds whatever server they come from — so plain HTTP from the mirror is
allowed for releases only because this build trusts a signing key.

### Later

A new llmgw release or llama.cpp version is another bundle. Make it the same
way — `--version v0.12.0`, `--engine llamacpp=b11200` — and on the control plane
(and the mirror, if separate):

```
sudo llmgw bundle import new-bundle.tar --sha256 …
```

The control plane sees the new checksums without a restart. Then set
`engines.llamacpp` in `server.yaml` to the new version and restart
`llmgw-server`, or run `sudo llmgw ctl upgrade v0.12.0` for the release. A
mirror on its own machine, which is not an agent, is upgraded from its own
copy: `sudo llmgw upgrade v0.12.0 --service llmgw-mirror --releases
http://127.0.0.1:8081/_llmgw/releases`. A control plane whose bundle lacks the
llama.cpp version it asks for says so in its log at start, and the node says
which `bundle create` would carry it. More models: `--model` in a bundle, or
copy the files into the mirror by hand — it is a directory.

A cluster set up online becomes offline the same way: `bundle import` on the
mirror and control plane, then the `offline:` block in `server.yaml` and a
restart.

## Stopping and starting the system

From a control plane:

```
llmgw status                     # does it serve; its control planes, nodes, models, store
llmgw stop --note "maintenance"  # every model taken down, every request answered 503
llmgw start                      # every model back as its catalog entry asks
sudo llmgw restart               # every agent and control plane started again, engines serving throughout
```

`llmgw stop` frees every GPU and answers requests for a model with 503
(`system_stopped`, with Retry-After) until `llmgw start`. The control planes,
agents and mirror keep running, so the console, the admin pages and `llmgw
start` keep working, and no model's definition changes. It is the cluster's
setting `serving.stopped`, so a restart does not undo it, and it is in the
audit with its note. `--wait` waits until every replica is down, or every
model has its replicas again.

`llmgw restart` restarts every node's agent, then the other control planes
over ssh (`--ssh-user`, the one sudo was run by), then this one — the leader
last, so it hands over to a standby already restarted. Engines keep serving
and are taken over again; nothing is placed anew.

One machine's own services are `llmgw service`:

```
sudo llmgw service status
sudo llmgw service stop          # agent, control plane, mirror — whichever this machine has
sudo llmgw service start
sudo llmgw service restart agent # or name one: server, agent, mirror
```

Stopping a node's agent leaves its engines serving — the agent takes them
over again when it starts — unless `llmgw service stop --engines` stops them
too, and the control plane then places their replicas elsewhere.

## Removing llmgw from a machine

```
sudo llmgw wipe              # services, configuration, certificates, state, users
sudo llmgw wipe --models     # and model weights, the mirror's files, engines
sudo llmgw wipe --store      # and the store's database here (Patroni member, etcd, or local PostgreSQL)
sudo llmgw wipe --all        # all of that, backups and the binary
```

It lists everything it will remove and asks for the machine's name (`--yes`
does not ask), then stops llmgw's services and the engines, takes the
firewall table out, and deletes `/etc/llmgw` — the authority and secrets
with it — `/var/lib/llmgw`, the agent's state, the sudo grant, the
authority from the system's trust, keepalived's configuration when llmgw
wrote it, and the `llmgw` and `llmgw-engine` users. Packages it installed
(PostgreSQL, Patroni, etcd, nftables, the node exporter) stay. Run it again
to finish what failed. A node wiped is then forgotten on the control plane
with `llmgw node remove <name>`. The console has it too: **Do → Wipe this
machine**.

This is how a machine with an older install is set up again from nothing:
wipe it, then install as above. Wiping a control plane without `--store`
leaves the store with the cluster's accounts, keys and settings in it, and a
new control plane against that store takes them up.

## Kubernetes

The control plane runs in Kubernetes from the Helm chart in
`deploy/helm/llmgw`; GPU nodes are best kept outside it, as ordinary hosts
joined with `node add`. Why, and how, is in [kubernetes.md](kubernetes.md).

## Upgrading

On a control plane:

```
sudo --preserve-env=SSH_AUTH_SOCK llmgw ctl upgrade v0.10.4
```

With two control planes it upgrades the standby first and the leader last,
whichever this machine is — the other over ssh as the user who ran sudo
(`--ssh-user`), with `llmgw upgrade` there, and this machine —
fetching the release for its platform, checking it against the release's
checksums, keeping the old binary as `/usr/local/bin/llmgw.bak-pre-<version>`,
restarting `llmgw-server` and waiting for every node to reconnect. The
leader's restart hands over once, to a standby already on the release.

Then the nodes, in waves: one first, then two, four, and so on up to
`--parallel` (20) at once. A node's agent is told the version over the stream
it already holds, fetches and checks the release itself, swaps it in and
restarts; no ssh into the nodes is needed. A fleet of hundreds takes minutes,
not hours. Release candidates (`v0.10.4-rc.2`) work the same way.

A node counts as done when it is back on the new version, ready, with as many
ready replicas as before. A wave not back within `--timeout` (three minutes)
stops the upgrade there — the first wave is one node, so a bad release reaches
one node rather than all of them; `llmgw ctl logs <node>` says why, and the node
keeps its previous binary beside the new one. Nodes already on the version are
skipped, so running the command again resumes where it stopped. `--nodes a,b`
limits it to some nodes, `--skip-control-plane` leaves the control planes alone,
and `--other-control-planes=false` this machine's partner.

**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its
next connection. The control plane holds no replica state across its own
restart either — it rebuilds the fleet from what the agents report.

**A mixed fleet is fine while you roll.** The agent protocol only ever gains
fields, and both sides treat an absent one as unknown rather than as a default.
Agents older than v0.10.4 cannot upgrade themselves: move those by hand once,
installing the release binary and restarting `llmgw-agent`, and every upgrade
after that is the command above.

**The mirror is upgraded on its own host**, since it is not an agent and its
service cannot replace its own binary:

```
sudo llmgw upgrade v0.10.5 --service llmgw-mirror
```

The same checks — the release's checksums, the old binary kept beside the new —
then a restart of the unit, waiting for it to stay up. A mirror older than
v0.10.5 does not have the command: install the release binary by hand once.

Then confirm what you have, rather than assuming:

```
llmgw ctl nodes       # every agent on the new version, LAST SEEN in seconds
llmgw ctl replicas    # same replicas, adopted rather than replaced
```

A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
`llmgw ctl restart <replica>` replaces it with one on the new profile, starting
the new one before stopping the old. Where the old one is all that stands in
the new one's way — a single GPU — it is stopped first, at once, and the model
is unserved while the new one loads; where the new profile fits nowhere even
then, the restart is refused and the old one keeps serving, which `ctl
replicas` shows as `restart refused:` with the reason. `--in-place` asks for
stop-then-start whatever room there is.

### Signed releases

Every install above checks a release against its published checksums, which
proves the download is what the server holds — not that the server holds what
was built. Signing closes that: CI signs the checksums, and a binary that
trusts a key installs only releases signed with it. Off until a key exists;
turning it on, once:

```
llmgw release keygen
```

1. Put the **private** key in the repository's Actions secret
   `RELEASE_SIGNING_KEY`. From the next tag, releases carry
   `checksums.txt.sig`.
2. Put the **public** key in `internal/release/trusted.pub` and release. Every
   binary built from then on carries it and refuses releases that are not
   signed with it — `ctl upgrade`, agents upgrading themselves, and
   `llmgw upgrade` alike.

In that order: a build that trusts a key refuses unsigned releases, so the
secret has to be signing before the key is trusted. The first trusting release
is itself installed by a binary that trusts nothing; every one after is checked
by the one before. `LLMGW_RELEASE_KEY` overrides the built-in key, for rotating
it. The onboarding script fetches with `curl` and checks checksums only; a node
is checked from its first upgrade on.

## Backing up

`llmgw server backup` writes one archive holding everything a control plane
cannot be rebuilt without:

- **the store** — users, API keys, the tenancy tree, limits and budgets, usage
  history, models added at runtime, measured capacity, and the secret every
  engine key is derived from (`pg_dump` in its custom format for PostgreSQL, a
  consistent `VACUUM INTO` snapshot for SQLite; both are safe against a running
  gateway);
- **`/etc/llmgw`** — the configuration, the store's address, and the
  certificate authority's key (`pki/`). Nothing in llmgw can reconstruct that
  key; losing it means reissuing every node's identity.

Archives go to `/var/backups/llmgw-server`, readable by root only because of the CA
key, and the newest 14 are kept. Run it nightly with the timer:

```
sudo install -m 0644 deploy/systemd/llmgw-backup.service deploy/systemd/llmgw-backup.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now llmgw-backup.timer
sudo systemctl start llmgw-backup.service     # one now, to see it work
ls -l /var/backups/llmgw-server
```

The control plane publishes when the newest archive was written as
`llmgw_backup_last_success_timestamp_seconds`, and **BackupTooOld** fires when
it is more than 26 hours old — including when there has never been one.

**Encrypt them.** Name who may open them in `/etc/llmgw/server.yaml`, and every
archive is encrypted to those keys with [age](https://age-encryption.org) and
named `.tar.gz.age`:

```yaml
backup_recipients:
  - age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p   # age-keygen's public key
  - ssh-ed25519 AAAAC3Nza... ops@example                              # or an SSH public key
```

Keep the matching private keys **off** this machine — an archive whose key
sits beside it is not protected by it. Encrypted, a copy is safe wherever it
lands.

**Copy the archives off the machine.** They sit on the disk they back up and do
not survive losing it. Anything that copies files will do — rsync from the
mirror, a NAS mount, an offsite bucket — as long as it keeps the archives
readable by root only.

**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued.

### Copying archives off the machine

```yaml
backup_recipients: [age1…]          # required: an archive leaves only encrypted
backup_copy_to:
  - /mnt/backups/llmgw              # a mounted share or another disk
  - ssh://backup@10.0.0.20/srv/llmgw-backups
backup_copy_key: /etc/llmgw/secrets/backup-ssh-key
```

Each archive is copied there as soon as it is written, and the same number is
kept there as here (`--keep`, 14). An ssh host's key is learnt the first time
and checked every time after, in `/etc/llmgw/backup_known_hosts`. A copy that
fails makes the backup fail, so the timer's unit shows it.

### Restoring

On a machine with llmgw installed — the same version as the archive's, or
newer — and, for PostgreSQL, the database and role the store's address names:

```
sudo llmgw server restore /var/backups/llmgw-server/llmgw-backup-<host>-<time>.tar.gz --dry-run
sudo llmgw server restore llmgw-backup-<host>-<time>.tar.gz.age --identity key.txt
```

It checks before it touches anything — that the archive is whole and opens,
that this llmgw is not older than the one that wrote it, where the store goes
and that `pg_restore` is there for a PostgreSQL dump — and `--dry-run` stops
there. Then it stops `llmgw-server`, moves `/etc/llmgw` aside to
`/etc/llmgw.before-restore-<time>`, puts the archive's in its place with each
file's mode and owner, restores the store (a SQLite file replaced, the old one
kept beside it; PostgreSQL by `pg_restore --clean` in one transaction), starts
the server and waits for it to answer. `--store` restores the store somewhere
else than the archive's `server.yaml` says.

Running engines keep serving throughout: agents reconnect to the restored
control plane and their replicas are adopted, and because the engine-key secret
came back with the store, the router can still reach them.

### A disaster recovery drill

Worth doing once before it is needed, on a spare machine, and again after a
major upgrade:

1. Install the same llmgw version on the spare machine (step 1).
2. Copy the newest archive there from wherever `backup_copy_to` put it, and
   the private key that opens it.
3. `sudo llmgw server restore <archive> --identity <key> --dry-run`, and read
   what it says. For PostgreSQL, create the database and role first, or point
   `--store` at a scratch database.
4. Without `--dry-run`. Then `llmgw ctl status`, `llmgw keys list --all`, and
   a request with a key that existed before.
5. Note how long it took: that is the number to plan an outage around.

The drill machine now holds the cluster's CA key and secrets: take it down
afterwards, or keep it as the standby it has become.

**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. A pinned version may no longer be upstream to download: that is
exactly what pinning it protected you from, and a copy of the directory is what
makes the protection outlive the disk.

## Reclaiming disk

`llmgw ctl nodes` has a DISK column: free space against the size it is free
out of, for the filesystems holding model weights. A dash means the node was
not asked — an agent with no `model_dirs`, or one older than this — and not
that it has no space. Shared filesystems are excluded from a node's figure,
because space on a filer belongs to everyone mounting it.

Weights llmgw downloaded for models that are no longer in the catalog can be
listed and removed:

```
llmgw ctl weights             # what could be freed, per node
llmgw ctl weights --collect   # delete it
```

**Only files llmgw fetched itself are ever candidates.** They are recognisable
because llmgw names them after a hash of their source, so the set still wanted
is computed from the catalog exactly rather than guessed. Anything you put on
a node — by scp, by a shared mount, by hand — is left alone however
unreferenced it looks, and no llmgw command will remove it. The nodes check
every path again before deleting it, so a control plane that is wrong cannot
talk one into losing a model that took a week to copy.

Which also means llmgw will not reclaim the storage most likely to be filling
your disk. Weights you placed yourself are yours to remove.

A profile may declare `min_disk_gb`, and a node without that much free is then
not a candidate for it — checked only on nodes that do not already hold the
weights, since those need no room for them. Without the field, disk is not
considered, which is how every profile written before it keeps behaving.

## When something does not work

**The node is `ready` but every replica fails.** Almost always a missing engine
or weights. `llmgw ctl replicas` shows the shard's error and the engine's last
output, which is the actual diagnosis.

**A model will not place.** `llmgw ctl explain-placement <model>` names each
node and the reason: wrong GPU class, not enough free cards, too little memory,
no fabric, a label it does not carry.

**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
two. `llmgw ctl status` shows how full the cluster is.

**Tool calls come back as text.** The model made the call and the engine did not
parse it. `llmgw ctl tool-check --endpoint http://<node>:<port>` distinguishes
that from a model ignoring its tools, which has a different fix.

**Requests fail at exactly sixty seconds.** `timeouts.response_header` is a
liveness check for streaming and a total generation budget for non-streaming
requests. On a slow model, set `timeouts.generation` higher.

## What to read next

- `docs/choosing-models.md` — which models to run and how many, if the answer
  to "we have GPUs, now what" is not already obvious
- `docs/data.md` — what is stored and what leaves the machine, for the review
  that happens before this is allowed near anyone's documents
- `docs/reference.md` — every configuration key, generated from the source, so
  it describes the version you are running rather than the one somebody last
  wrote about
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
  and what belongs on which port
- `docs/kubernetes.md` — the control plane in Kubernetes from the Helm chart,
  and why GPU nodes stay outside it
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
  each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
  checking a node, and what to look at when a replica spanning nodes will not run
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
  deliberately does not do
- `docs/onboarding.md` — giving people accounts and keys
- `llmgw-plan.md` — why the system is shaped the way it is

## Known gaps, so you find them here rather than at 2am

- **Every node fetches its own copy.** There is no seeding between nodes, so
  bringing up eight nodes on a new model is eight downloads. The mirror makes
  those eight local, which is most of the problem; on a fabric where the nodes
  can see each other, seeding between them is the obvious next thing to build.
- **A budget without a price never binds.** Set `default_price` or `pricing`
  (step 2) or every request costs zero, and a budget is something the dashboard
  displays rather than something that stops anything. The Limits tab says so
  where it can, which is not the same as being told before you rely on it.
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
  no build from 0.20 on has Volta kernels, and the node picks the newest that
  has. llmgw's flags are checked against the current version, so run `llmgw
  ctl engine-check vllm` on a V100 node once.
- **Two control planes have not been run for long.** The leader election and
  agent failover work and there is a pattern in `deploy/ha/`, but every cluster
  this has been run on in anger has had one. A single control plane is the
  well-trodden path.
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
  scheduling, the fabric identity, the maps and the launch are written against
  the real tools' formats and tested against them, not yet run on hardware that
  has them. Single-node placement on real GPUs is the well-trodden path; if you
  are the first to run a TP group across real cards, run `llmgw doctor --json`
  on each node first and keep the reports.
- **One engine setup is weaker than the rest.** vLLM's `launch_mode: ray`
  starts a Ray head without authentication, so anyone who can reach its
  port can run code on the node, and Ray's ports are not among those
  llmgw's firewall guards. The dashboard is off, but firewall the port to
  the other nodes in the replica, or use the default `mp` launch mode, which
  needs no Ray at all.