# Installing llmgw from nothing

Written for whoever is standing up the cluster: you have servers, you have root
on them, and nothing is installed yet. For only the commands, read
[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a
node in the fleet, and a model answering requests.

It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond
the package names. Everything here has been done on a real cluster except where
this page says otherwise, and where it says otherwise it says so plainly.

## What llmgw does not install

Read this first, because it is the difference between an afternoon and a week.

llmgw runs engines and routes requests. It does **not** install GPU drivers,
CUDA, or the inference engines themselves. A node missing any of them still
registers, still reports `ready`, and still fails every replica placed on it —
so the useful order is to make a node able to serve *before* onboarding it, not
after.

| You provide | llmgw provides |
| --- | --- |
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
| Model weights, from somewhere | Fetching them, placement, routing, accounting |

The reason is boring and unavoidable: drivers and engines are platform decisions
with kernel modules and version matrices attached.

Weights are the one item on that list llmgw does handle. It includes a mirror —
step 3 — which holds the files on your own network and hands them to the nodes,
so getting weights onto every machine is not a problem you solve before
starting. What llmgw will not do is choose the models for you.

## 0. Before you begin

Four machines is a reasonable first cluster: one control plane and three nodes.
One of each also works.

**The control plane** needs no GPU. It is a single process holding a SQLite
file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable
from every node on the agent port, and from your users on the client port.

**Nodes** need whatever the model needs, plus about 8 GB for the operating
system and the agent. They dial the control plane; the control plane never dials
them, so nodes need no inbound access except your own SSH.

**Ports**

| Port | Who connects | Expose to |
| --- | --- | --- |
| 8080 | clients, the web UI | your users, usually via a reverse proxy |
| 7443 | node agents | the node network only |
| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
| 8081 | node agents fetching weights | the node network only (the mirror machine) |
| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |

Decide now which address nodes will use to reach the control plane. It goes into
a certificate: the machine's own addresses are found for you, and any other name
— a DNS alias, a NAT or floating address — has to be given with `--host`.

## 1. The binary

One static binary, no runtime dependencies, the same on every machine.

```
VERSION=v0.8.0    # check the releases page for the current tag
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
curl -fsSL  https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```

`arm64` and `darwin` builds are published alongside. They are built with
`CGO_ENABLED=0` deliberately, so a binary built today runs on a node whose libc
is older than the builder's.

To build instead of download, you need Go 1.26 or later and nothing else:

```
git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw
CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw
```

## 2. The control plane

On the machine that will be the control plane, as root:

```
sudo llmgw server init --admin you@example.com
```

That is the whole step. In order, it:

1. creates the `llmgw` system user, `/etc/llmgw` and `/var/lib/llmgw`
2. creates the cluster's certificate authority in `/etc/llmgw/pki`
3. issues the control plane's certificate for this machine's hostname and
   every address it has, and copies `ca.pem` beside it
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
   and a price, and a control plane with an empty catalog
5. creates the store, the `default` project, and your account, already approved
   and promoted to administrator — the password is generated and **printed
   once**
6. installs the systemd unit, starts it, and waits until `/health` answers
7. runs a model mirror beside it (`/srv/models`, bytes on `:8081`, its
   management API on loopback) and points the control plane at it
8. makes this machine a node of its own control plane — on its GPUs when
   `nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
   engine
9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
   first thing anyone opens `/chat` to find is a model that answers

`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
card with no working driver is reported rather than silently ignored; install
the driver and run init again to serve on it.

Add `--postgres` to keep accounts and keys in PostgreSQL rather than SQLite:
init installs it and sets it up, and on a machine already running on SQLite it
moves everything across. See [PostgreSQL](#postgresql-when-sqlite-is-no-longer-the-right-shape).

Every step looks before it acts. Running it again finishes an install that
stopped halfway and changes nothing on one that did not; it never overwrites
`server.yaml`, the authority, or an existing account's password.

**The addresses in the certificate matter.** Agents check that the address
they dial is named in it, and the certificate names only what this machine
could see of itself. If nodes will reach it by another name — a DNS alias, a
NAT address, a floating IP — give it:

```
sudo llmgw server init --host llmgw-ctl.internal --host 203.0.113.10
```

Run that at any time: the certificate is reissued with the new names added to
the old ones, and the service restarted.

**`ca-key.pem` signs every node's identity.** `init` leaves it in
`/etc/llmgw/pki`, because onboarding a node needs it and the control plane is
where most people run onboarding from. Back it up; there is no recovering it.
If this machine is exposed, move the directory elsewhere once the nodes are in
and pass `--pki` to onboarding.

This covers the fleet-facing port only. The client-facing API is public and
belongs behind an ordinary publicly trusted certificate, terminated by a
reverse proxy in front of port 8080. [deploy/proxy/](../deploy/proxy/) has a
Caddyfile and an nginx configuration that get streaming, timeouts and the
forwarded headers right; both are attached to every release.

### Check it

```
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health   # 200
llmgw ctl status
```

`ctl status` will report no nodes and no models, which is correct — nothing has
been added yet. Sign in at `http://<control-plane>:8080/login` with the printed
password, change it, and issue yourself a key from the profile page.

### The configuration

`/etc/llmgw/server.yaml` is yours from here on. Three things in it are worth
knowing before you edit it:

- `admin_listen` is on loopback on purpose. It serves pprof, which can dump the
  process heap, and API keys sit in that heap in plaintext. To see the dashboard
  from elsewhere use `/admin` on the client port — do not move this listener.
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
  budgets mean anything. Without a price every request costs zero and a budget
  never binds; without the defaults, a project created without limits is
  metered by nothing at all.
- `control.enabled: true` is what makes this a control plane while its catalog
  is empty. Without it, a file with no models is refused as having nothing to
  serve.

Every key is in [reference.md](reference.md).

### Keeping the authority off the control plane

Create it on another machine, issue the control plane's certificate there, and
bring only the results:

```
# on the authority's machine
llmgw ctl cert init-ca --dir /etc/llmgw/pki
llmgw ctl cert issue-server llmgw-ctl --host 10.0.0.140 --host llmgw-ctl.internal \
  --dir /etc/llmgw/pki --out ./ctl

# copy ca.pem, ctl/server.pem and ctl/server-key.pem into /etc/llmgw on the
# control plane, then there:
sudo llmgw server init --admin you@example.com
```

`init` finds certificates in `/etc/llmgw` and no authority beside them, and
keeps them as they are. Adding an address later is then done where the
authority is; `init --host` says so rather than minting a second authority no
node trusts.

## 3. The mirror

Do this before the nodes, not after. Every node fetches its own copy of a
model's weights, and where they fetch from decides whether bringing up a cluster
takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
Face is a day's work, and on a good number of corporate networks it is not
possible at all — the nodes have no route out, which is usually the point.

The mirror is one machine holding the files on your own network. It is the same
binary again, and it is the highest-value piece of infrastructure around llmgw
for the least work.

### Running it

```
sudo install -d -o llmgw -g llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
```

In production, the unit:

```
sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
```

**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
what every agent talks to, and it is plain HTTP because a weights fetch is a
public file transfer that is checksummed anyway. `:8082` is the management API:
it can start downloads and delete models, and only the control plane needs it.
The split is what lets you lock the second one down without re-certifying every
agent that fetches from the first.

Give the management port the cluster's own CA and it takes client certificates:

```
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
  --tls-ca /etc/llmgw/ca.pem \
  --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
```

Without those three flags the management port is open to anyone who can reach
it, and the mirror says so in its own log at every start rather than letting you
find out later.

### The layout

```
/srv/models/
  qwen2.5-14b-instruct/
    2026-03/
      q4_k_m.gguf
```

Model, then version, then files. The version directory is the point of the
layout rather than decoration: a repository that changes upstream changes what
you are running, and the first sign is a prompt that worked last week answering
differently this week with nobody able to say why. Pin it here and that cannot
happen.

A version holding several files is a sharded model, and is served as a set: the
agent asks the directory what it consists of and fetches all of it into one
place, marking it complete only when the last file lands. A catalogue entry for
one names the directory, trailing slash included.

### Telling the control plane about it

```yaml
# /etc/llmgw/server.yaml
mirror:
  api: https://models.internal:8082
  tls:
    ca: /etc/llmgw/ca.pem
    cert: /etc/llmgw/server.pem
    key: /etc/llmgw/server-key.pem
```

With this set, `/admin/mirror` appears: what the mirror holds, what it is
downloading and how far along, a search of Hugging Face, and a form that puts a
mirrored model into service without anyone writing YAML. Without it that page
stays unmounted and the mirror is still perfectly usable — a model's `source` is
just a URL, and the URL can be yours.

If the control plane cannot reach the internet, `mirror.registry` points the
search at an internal registry mirror instead.

### Filling it

```
llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
  hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
```

or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.

## 4. Preparing a node

**Do this before onboarding.** A node missing its driver or engine registers
healthy and fails on first placement, and the reason is three logs away.

### The driver

```
sudo apt-get update
sudo apt-get install -y nvidia-driver-550   # or whatever your cards need
sudo reboot
nvidia-smi                                   # must print your GPUs
```

`nvidia-smi` must be on `PATH` for the user the agent runs as, which is root.
Containerised driver installs often put it elsewhere; if so, note the path and
set `discovery.nvidia_smi` in the agent configuration rather than moving it.

A node with no GPU at all is fine and is not an error. It reports zero cards and
can host CPU profiles, which is a good way to try llmgw before committing
hardware to it.

### NVLink, if your GPUs have it

The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
joins, each card's NUMA node, and which network adapter is nearest each. A
tensor-parallel group is then the set of cards with the most NVLink between
them — on a PCIe box where bridges pair specific cards, a group straddling two
bridges falls back to PCIe and runs several times slower, and nothing else
would report it.

On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
checks it:

```
sudo llmgw doctor
```

A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.

### InfiniBand, if you have it

```
sudo apt-get install -y rdma-core ibverbs-utils infiniband-diags
ibstat            # must list your HCAs with State: Active
ls /sys/class/infiniband
```

Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
node, and llmgw checks the name against what the adapters report: every port on
one fabric sees the same subnet managers, so two independent fabrics given one
name are placed as two, and the cluster page says so. An adapter that is not
there, or whose port goes down, takes that fabric off the node within seconds,
and gives it back when the port returns.

[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
fabric pages show, and what to look at when a replica spanning nodes will not
run. Run `llmgw doctor` on the node before it joins.

### Engines

Nothing to do: the agent installs them once the node joins, under
`/opt/llmgw/engines`, and checks each one runs before the node offers it.

| Engine | On | How |
|---|---|---|
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |

vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:

- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
  vLLM is built for CUDA 13. A V100 cannot run it;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.

A node with no GPU skips vLLM and says nothing about it. A node that has a GPU
but cannot run it — an old driver, a V100, a small disk — says why on the
cluster page. `llmgw doctor` checks the same before the node joins.

The versions are the control plane's, in `server.yaml`:

```yaml
engines:
  vllm: "0.30.0"      # the default; "" leaves vLLM to you
  sglang: "0.5.20"    # SGLang only when named
```

A changed version reaches every node when it next connects. To use an engine
you installed yourself instead, set its version to `""` and name its path in
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).

After changing a version, check it still takes the flags llmgw builds for it,
which drift between releases:

```
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```

### Weights

Nothing to do here if you built the mirror in step 3 — the agent fetches what a
model's `source` names, and that source is your mirror. This section is for the
arrangements where the files are already on the machine.

Put them where the node can read them. `scp`, a shared filesystem, or the
engine's own downloader are all valid, and which you want depends on how many
nodes you have.

```
sudo install -d -o root -g root /var/lib/llmgw-models
# then place your .gguf or model directory there
```

For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on
CPU, and answers fast enough to prove the whole path works.

### Letting llmgw fetch them

The simple arrangement, and the default: every node keeps its own copy. Point a
model's `source` at a file to download and the agent fetches it once, before the
engine is launched:

```yaml
# your mirror, which is what step 3 was for
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
# a whole version, for a model of several files — the trailing slash is the signal
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
# or straight from Hugging Face, where the nodes can reach it
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
```

A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
the dashboard both show how far along a node is. A replica pulling a hundred
gigabytes and one that is wedged do not look alike.

Eight shards of one replica starting together produce one download, not eight.
An interrupted fetch resumes rather than starting over, and a partial file is
never visible under the model's name — so a half-downloaded model cannot be
picked up by the next scan and handed to an engine.

Three kinds of source, and only one of them is fetched:

| `source` | What happens |
| --- | --- |
| `/var/lib/llmgw-models/x.gguf` | Checked, never copied. However it got there. |
| `hf:org/repo/file.gguf`, `https://…` | Fetched once into the cache, then reused |
| `Qwen/Qwen2.5-7B-Instruct` | Left alone — vLLM and SGLang download repositories themselves |

Which means the exotic arrangements need no configuration at all. Weights on a
Lustre mount, or an image exported over the fabric with SRP or NVMe-oF, are
simply *present*, and present means nothing is fetched.

### Telling llmgw where they are

Optional, and unset by default. Naming the directories in the agent's
`model_dirs` lets a node report which models it can see, and placement then
prefers a node that already holds one over a node that must first move tens of
gigabytes:

```yaml
model_dirs:
  - /var/lib/llmgw-models
  - /mnt/lustre/models
```

llmgw does not manage that storage and does not care how the files arrived. A
Lustre or NFS mount, an image exported over the fabric with SRP or NVMe-oF,
local NVMe, or an `scp` all look identical to it. A node with nothing configured
reports nothing and is treated exactly as every node was before this existed.

**It matters most when your fabrics are not connected to each other.** Weights
in `ib-a` are not weights in `ib-b`: the only path between them is the
management network, which is the slow one. `llmgw ctl status` shows what each
fabric holds, so a model seeded into one fabric does not quietly leave the rest
of the cluster unable to serve it.

How much disk a node needs follows from the models rather than a rule of thumb.
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
bf16 is 141 GB and the same model at q4 is 43 GB.

## 5. Adding the node

Two halves, because they need different access: building the machine needs the
hypervisor, and issuing its identity needs the certificate authority. Where one
person holds both — the common case — one command does it:

```
llmgw-add-node \
  --name llmgw-node01 --ip 10.0.0.141/24 \
  --server https://10.0.0.140:7443 \
  --cores 16 --memory 131072 --disk 200 \
  --fabric ib-a:mlx5_0,mlx5_1 \
  --llamacpp-bin /opt/llama.cpp/llama-server
```

**The VM name, the hostname, the node id and the certificate's common name are
all the same string.** They do not have to be, and every time they are not,
somebody spends an afternoon working out which of the four a message refers to.

For a machine that already exists — bare metal, or a VM you built yourself — skip
provisioning and issue the identity only:

```
llmgw-onboard-node \
  --node llmgw-node01 --addr 10.0.0.141 \
  --server https://10.0.0.140:7443 \
  --fabric ib-a:mlx5_0,mlx5_1 \
  --llamacpp-bin /opt/llama.cpp/llama-server
```

Onboarding needs SSH from the CA host to the node. It waits for SSH rather than
failing on it, and learns the host key of a machine it has not seen — a new
machine spends its first minute regenerating host keys and restarting sshd.

It also checks that a configured engine actually runs, and says so loudly when
it does not. That check exists because the failure is otherwise invisible.

Within a few seconds:

```
llmgw ctl nodes
```

```
NODE          STATE  GPUS      MEM   CLASSES   FABRICS  REPLICAS  LAST SEEN
llmgw-node01  ready  8/8 free  503G  h100-80g  ib-a     0         2s
```

If it does not appear, the control plane's log says why. A certificate whose
common name does not match the claimed node id is refused by design — that check
is what makes a node's identity something it proves rather than something it
picks.

Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
name; nodes with no InfiniBand take no `--fabric` at all and are each their own
island, which is fine for single-node models.

## 6. Setting the node's CPU model, if it is a VM

Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. Inference then falls back to scalar code and is not slightly
slower but unusable. Measured on one 8-core VM, same weights, same threads, the
only change being `--cpu host`:

| Qwen2.5-Coder | kvm64 | host |
| --- | ---: | ---: |
| 1.5B q4_K_M | 5.80 | **39.09** tok/s |
| 3B q4_K_M | 2.90 | **20.51** |
| 7B q4_K_M | 1.33 | **9.88** |

`llmgw-provision-node` sets `--cpu host` for you. A VM you built yourself
probably did not, and nothing about it looks broken — every number is simply
small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.

## 7. Adding a model

Write the definition. This is the same shape a model takes in `server.yaml`, so
an entry moves between the two by being copied:

```yaml
# qwen-coder-3b.yaml
name: qwen2.5-coder-3b-instruct
aliases: [coder]
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
routing_profile: chat
# What a client may use in one request. Worth stating: llama.cpp divides
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
# 32768 each, and nothing else will tell you that.
context_length: 32768
max_output: 4096
scaling: { min: 1, max: 1 }
profiles:
  - name: cpu
    engine: llamacpp
    weight: 1
    priority: 100
    startup_timeout: 5m
    min_memory_gb: 8
    args: ["--ctx-size", "65536"]
```

```
llmgw ctl model add -f qwen-coder-3b.yaml
llmgw ctl models
```

Adding a model checks that the source is actually there before accepting it, so
a typo or a mirror that is down is a sentence now rather than a replica failing
on a node in five minutes with an error naming the node. `--no-source-check`
skips it, which is right when the nodes reach the mirror by an address this
machine does not.

With a mirror configured you can skip the file entirely: `/admin/mirror` lists
what the mirror holds and offers a form that writes the same definition, with
the engine, the source and the memory figure already filled in from the file
itself.

```
MODEL                      SERVING  REPLICAS  STATUS   ALIASES
qwen2.5-coder-3b-instruct  yes      1/1       ready    coder
```

No restart, and it survives one: models added this way are stored, while
anything in `server.yaml` is declarative and returns on every start.

### Models that load when someone asks

A cluster with more models than GPUs can hold every model in its catalog and
only the wanted ones in its cards:

```yaml
scaling: { min: 0, max: 1, idle_timeout: 15m }
```

`min: 0` with a `max` above it means load on demand. The first request for the
model starts it and waits briefly; if it is not serving by then the client gets
a 503 with `Retry-After` and an honest "the model is loading", rather than being
told nothing serves it — a client told to retry succeeds where one told the
model does not exist gives up. Once nobody has asked for it in `idle_timeout`,
the replicas are stopped and the cards go back to the pool.

`min: 0, max: 0` is different and equally useful: configured, and deliberately
not running. A model with no `scaling` stanza at all gets that, because making
every unscaled model loadable on demand would let any client request start a
70B nobody intended to run.

Two things to weigh. A cold start is visible to whoever triggers it, so this
suits the long tail people are experimenting with rather than the two or three
models used daily — keep those at `min: 1`. And stopping a replica throws away
the KV cache prefix routing spent its effort filling, so an idle timeout wants
to be minutes rather than seconds.

When the cluster is full, a model somebody asks for will stop an idle one to
make room. Only idle replicas of other on-demand models: a model at `min: 1` is
a promise that it stays up, and breaking that to serve a different request would
make every guarantee in the catalog conditional on load. Only single-node
replicas, because evicting a multi-node one frees several machines and costs
tens of minutes to rebuild — a decision worth making deliberately. One at a
time. And never a replica that has just started, which is what stops two models
on a cluster with room for one from taking turns evicting each other.

For a GPU model, name the class and the width instead:

```yaml
profiles:
  - name: tp2
    engine: vllm
    gpu_class: h100-80g    # as `llmgw ctl nodes` reports it
    tp: 2
    priority: 100
```

If it does not place, ask:

```
llmgw ctl explain-placement qwen2.5-coder-3b-instruct
```

which names every candidate and why each was rejected, rather than leaving you
to infer it.

### How many replicas, and how much each one carries

Replica count is changed while the cluster runs, without editing the
definition:

```
llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
```

Models added at runtime only. One declared in `server.yaml` is read again on
every start, so a number set here would hold until the next restart and then
revert — change it in the file instead.

How much traffic each replica is given is the profile's `weight`, and guessing
it is how a cluster ends up with one saturated replica beside two idle ones.
Measure it instead:

```
llmgw ctl bench qwen2.5-coder-3b-instruct --save
```

Concurrency climbs until the latency target stops being met, and the last level
that held it is the capacity — peak throughput arrives long after latency has
become unusable, so throughput alone would overstate what the hardware serves.
It prints the `weight` to put in the profile, and `--save` records the figure
against the model so the dashboard can show what the cluster is measured to
carry rather than what somebody typed.

This drives real traffic and competes with real requests. Run it against a
cordoned replica, or in a window where that is acceptable.

An engine that publishes no metrics endpoint is measured from the outside only.
The run says so, and queue depth and KV occupancy are then not part of the
answer.

## 8. The first request

```
curl http://<control-plane>:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "coder", "messages": [{"role":"user","content":"hello"}]}'
```

Or open `/chat` in a browser and use it without a key at all — the page carries
your session, and the request still goes through admission, rate limits, budget
and accounting exactly as an API call does.

## Watching it

The control plane exports both halves on one endpoint: what the cluster is
serving — requests, latency, time to first token, tokens, admission, per
endpoint — and what it is made of, per node and per GPU, including memory,
temperature, ECC faults and utilisation.

**The dashboard is the answer for most clusters.** `/admin` shows capacity per
GPU class, free whole nodes per fabric, which replicas are holding hardware
nothing is asking for, who is using the cluster and what they are running, and
the same node and model tables `ctl` prints — with no monitoring stack to stand
up beside it. One binary, no dependencies, which is the point.

Prometheus is worth adding when you want history, alerts, or one view across
several clusters. The dashboard shows what is true now; it keeps nothing.

Metrics then get their own listener, because the admin one also carries pprof
and pprof can dump the process heap, where API keys sit in plaintext:

```yaml
admin_listen:   "127.0.0.1:9090"   # dashboard, management API, pprof — keep local
metrics_listen: "0.0.0.0:9091"     # metrics only, safe to reach
```

One scrape target for the whole fleet. The node and GPU figures come from the
control plane, which already holds them, so there is nothing to install on the
nodes and no Prometheus configuration per machine.

A starting configuration, with the alerts worth having and a provisioned
dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
to run it, which is not the control plane: monitoring that dies with the thing
it monitors tells you nothing at the moment you need it.

Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
rather than zero when the driver does not report it — a flat zero across a
fleet means the measurement is missing, not that the fleet is idle. And
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
which is what a binary built without the nvml tag reports.

## PostgreSQL, when SQLite is no longer the right shape

SQLite is the default and stays the right answer for most installations: one
file, no server, and a request path that barely touches it — usage is written
in batches, budgets are counted in memory, authentication is cached.

Two things change that. Conversations and agent notes read and write on every
turn, which is the first per-request load this store has had; and SQLite takes
one writer at a time, so those turns serialise. And a second control plane is
impossible while the store is a file on one machine's disk.

On the control plane, one command does all of it — installs PostgreSQL if the
machine has none, creates its role and database with a generated password kept
in `/etc/llmgw/secrets/store-dsn`, stops the gateway, copies the SQLite store
into it so every account and key survives, points `server.yaml` at it, and
starts the gateway again:

```
sudo llmgw server init --postgres
```

It can be given on the first run too, and then there is never a SQLite store.
The rest of this section is what that command does, for a PostgreSQL somewhere
else.

The setting is the same one, and the address decides:

```yaml
auth:
  store: "@/etc/llmgw/secrets/store-dsn"
```

A path is SQLite, a `postgres://` URL is PostgreSQL, and `@` followed by a path
reads the address out of that file — which is how a DSN carrying a password
stays out of a configuration that gets copied, diffed and pasted into issues.
Keep that file at `0600` and owned by the user llmgw runs as.

Moving an existing installation is a copy, not a fresh start. Keys are stored
hashed, so starting fresh means reissuing every one of them and being unable
to say which was which:

```
sudo systemctl stop llmgw-server
llmgw keys move-store --from /var/lib/llmgw/llmgw.db --to "@/etc/llmgw/secrets/store-dsn"
# point auth.store at the new address, then
sudo systemctl start llmgw-server
```

Stop the gateway first. The copy makes no attempt to follow a store being
written to: rows landing after the table they belong to has been passed would
simply be absent, and nothing would say so. The destination must be empty, and
the source is left untouched — so the way back is to point the configuration at
the file again.

On a machine that also runs the control plane, PostgreSQL's defaults assume it
owns the box. A gateway swapped out to give the database cache is a worse
outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
`max_connections = 40` are enough for a store whose working set is measured in
megabytes.

## Conversations across models

Off everywhere until a project asks. llmgw otherwise stores no message content
at all — usage rows carry token counts, costs and latencies, never text, and
even its own chat page keeps the conversation in the browser. A gateway that
keeps nothing anyone typed is a much easier thing to have in front of a team,
so keeping it is a decision someone makes rather than a default they discover.

```
llmgw keys threads --project default --retain-hours 24
```

The retention is required. A conversation store with no expiry is a transcript
of everything a team ever typed, kept forever because nobody chose a number.
The window is measured from the last turn, so a conversation still in progress
does not expire underneath it.

A client then sends a thread id and only its new turn:

```
curl $BASE/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "x-llmgw-thread: my-conversation" \
  -d '{"model": "<model-name>", "messages": [{"role":"user","content":"and then?"}]}'
```

llmgw puts the stored history in front of that turn and sends it to whichever
model is serving — **including a different one from the one the conversation
started on**. That is the point: a thread begun on a large model can continue
on a small one, and a caller dropped to a cheaper model by a spent budget keeps
their conversation.

When the history does not fit the model now serving, the oldest turns are
dropped until it does, and the reply carries `x-llmgw-thread-trimmed` with how
many. System messages are never dropped, and neither is the turn the caller
just sent.

`DELETE /v1/threads/{id}` forgets one. It is a client endpoint on purpose: the
person who said the thing should be able to unsay it without asking an
operator to read it first. A thread belonging to another project answers the
same as one that never existed.

Turning it off again stops new storage, and existing conversations go at the
next hourly prune:

```
llmgw keys threads --project default --off
```

## Memory notes for agents

Also off until a project asks, for the same reason. Where conversations are what
was said, notes are what an agent chose to keep, and they persist across
conversations:

```
llmgw keys memory --project default --retain-hours 720
```

The retention is required and is measured from the last write to a note. An agent
then saves, searches and deletes notes through `/v1/memory` or, as tools, through
the MCP server at `/mcp` (`remember`, `recall`, `forget`). Both need the API key
of the project, and notes are private to the project and to the agent id sent in
`x-llmgw-agent`. A note in another project or agent answers the same as one that
does not exist. llmgw does not add these tools to requests itself.

Turning it off stops new storage, and existing notes go at the next hourly
prune:

```
llmgw keys memory --project default --off
```

Without authentication, `threads.store` and `threads.retain_hours` in
`server.yaml` keep both conversations and notes in one shared pool. That is for
one trusted machine only; see [quickstart.md](quickstart.md).

## Two control planes

llmgw handles the hard half itself: only one places work, and agents find
whichever is leading. What it does not do is move client traffic, which
happens in front of both.

[deploy/ha/](../deploy/ha/) has the pattern and a keepalived example. The one
thing to get right is the health check: point it at `/ready`, not `/health`.
`/health` is 200 on a standby, which is alive and holds no endpoints at all,
so a load balancer using it would send half the traffic to something that
refuses every request.

## Upgrading

One binary on every machine, so an upgrade is a copy and a restart. The part
worth care is the copy, not the order.

**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its
next connection. The control plane holds no replica state across its own
restart either — it rebuilds the fleet from what the agents report. A rolling
upgrade of the whole cluster interrupts no request that is not in flight at the
moment of the restart.

**A mixed fleet is fine while you roll.** The agent protocol only ever gains
fields, and both sides treat an absent one as unknown rather than as a default.
Order does not matter; finish the roll the same day anyway, because
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
is one more thing to hold in your head.

```
VERSION=v0.9.0
# copy to every machine first, install nowhere yet
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
  ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
done
```

**Check every size before installing anything.** This is not defensive writing:
on a host whose login shell prints anything, `scp` fails outright, and the
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
and writes a *zero-byte file* while reporting success. Installing that on a
fleet takes the fleet down. Compare the number on each machine with the local
one, then:

```
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
sudo systemctl restart llmgw-agent     # or llmgw-server, or llmgw-mirror
llmgw --version
```

Then confirm what you have, rather than assuming:

```
llmgw ctl nodes       # every agent back, LAST SEEN in seconds
llmgw ctl replicas    # same replicas, adopted rather than replaced
```

A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
Restart it when you want the new profile to take effect; the flag is there so
the choice is yours rather than silent.

## Backing up

Three things, in descending order of how badly you want them.

**The certificate authority.** `ca-key.pem`, `/etc/llmgw/pki` unless you kept it elsewhere (step 2).
Losing it does not stop the cluster, and it means every node identity has to be
reissued the next time you touch one. Nothing in llmgw can reconstruct it.

**The store.** Users, API keys, organisations, teams, projects, every limit and
budget, all usage history, models added at runtime, and measured capacity.

```
# SQLite
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"

# PostgreSQL
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
```

`.backup` and `pg_dump` are both safe against a running gateway. Copying the
SQLite file with `cp` while it is being written is not.

**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued — which is why the store is worth backing up on a schedule rather than
before upgrades.

**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. Worth remembering that a pinned version may no longer be upstream to
download: that is exactly what pinning it protected you from, and a backup is
what makes the protection outlive the disk.

## Reclaiming disk

`llmgw ctl nodes` has a DISK column: free space against the size it is free
out of, for the filesystems holding model weights. A dash means the node was
not asked — an agent with no `model_dirs`, or one older than this — and not
that it has no space. Shared filesystems are excluded from a node's figure,
because space on a filer belongs to everyone mounting it.

Weights llmgw downloaded for models that are no longer in the catalog can be
listed and removed:

```
llmgw ctl weights             # what could be freed, per node
llmgw ctl weights --collect   # delete it
```

**Only files llmgw fetched itself are ever candidates.** They are recognisable
because llmgw names them after a hash of their source, so the set still wanted
is computed from the catalog exactly rather than guessed. Anything you put on
a node — by scp, by a shared mount, by hand — is left alone however
unreferenced it looks, and no llmgw command will remove it. The nodes check
every path again before deleting it, so a control plane that is wrong cannot
talk one into losing a model that took a week to copy.

Which also means llmgw will not reclaim the storage most likely to be filling
your disk. Weights you placed yourself are yours to remove.

A profile may declare `min_disk_gb`, and a node without that much free is then
not a candidate for it — checked only on nodes that do not already hold the
weights, since those need no room for them. Without the field, disk is not
considered, which is how every profile written before it keeps behaving.

## When something does not work

**The node is `ready` but every replica fails.** Almost always a missing engine
or weights. `llmgw ctl replicas` shows the shard's error and the engine's last
output, which is the actual diagnosis.

**A model will not place.** `llmgw ctl explain-placement <model>` names each
node and the reason: wrong GPU class, not enough free cards, too little memory,
no fabric, a label it does not carry.

**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
two. `llmgw ctl status` shows how full the cluster is.

**Tool calls come back as text.** The model made the call and the engine did not
parse it. `llmgw ctl tool-check --endpoint http://<node>:<port>` distinguishes
that from a model ignoring its tools, which has a different fix.

**Requests fail at exactly sixty seconds.** `timeouts.response_header` is a
liveness check for streaming and a total generation budget for non-streaming
requests. On a slow model, set `timeouts.generation` higher.

## What to read next

- `docs/choosing-models.md` — which models to run and how many, if the answer
  to "we have GPUs, now what" is not already obvious
- `docs/data.md` — what is stored and what leaves the machine, for the review
  that happens before this is allowed near anyone's documents
- `docs/reference.md` — every configuration key, generated from the source, so
  it describes the version you are running rather than the one somebody last
  wrote about
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
  and what belongs on which port
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
  each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
  checking a node, and what to look at when a replica spanning nodes will not run
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
  deliberately does not do
- `docs/onboarding.md` — giving people accounts and keys
- `llmgw-plan.md` — why the system is shaped the way it is

## Known gaps, so you find them here rather than at 2am

- **Every node fetches its own copy.** There is no seeding between nodes, so
  bringing up eight nodes on a new model is eight downloads. The mirror makes
  those eight local, which is most of the problem; on a fabric where the nodes
  can see each other, seeding between them is the obvious next thing to build.
- **A budget without a price never binds.** Set `default_price` or `pricing`
  (step 2) or every request costs zero, and a budget is something the dashboard
  displays rather than something that stops anything. The quotas page says so
  where it can, which is not the same as being told before you rely on it.
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
  and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
  install a vLLM built for your CUDA yourself.
- **Two control planes have not been run for long.** The leader election and
  agent failover work and there is a pattern in `deploy/ha/`, but every cluster
  this has been run on in anger has had one. A single control plane is the
  well-trodden path.
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
  scheduling, the fabric identity, the maps and the launch are written against
  the real tools' formats and tested against them, not yet run on hardware that
  has them. Single-node placement on real GPUs is the well-trodden path; if you
  are the first to run a TP group across real cards, run `llmgw doctor --json`
  on each node first and keep the reports.