Make the install guide describe the system that exists

Read start to finish as somebody standing up a new cluster, it sent them
down a path we no longer take.

The mirror was absent from all 821 lines, and the page still said that
distributing weights was "a known gap, not an oversight" — so the reader
would have gone away to solve, by hand, the problem the binary in front of
them solves. It is step 4 now, before the nodes, because where a node
fetches from is the difference between minutes and a day, and on a customer
network the nodes often have no route out at all.

Two of the listed gaps had been closed and were still listed: a fetch does
report its progress to the control plane, and there is an HA pattern. A
gap somebody believes in is worse than one they discover.

Added what this week made obvious was missing. Upgrading, including the
zero-byte copy that ssh -n silently produces and the size check that is the
only thing between that and a fleet on a truncated binary. Backing up, in
the order that matters, and the fact that API keys are hashed and do not
come back. Pricing in the first configuration, because a budget without a
price is displayed, counted against nothing, and never binds. And scale,
bench and the source check, none of which were written down anywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-28 00:01:39 +02:00
1 parent 03e38e7dda
commit d8fb198aa5
1 file changed
+291 -31
+291 -31
View File
@@ -13,7 +13,7 @@ this page says otherwise, and where it says otherwise it says so plainly.
Read this first, because it is the difference between an afternoon and a week.
llmgw runs engines and routes requests. It does **not** install GPU drivers,
CUDA, inference engines, or model weights. A node missing any of them still
CUDA, or the inference engines themselves. A node missing any of them still
registers, still reports `ready`, and still fails every replica placed on it —
so the useful order is to make a node able to serve *before* onboarding it, not
after.
@@ -23,12 +23,15 @@ after.
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
| Model weights on disk | Placement, routing, accounting |
| Model weights, from somewhere | Fetching them, placement, routing, accounting |
The reason is boring and unavoidable: drivers and engines are platform decisions
with kernel modules and version matrices attached, and distributing weights is
its own problem — hundreds of gigabytes that want a shared filesystem, not a
copy from somebody's laptop. It is a known gap, not an oversight.
with kernel modules and version matrices attached.
Weights are the one item on that list llmgw does handle. It includes a mirror —
step 4 — which holds the files on your own network and hands them to the nodes,
so getting weights onto every machine is not a problem you solve before
starting. What llmgw will not do is choose the models for you.
## 0. Before you begin
@@ -49,7 +52,10 @@ them, so nodes need no inbound access except your own SSH.
| --- | --- | --- |
| 8080 | clients, the web UI | your users, usually via a reverse proxy |
| 7443 | node agents | the node network only |
| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
| 8081 | node agents fetching weights | the node network only (the mirror machine) |
| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
Decide now which address nodes will use to reach the control plane. It goes into
a certificate and changing it later means reissuing one.
@@ -148,11 +154,22 @@ auth:
# that is the bill or the outage.
default_rpm: 60
default_tpm: 200000
# What a request costs, which is what makes a budget mean anything. Without
# a price every request costs zero, so a budget is displayed, counted against
# nothing, and never binds — and nobody finds out until they expected it to
# stop something. Self-hosted models have no per-token price, so the unit is
# yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
# turns a budget into a plain token quota.
default_price:
prompt_per_1k: 1000
completion_per_1k: 1000
# pricing:
# qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
landing:
contact: "platform@example.com"
# Start with none. Models are added while the cluster runs — see step 7 —
# Start with none. Models are added while the cluster runs — see step 8 —
# and anything listed here is declarative: it is in the catalog on every start
# and cannot be removed without editing this file.
models: []
@@ -200,7 +217,103 @@ sudo -u llmgw llmgw ctl status
been added yet. It is worth running now so its output is familiar before you
need it.
## 4. Preparing a node
## 4. The mirror
Do this before the nodes, not after. Every node fetches its own copy of a
model's weights, and where they fetch from decides whether bringing up a cluster
takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
Face is a day's work, and on a good number of corporate networks it is not
possible at all — the nodes have no route out, which is usually the point.
The mirror is one machine holding the files on your own network. It is the same
binary again, and it is the highest-value piece of infrastructure around llmgw
for the least work.
### Running it
```
sudo install -d -o llmgw -g llmgw /srv/models
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
```
In production, the unit:
```
sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
```
**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
what every agent talks to, and it is plain HTTP because a weights fetch is a
public file transfer that is checksummed anyway. `:8082` is the management API:
it can start downloads and delete models, and only the control plane needs it.
The split is what lets you lock the second one down without re-certifying every
agent that fetches from the first.
Give the management port the cluster's own CA and it takes client certificates:
```
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
--tls-ca /etc/llmgw/ca.pem \
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
```
Without those three flags the management port is open to anyone who can reach
it, and the mirror says so in its own log at every start rather than letting you
find out later.
### The layout
```
/srv/models/
qwen2.5-14b-instruct/
2026-03/
q4_k_m.gguf
```
Model, then version, then files. The version directory is the point of the
layout rather than decoration: a repository that changes upstream changes what
you are running, and the first sign is a prompt that worked last week answering
differently this week with nobody able to say why. Pin it here and that cannot
happen.
A version holding several files is a sharded model, and is served as a set: the
agent asks the directory what it consists of and fetches all of it into one
place, marking it complete only when the last file lands. A catalogue entry for
one names the directory, trailing slash included.
### Telling the control plane about it
```yaml
# /etc/llmgw/server.yaml
mirror:
api: https://models.internal:8082
tls:
ca: /etc/llmgw/ca.pem
cert: /etc/llmgw/server.pem
key: /etc/llmgw/server-key.pem
```
With this set, `/admin/mirror` appears: what the mirror holds, what it is
downloading and how far along, a search of Hugging Face, and a form that puts a
mirrored model into service without anyone writing YAML. Without it that page
stays unmounted and the mirror is still perfectly usable — a model's `source` is
just a URL, and the URL can be yours.
If the control plane cannot reach the internet, `mirror.registry` points the
search at an internal registry mirror instead.
### Filling it
```
llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
```
or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.
## 5. Preparing a node
**Do this before onboarding.** A node missing its driver or engine registers
healthy and fails on first placement, and the reason is three logs away.
@@ -305,10 +418,13 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
### Weights
Nothing to do here if you built the mirror in step 4 — the agent fetches what a
model's `source` names, and that source is your mirror. This section is for the
arrangements where the files are already on the machine.
Put them where the node can read them. `scp`, a shared filesystem, or the
engine's own downloader are all valid, and which you want depends on how many
nodes you have. If you would rather not choose, skip to the next section and
let llmgw fetch them.
nodes you have.
```
sudo install -d -o root -g root /var/lib/llmgw-models
@@ -325,11 +441,18 @@ model's `source` at a file to download and the agent fetches it once, before the
engine is launched:
```yaml
# your mirror, which is what step 4 was for
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
# a whole version, for a model of several files — the trailing slash is the signal
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
# or straight from Hugging Face, where the nodes can reach it
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
# or an ordinary URL
source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
```
A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
the dashboard both show how far along a node is. A replica pulling a hundred
gigabytes and one that is wedged do not look alike.
Eight shards of one replica starting together produce one download, not eight.
An interrupted fetch resumes rather than starting over, and a partial file is
never visible under the model's name — so a half-downloaded model cannot be
@@ -375,7 +498,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
bf16 is 141 GB and the same model at q4 is 43 GB.
## 5. Adding the node
## 6. Adding the node
Two halves, because they need different access: building the machine needs the
hypervisor, and issuing its identity needs the certificate authority. Where one
@@ -432,7 +555,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
name; nodes with no InfiniBand take no `--fabric` at all and are each their own
island, which is fine for single-node models.
## 6. Setting the node's CPU model, if it is a VM
## 7. Setting the node's CPU model, if it is a VM
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
from the guest. Inference then falls back to scalar code and is not slightly
@@ -449,7 +572,7 @@ only change being `--cpu host`:
probably did not, and nothing about it looks broken — every number is simply
small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
## 7. Adding a model
## 8. Adding a model
Write the definition. This is the same shape a model takes in `server.yaml`, so
an entry moves between the two by being copied:
@@ -458,7 +581,7 @@ an entry moves between the two by being copied:
# qwen-coder-3b.yaml
name: qwen2.5-coder-3b-instruct
aliases: [coder]
source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
routing_profile: chat
# What a client may use in one request. Worth stating: llama.cpp divides
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
@@ -481,6 +604,17 @@ llmgw ctl model add -f qwen-coder-3b.yaml
llmgw ctl models
```
Adding a model checks that the source is actually there before accepting it, so
a typo or a mirror that is down is a sentence now rather than a replica failing
on a node in five minutes with an error naming the node. `--no-source-check`
skips it, which is right when the nodes reach the mirror by an address this
machine does not.
With a mirror configured you can skip the file entirely: `/admin/mirror` lists
what the mirror holds and offers a form that writes the same definition, with
the engine, the source and the memory figure already filled in from the file
itself.
```
MODEL SERVING REPLICAS STATUS ALIASES
qwen2.5-coder-3b-instruct yes 1/1 ready coder
@@ -545,7 +679,42 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
which names every candidate and why each was rejected, rather than leaving you
to infer it.
## 8. The first request
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
definition:
```
llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
```
Models added at runtime only. One declared in `server.yaml` is read again on
every start, so a number set here would hold until the next restart and then
revert — change it in the file instead.
How much traffic each replica is given is the profile's `weight`, and guessing
it is how a cluster ends up with one saturated replica beside two idle ones.
Measure it instead:
```
llmgw ctl bench qwen2.5-coder-3b-instruct --save
```
Concurrency climbs until the latency target stops being met, and the last level
that held it is the capacity — peak throughput arrives long after latency has
become unusable, so throughput alone would overstate what the hardware serves.
It prints the `weight` to put in the profile, and `--save` records the figure
against the model so the dashboard can show what the cluster is measured to
carry rather than what somebody typed.
This drives real traffic and competes with real requests. Run it against a
cordoned replica, or in a window where that is acceptable.
An engine that publishes no metrics endpoint is measured from the outside only.
The run says so, and queue depth and KV occupancy are then not part of the
answer.
## 9. The first request
```
curl http://<control-plane>:8080/v1/chat/completions \
@@ -586,8 +755,10 @@ One scrape target for the whole fleet. The node and GPU figures come from the
control plane, which already holds them, so there is nothing to install on the
nodes and no Prometheus configuration per machine.
A starting configuration, with the alerts worth having, is in
[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml).
A starting configuration, with the alerts worth having and a provisioned
dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
to run it, which is not the control plane: monitoring that dies with the thing
it monitors tells you nothing at the moment you need it.
Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
rather than zero when the driver does not report it — a flat zero across a
@@ -730,6 +901,91 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
so a load balancer using it would send half the traffic to something that
refuses every request.
## Upgrading
One binary on every machine, so an upgrade is a copy and a restart. The part
worth care is the copy, not the order.
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
keep running when the agent restarts, and the agent reports them back on its
next connection. The control plane holds no replica state across its own
restart either — it rebuilds the fleet from what the agents report. A rolling
upgrade of the whole cluster interrupts no request that is not in flight at the
moment of the restart.
**A mixed fleet is fine while you roll.** The agent protocol only ever gains
fields, and both sides treat an absent one as unknown rather than as a default.
Order does not matter; finish the roll the same day anyway, because
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
is one more thing to hold in your head.
```
VERSION=v0.9.0
# copy to every machine first, install nowhere yet
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
done
```
**Check every size before installing anything.** This is not defensive writing:
on a host whose login shell prints anything, `scp` fails outright, and the
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
and writes a *zero-byte file* while reporting success. Installing that on a
fleet takes the fleet down. Compare the number on each machine with the local
one, then:
```
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror
llmgw --version
```
Then confirm what you have, rather than assuming:
```
llmgw ctl nodes # every agent back, LAST SEEN in seconds
llmgw ctl replicas # same replicas, adopted rather than replaced
```
A replica may come back marked **`ready (stale profile)`**. That means it is
running a profile that has since been edited — the reconciler leaves a serving
replica alone on purpose, because churning one costs more than the difference.
Restart it when you want the new profile to take effect; the flag is there so
the choice is yours rather than silent.
## Backing up
Three things, in descending order of how badly you want them.
**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
Losing it does not stop the cluster, and it means every node identity has to be
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
**The store.** Users, API keys, organisations, teams, projects, every limit and
budget, all usage history, models added at runtime, and measured capacity.
```
# SQLite
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
# PostgreSQL
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
```
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
SQLite file with `cp` while it is being written is not.
**Keys are stored hashed and cannot be recovered from a backup or from
anywhere else.** Restoring an old store restores which keys exist and who owns
them, not their secrets. A key created after the backup is gone and has to be
reissued — which is why the store is worth backing up on a schedule rather than
before upgrades.
**The mirror's directory** is the one thing you can rebuild, by downloading it
all again. Worth remembering that a pinned version may no longer be upstream to
download: that is exactly what pinning it protected you from, and a backup is
what makes the protection outlive the disk.
## Reclaiming disk
`llmgw ctl nodes` has a DISK column: free space against the size it is free
@@ -772,7 +1028,7 @@ output, which is the actual diagnosis.
node and the reason: wrong GPU class, not enough free cards, too little memory,
no fabric, a label it does not carry.
**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
two. `llmgw ctl status` shows how full the cluster is.
@@ -793,6 +1049,10 @@ requests. On a slow model, set `timeouts.generation` higher.
- `docs/reference.md` — every configuration key, generated from the source, so
it describes the version you are running rather than the one somebody last
wrote about
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
and what belongs on which port
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
each one assumes about this system
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
deliberately does not do
- `docs/onboarding.md` — giving people accounts and keys
@@ -800,20 +1060,20 @@ requests. On a slow model, set `timeouts.generation` higher.
## Known gaps, so you find them here rather than at 2am
- **A fetch reports its progress where nobody is looking.** The agent logs a
percentage as it downloads, so `journalctl -u llmgw-agent` on that node knows
exactly how far along it is. Nothing carries it to the control plane, so from
`ctl` or the dashboard a replica pulling a hundred gigabytes and one that is
wedged still look identical — which is the comparison the log line itself
says matters.
- **Every node fetches its own copy.** There is no seeding between nodes, so
bringing up eight nodes on a new model is eight downloads from the same
origin. On a fabric where the nodes can see each other this is wasteful, and
it is the obvious next thing to build.
bringing up eight nodes on a new model is eight downloads. The mirror makes
those eight local, which is most of the problem; on a fabric where the nodes
can see each other, seeding between them is the obvious next thing to build.
- **A budget without a price never binds.** Set `default_price` or `pricing`
(step 3) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **Engines are yours to install.** Onboarding verifies one runs; it installs
nothing.
- **One control plane.** No HA yet; it is a single process with a SQLite file.
Back up `/var/lib/llmgw/llmgw.db`.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
well-trodden path.
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
the fabric invariant and the rendezvous are exercised against simulated
hardware only. Single-node placement on real GPUs is the well-trodden path;