From d8fb198aa505bc37c2703a395608115075989737 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Mon, 28 Sep 2026 00:01:39 +0200 Subject: [PATCH] Make the install guide describe the system that exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Read start to finish as somebody standing up a new cluster, it sent them down a path we no longer take. The mirror was absent from all 821 lines, and the page still said that distributing weights was "a known gap, not an oversight" — so the reader would have gone away to solve, by hand, the problem the binary in front of them solves. It is step 4 now, before the nodes, because where a node fetches from is the difference between minutes and a day, and on a customer network the nodes often have no route out at all. Two of the listed gaps had been closed and were still listed: a fetch does report its progress to the control plane, and there is an HA pattern. A gap somebody believes in is worse than one they discover. Added what this week made obvious was missing. Upgrading, including the zero-byte copy that ssh -n silently produces and the size check that is the only thing between that and a fleet on a truncated binary. Backing up, in the order that matters, and the fact that API keys are hashed and do not come back. Pricing in the first configuration, because a budget without a price is displayed, counted against nothing, and never binds. And scale, bench and the source check, none of which were written down anywhere. Co-Authored-By: Claude Opus 5 (1M context) --- docs/install.md | 322 +++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 291 insertions(+), 31 deletions(-) diff --git a/docs/install.md b/docs/install.md index aece215..3e1f88f 100644 --- a/docs/install.md +++ b/docs/install.md @@ -13,7 +13,7 @@ this page says otherwise, and where it says otherwise it says so plainly. Read this first, because it is the difference between an afternoon and a week. llmgw runs engines and routes requests. It does **not** install GPU drivers, -CUDA, inference engines, or model weights. A node missing any of them still +CUDA, or the inference engines themselves. A node missing any of them still registers, still reports `ready`, and still fails every replica placed on it — so the useful order is to make a node able to serve *before* onboarding it, not after. @@ -23,12 +23,15 @@ after. | NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health | | InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one | | vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision | -| Model weights on disk | Placement, routing, accounting | +| Model weights, from somewhere | Fetching them, placement, routing, accounting | The reason is boring and unavoidable: drivers and engines are platform decisions -with kernel modules and version matrices attached, and distributing weights is -its own problem — hundreds of gigabytes that want a shared filesystem, not a -copy from somebody's laptop. It is a known gap, not an oversight. +with kernel modules and version matrices attached. + +Weights are the one item on that list llmgw does handle. It includes a mirror — +step 4 — which holds the files on your own network and hands them to the nodes, +so getting weights onto every machine is not a problem you solve before +starting. What llmgw will not do is choose the models for you. ## 0. Before you begin @@ -49,7 +52,10 @@ them, so nodes need no inbound access except your own SSH. | --- | --- | --- | | 8080 | clients, the web UI | your users, usually via a reverse proxy | | 7443 | node agents | the node network only | -| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap | +| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap | +| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here | +| 8081 | node agents fetching weights | the node network only (the mirror machine) | +| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads | Decide now which address nodes will use to reach the control plane. It goes into a certificate and changing it later means reissuing one. @@ -148,11 +154,22 @@ auth: # that is the bill or the outage. default_rpm: 60 default_tpm: 200000 + # What a request costs, which is what makes a budget mean anything. Without + # a price every request costs zero, so a budget is displayed, counted against + # nothing, and never binds — and nobody finds out until they expected it to + # stop something. Self-hosted models have no per-token price, so the unit is + # yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which + # turns a budget into a plain token quota. + default_price: + prompt_per_1k: 1000 + completion_per_1k: 1000 + # pricing: + # qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 } landing: contact: "platform@example.com" -# Start with none. Models are added while the cluster runs — see step 7 — +# Start with none. Models are added while the cluster runs — see step 8 — # and anything listed here is declarative: it is in the catalog on every start # and cannot be removed without editing this file. models: [] @@ -200,7 +217,103 @@ sudo -u llmgw llmgw ctl status been added yet. It is worth running now so its output is familiar before you need it. -## 4. Preparing a node +## 4. The mirror + +Do this before the nodes, not after. Every node fetches its own copy of a +model's weights, and where they fetch from decides whether bringing up a cluster +takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging +Face is a day's work, and on a good number of corporate networks it is not +possible at all — the nodes have no route out, which is usually the point. + +The mirror is one machine holding the files on your own network. It is the same +binary again, and it is the highest-value piece of infrastructure around llmgw +for the least work. + +### Running it + +``` +sudo install -d -o llmgw -g llmgw /srv/models +llmgw mirror serve --dir /srv/models --base http://models.internal:8081 +``` + +In production, the unit: + +``` +sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/ +sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror +``` + +**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is +what every agent talks to, and it is plain HTTP because a weights fetch is a +public file transfer that is checksummed anyway. `:8082` is the management API: +it can start downloads and delete models, and only the control plane needs it. +The split is what lets you lock the second one down without re-certifying every +agent that fetches from the first. + +Give the management port the cluster's own CA and it takes client certificates: + +``` +llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \ + --tls-ca /etc/llmgw/ca.pem \ + --tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem +``` + +Without those three flags the management port is open to anyone who can reach +it, and the mirror says so in its own log at every start rather than letting you +find out later. + +### The layout + +``` +/srv/models/ + qwen2.5-14b-instruct/ + 2026-03/ + q4_k_m.gguf +``` + +Model, then version, then files. The version directory is the point of the +layout rather than decoration: a repository that changes upstream changes what +you are running, and the first sign is a prompt that worked last week answering +differently this week with nobody able to say why. Pin it here and that cannot +happen. + +A version holding several files is a sharded model, and is served as a set: the +agent asks the directory what it consists of and fetches all of it into one +place, marking it complete only when the last file lands. A catalogue entry for +one names the directory, trailing slash included. + +### Telling the control plane about it + +```yaml +# /etc/llmgw/server.yaml +mirror: + api: https://models.internal:8082 + tls: + ca: /etc/llmgw/ca.pem + cert: /etc/llmgw/server.pem + key: /etc/llmgw/server-key.pem +``` + +With this set, `/admin/mirror` appears: what the mirror holds, what it is +downloading and how far along, a search of Hugging Face, and a form that puts a +mirrored model into service without anyone writing YAML. Without it that page +stays unmounted and the mirror is still perfectly usable — a model's `source` is +just a URL, and the URL can be yours. + +If the control plane cannot reach the internet, `mirror.registry` points the +search at an internal registry mirror instead. + +### Filling it + +``` +llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \ + hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf +``` + +or from the admin page, or by copying files into the directory by hand — the +mirror is a directory, and anything that puts files in it works. + +## 5. Preparing a node **Do this before onboarding.** A node missing its driver or engine registers healthy and fails on first placement, and the reason is three logs away. @@ -305,10 +418,13 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm ### Weights +Nothing to do here if you built the mirror in step 4 — the agent fetches what a +model's `source` names, and that source is your mirror. This section is for the +arrangements where the files are already on the machine. + Put them where the node can read them. `scp`, a shared filesystem, or the engine's own downloader are all valid, and which you want depends on how many -nodes you have. If you would rather not choose, skip to the next section and -let llmgw fetch them. +nodes you have. ``` sudo install -d -o root -g root /var/lib/llmgw-models @@ -325,11 +441,18 @@ model's `source` at a file to download and the agent fetches it once, before the engine is launched: ```yaml +# your mirror, which is what step 4 was for +source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf +# a whole version, for a model of several files — the trailing slash is the signal +source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/ +# or straight from Hugging Face, where the nodes can reach it source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf -# or an ordinary URL -source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf ``` +A fetch reports its progress to the control plane, so `llmgw ctl replicas` and +the dashboard both show how far along a node is. A replica pulling a hundred +gigabytes and one that is wedged do not look alike. + Eight shards of one replica starting together produce one download, not eight. An interrupted fetch resumes rather than starting over, and a partial file is never visible under the model's name — so a half-downloaded model cannot be @@ -375,7 +498,7 @@ How much disk a node needs follows from the models rather than a rule of thumb. `llmgw ctl model plan ` prints the sizes; as a starting point, a 70B at bf16 is 141 GB and the same model at q4 is 43 GB. -## 5. Adding the node +## 6. Adding the node Two halves, because they need different access: building the machine needs the hypervisor, and issuing its identity needs the certificate authority. Where one @@ -432,7 +555,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric` name; nodes with no InfiniBand take no `--fabric` at all and are each their own island, which is fine for single-node models. -## 6. Setting the node's CPU model, if it is a VM +## 7. Setting the node's CPU model, if it is a VM Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2 from the guest. Inference then falls back to scalar code and is not slightly @@ -449,7 +572,7 @@ only change being `--cpu host`: probably did not, and nothing about it looks broken — every number is simply small. Check with `grep -qw avx2 /proc/cpuinfo` on the node. -## 7. Adding a model +## 8. Adding a model Write the definition. This is the same shape a model takes in `server.yaml`, so an entry moves between the two by being copied: @@ -458,7 +581,7 @@ an entry moves between the two by being copied: # qwen-coder-3b.yaml name: qwen2.5-coder-3b-instruct aliases: [coder] -source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf +source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf routing_profile: chat # What a client may use in one request. Worth stating: llama.cpp divides # --ctx-size across its parallel slots, so a pool of 65536 over two slots is @@ -481,6 +604,17 @@ llmgw ctl model add -f qwen-coder-3b.yaml llmgw ctl models ``` +Adding a model checks that the source is actually there before accepting it, so +a typo or a mirror that is down is a sentence now rather than a replica failing +on a node in five minutes with an error naming the node. `--no-source-check` +skips it, which is right when the nodes reach the mirror by an address this +machine does not. + +With a mirror configured you can skip the file entirely: `/admin/mirror` lists +what the mirror holds and offers a form that writes the same definition, with +the engine, the source and the memory figure already filled in from the file +itself. + ``` MODEL SERVING REPLICAS STATUS ALIASES qwen2.5-coder-3b-instruct yes 1/1 ready coder @@ -545,7 +679,42 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct which names every candidate and why each was rejected, rather than leaving you to infer it. -## 8. The first request +### How many replicas, and how much each one carries + +Replica count is changed while the cluster runs, without editing the +definition: + +``` +llmgw ctl scale qwen2.5-coder-3b-instruct --min 3 +``` + +Models added at runtime only. One declared in `server.yaml` is read again on +every start, so a number set here would hold until the next restart and then +revert — change it in the file instead. + +How much traffic each replica is given is the profile's `weight`, and guessing +it is how a cluster ends up with one saturated replica beside two idle ones. +Measure it instead: + +``` +llmgw ctl bench qwen2.5-coder-3b-instruct --save +``` + +Concurrency climbs until the latency target stops being met, and the last level +that held it is the capacity — peak throughput arrives long after latency has +become unusable, so throughput alone would overstate what the hardware serves. +It prints the `weight` to put in the profile, and `--save` records the figure +against the model so the dashboard can show what the cluster is measured to +carry rather than what somebody typed. + +This drives real traffic and competes with real requests. Run it against a +cordoned replica, or in a window where that is acceptable. + +An engine that publishes no metrics endpoint is measured from the outside only. +The run says so, and queue depth and KV occupancy are then not part of the +answer. + +## 9. The first request ``` curl http://:8080/v1/chat/completions \ @@ -586,8 +755,10 @@ One scrape target for the whole fleet. The node and GPU figures come from the control plane, which already holds them, so there is nothing to install on the nodes and no Prometheus configuration per machine. -A starting configuration, with the alerts worth having, is in -[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml). +A starting configuration, with the alerts worth having and a provisioned +dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where +to run it, which is not the control plane: monitoring that dies with the thing +it monitors tells you nothing at the moment you need it. Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent rather than zero when the driver does not report it — a flat zero across a @@ -730,6 +901,91 @@ thing to get right is the health check: point it at `/ready`, not `/health`. so a load balancer using it would send half the traffic to something that refuses every request. +## Upgrading + +One binary on every machine, so an upgrade is a copy and a restart. The part +worth care is the copy, not the order. + +**Nothing stops serving.** Engines are started with `KillMode=process`, so they +keep running when the agent restarts, and the agent reports them back on its +next connection. The control plane holds no replica state across its own +restart either — it rebuilds the fleet from what the agents report. A rolling +upgrade of the whole cluster interrupts no request that is not in flight at the +moment of the restart. + +**A mixed fleet is fine while you roll.** The agent protocol only ever gains +fields, and both sides treat an absent one as unknown rather than as a default. +Order does not matter; finish the roll the same day anyway, because +`ctl nodes` reports each agent's version and a fleet that disagrees with itself +is one more thing to hold in your head. + +``` +VERSION=v0.9.0 +# copy to every machine first, install nowhere yet +for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do + ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64 +done +``` + +**Check every size before installing anything.** This is not defensive writing: +on a host whose login shell prints anything, `scp` fails outright, and the +obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null` +and writes a *zero-byte file* while reporting success. Installing that on a +fleet takes the fleet down. Compare the number on each machine with the local +one, then: + +``` +sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw +sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror +llmgw --version +``` + +Then confirm what you have, rather than assuming: + +``` +llmgw ctl nodes # every agent back, LAST SEEN in seconds +llmgw ctl replicas # same replicas, adopted rather than replaced +``` + +A replica may come back marked **`ready (stale profile)`**. That means it is +running a profile that has since been edited — the reconciler leaves a serving +replica alone on purpose, because churning one costs more than the difference. +Restart it when you want the new profile to take effect; the flag is there so +the choice is yours rather than silent. + +## Backing up + +Three things, in descending order of how badly you want them. + +**The certificate authority.** `ca-key.pem`, wherever you put it in step 2. +Losing it does not stop the cluster, and it means every node identity has to be +reissued the next time you touch one. Nothing in llmgw can reconstruct it. + +**The store.** Users, API keys, organisations, teams, projects, every limit and +budget, all usage history, models added at runtime, and measured capacity. + +``` +# SQLite +sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'" + +# PostgreSQL +pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump +``` + +`.backup` and `pg_dump` are both safe against a running gateway. Copying the +SQLite file with `cp` while it is being written is not. + +**Keys are stored hashed and cannot be recovered from a backup or from +anywhere else.** Restoring an old store restores which keys exist and who owns +them, not their secrets. A key created after the backup is gone and has to be +reissued — which is why the store is worth backing up on a schedule rather than +before upgrades. + +**The mirror's directory** is the one thing you can rebuild, by downloading it +all again. Worth remembering that a pinned version may no longer be upstream to +download: that is exactly what pinning it protected you from, and a backup is +what makes the protection outlive the disk. + ## Reclaiming disk `llmgw ctl nodes` has a DISK column: free space against the size it is free @@ -772,7 +1028,7 @@ output, which is the actual diagnosis. node and the reason: wrong GPU class, not enough free cards, too little memory, no fabric, a label it does not carry. -**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the +**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of two. `llmgw ctl status` shows how full the cluster is. @@ -793,6 +1049,10 @@ requests. On a slow model, set `timeouts.generation` higher. - `docs/reference.md` — every configuration key, generated from the source, so it describes the version you are running rather than the one somebody last wrote about +- `deploy/mirror/README.md` — the mirror in more detail, including the layout + and what belongs on which port +- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what + each one assumes about this system - `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding deliberately does not do - `docs/onboarding.md` — giving people accounts and keys @@ -800,20 +1060,20 @@ requests. On a slow model, set `timeouts.generation` higher. ## Known gaps, so you find them here rather than at 2am -- **A fetch reports its progress where nobody is looking.** The agent logs a - percentage as it downloads, so `journalctl -u llmgw-agent` on that node knows - exactly how far along it is. Nothing carries it to the control plane, so from - `ctl` or the dashboard a replica pulling a hundred gigabytes and one that is - wedged still look identical — which is the comparison the log line itself - says matters. - **Every node fetches its own copy.** There is no seeding between nodes, so - bringing up eight nodes on a new model is eight downloads from the same - origin. On a fabric where the nodes can see each other this is wasteful, and - it is the obvious next thing to build. + bringing up eight nodes on a new model is eight downloads. The mirror makes + those eight local, which is most of the problem; on a fabric where the nodes + can see each other, seeding between them is the obvious next thing to build. +- **A budget without a price never binds.** Set `default_price` or `pricing` + (step 3) or every request costs zero, and a budget is something the dashboard + displays rather than something that stops anything. The quotas page says so + where it can, which is not the same as being told before you rely on it. - **Engines are yours to install.** Onboarding verifies one runs; it installs nothing. -- **One control plane.** No HA yet; it is a single process with a SQLite file. - Back up `/var/lib/llmgw/llmgw.db`. +- **Two control planes have not been run for long.** The leader election and + agent failover work and there is a pattern in `deploy/ha/`, but every cluster + this has been run on in anger has had one. A single control plane is the + well-trodden path. - **Multi-node replicas have not been run on real InfiniBand.** The scheduling, the fabric invariant and the rendezvous are exercised against simulated hardware only. Single-node placement on real GPUs is the well-trodden path;