Docs: one way in — quick-install starts at the setup screens, quickstart becomes development, install.md no longer says engines are yours to install

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 17:32:54 +02:00
1 parent 4979e08d64
commit 6abd510628
5 files changed
+103 -72

No files matched your search

+20 -20
View File
@@ -13,34 +13,32 @@ this page says otherwise, and where it says otherwise it says so plainly.
Read this first, because it is the difference between an afternoon and a week.
llmgw runs engines and routes requests. It does **not** install GPU drivers,
CUDA, or the inference engines themselves. A node missing any of them still
registers, still reports `ready`, and still fails every replica placed on it —
so the useful order is to make a node able to serve *before* onboarding it, not
after.
llmgw installs itself, its engines and its store. It does **not** install the
GPU driver or the InfiniBand stack: those are kernel modules with version
matrices attached, and a platform decision. A node missing them still
registers and reports `ready`, and fails every replica that needs the GPU or
the fabric — so make a node able to serve *before* onboarding it.
`llmgw doctor` on the node says what is missing and the command that fixes it.
| You provide | llmgw provides |
| --- | --- |
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
| Model weights, from somewhere | Fetching them, placement, routing, accounting |
| Nothing — the nodes install them | llama.cpp, vLLM (and SGLang when asked), with the right build for each node's driver and GPU: [Engines](#engines) |
| Which models to run | The mirror that fetches them once, placement, routing, accounting |
The reason is boring and unavoidable: drivers and engines are platform decisions
with kernel modules and version matrices attached.
Weights are the one item on that list llmgw does handle. It includes a mirror —
step 3 — which holds the files on your own network and hands them to the nodes,
so getting weights onto every machine is not a problem you solve before
starting. What llmgw will not do is choose the models for you.
Weights are handled too. llmgw includes a mirror — step 3 — which holds the
files on your own network and hands them to the nodes, so getting weights onto
every machine is not a problem you solve before starting. What llmgw will not do is choose the models for you.
## 0. Before you begin
Four machines is a reasonable first cluster: one control plane and three nodes.
One of each also works.
**The control plane** needs no GPU. It is a single process holding a SQLite
file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable
**The control plane** needs no GPU: one machine, holding its store in SQLite
or PostgreSQL, or three for high availability, with the store in PostgreSQL replicated under
Patroni. 2 GB of RAM and a few gigabytes of disk each is generous. It must be reachable
from every node on the agent port, and from your users on the client port.
**Nodes** need whatever the model needs, plus about 8 GB for the operating
@@ -67,9 +65,11 @@ a certificate: the machine's own addresses are found for you, and any other name
One static binary, no runtime dependencies, the same on every machine.
```
VERSION=v0.8.0 # check the releases page for the current tag
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64
curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
llmgw --version
```
@@ -1101,7 +1101,7 @@ llmgw keys memory --project default --off
Without authentication, `threads.store` and `threads.retain_hours` in
`server.yaml` keep both conversations and notes in one shared pool. That is for
one trusted machine only; see [quickstart.md](quickstart.md).
one trusted machine only; see [development.md](development.md).
## Two control planes