diff --git a/README.md b/README.md index f4a6b61..5eae3d3 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,18 @@ heterogeneous GPUs. It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). +## Getting started + +Starting from nothing — no packages installed, no drivers, no nodes — follow +[docs/install.md](docs/install.md): it builds a control plane, adds a node and +serves a model, and says plainly which parts llmgw does not install for you. + +| Guide | For | +|---|---| +| [docs/install.md](docs/install.md) | Standing up a cluster from scratch | +| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists | +| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys | + ## Status Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..f0f954d --- /dev/null +++ b/docs/install.md @@ -0,0 +1,472 @@ +# Installing llmgw from nothing + +Written for whoever is standing up the cluster: you have servers, you have root +on them, and nothing is installed yet. By the end there is a control plane, a +node in the fleet, and a model answering requests. + +It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond +the package names. Everything here has been done on a real cluster except where +this page says otherwise, and where it says otherwise it says so plainly. + +## What llmgw does not install + +Read this first, because it is the difference between an afternoon and a week. + +llmgw runs engines and routes requests. It does **not** install GPU drivers, +CUDA, inference engines, or model weights. A node missing any of them still +registers, still reports `ready`, and still fails every replica placed on it — +so the useful order is to make a node able to serve *before* onboarding it, not +after. + +| You provide | llmgw provides | +| --- | --- | +| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health | +| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one | +| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision | +| Model weights on disk | Placement, routing, accounting | + +The reason is boring and unavoidable: drivers and engines are platform decisions +with kernel modules and version matrices attached, and distributing weights is +its own problem — hundreds of gigabytes that want a shared filesystem, not a +copy from somebody's laptop. It is a known gap, not an oversight. + +## 0. Before you begin + +Four machines is a reasonable first cluster: one control plane and three nodes. +One of each also works. + +**The control plane** needs no GPU. It is a single process holding a SQLite +file; 2 GB of RAM and a few gigabytes of disk is generous. It must be reachable +from every node on the agent port, and from your users on the client port. + +**Nodes** need whatever the model needs, plus about 8 GB for the operating +system and the agent. They dial the control plane; the control plane never dials +them, so nodes need no inbound access except your own SSH. + +**Ports** + +| Port | Who connects | Expose to | +| --- | --- | --- | +| 8080 | clients, the web UI | your users, usually via a reverse proxy | +| 7443 | node agents | the node network only | +| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap | + +Decide now which address nodes will use to reach the control plane. It goes into +a certificate and changing it later means reissuing one. + +## 1. The binary + +One static binary, no runtime dependencies, the same on every machine. + +``` +VERSION=v0.6.0 # check the releases page for the current tag +curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/llmgw-$VERSION-linux-amd64 +curl -fsSL https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION/checksums.txt | sha256sum -c --ignore-missing +sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw +llmgw --version +``` + +`arm64` and `darwin` builds are published alongside. They are built with +`CGO_ENABLED=0` deliberately, so a binary built today runs on a node whose libc +is older than the builder's. + +To build instead of download, you need Go 1.26 or later and nothing else: + +``` +git clone https://gitea.mynodes.xyz/ilker/llmgw && cd llmgw +CGO_ENABLED=0 go build -o llmgw ./cmd/llmgw +``` + +## 2. The certificate authority + +Agents authenticate to the control plane with mutual TLS, and a node's identity +is taken from its certificate's common name. A public authority cannot supply +these: Let's Encrypt and its peers stopped issuing certificates carrying the +`clientAuth` usage, so nothing they hand out can authenticate an agent. + +Run this wherever you intend to keep the authority. It talks to nothing and +works on files, so it does not have to be a cluster machine — and there is an +argument that it should not be. + +``` +sudo llmgw ctl cert init-ca --dir /etc/llmgw/pki +``` + +That writes `ca.pem` and `ca-key.pem`. **Whoever holds `ca-key.pem` can mint any +node's identity**, so it stays here; `ca.pem` is copied to every machine and is +not a secret. + +Now the control plane's own certificate. Every address an agent might dial must +appear in `--host`, bare IP addresses included — hostname verification reads +subject alternative names and ignores the common name entirely, so a certificate +naming the host only in its subject is rejected by every client: + +``` +sudo llmgw ctl cert issue-server llmgw-ctl \ + --host 10.0.0.140 --host llmgw-ctl.internal \ + --dir /etc/llmgw/pki --out /etc/llmgw +``` + +This covers the fleet-facing port only. The client-facing API is public and +belongs behind an ordinary publicly trusted certificate, usually terminated by a +reverse proxy in front of port 8080. + +## 3. The control plane + +### The user and directories + +``` +sudo useradd --system --home /var/lib/llmgw --shell /usr/sbin/nologin llmgw +sudo install -d -o llmgw -g llmgw -m 0750 /var/lib/llmgw +sudo install -d -m 0755 /etc/llmgw +sudo chown root:llmgw /etc/llmgw/ca.pem /etc/llmgw/server.pem /etc/llmgw/server-key.pem +sudo chmod 0640 /etc/llmgw/server-key.pem +``` + +### The configuration + +`/etc/llmgw/server.yaml`. This is the smallest file that is not a toy: mutual +TLS on, authentication on, one model, and limits that apply to anyone nobody has +given limits to. + +```yaml +listen: ":8080" +admin_listen: "127.0.0.1:9090" +agent_listen: ":7443" + +agent_tls: + ca: /etc/llmgw/ca.pem + cert: /etc/llmgw/server.pem + key: /etc/llmgw/server-key.pem + +auth: + enabled: true + store: /var/lib/llmgw/llmgw.db + signup_open: true + # Applied to any key no scope gives a limit to. Without these, a project + # created without limits is metered by nothing at all, and the first sign of + # that is the bill or the outage. + default_rpm: 60 + default_tpm: 200000 + +landing: + contact: "platform@example.com" + +# Start with none. Models are added while the cluster runs — see step 7 — +# and anything listed here is declarative: it is in the catalog on every start +# and cannot be removed without editing this file. +models: [] +``` + +`admin_listen` is on loopback on purpose. It serves pprof, which can dump the +process heap, and API keys sit in that heap in plaintext. To see the dashboard +from elsewhere, promote an account and use `/admin` on the client port — do not +move this listener. + +The store file is created on first start; there is no separate migration step. + +### The service + +``` +sudo cp deploy/systemd/llmgw-server.service /etc/systemd/system/ +sudo systemctl daemon-reload +sudo systemctl enable --now llmgw-server +systemctl status llmgw-server +``` + +### Your account + +Accounts are created and approved from a shell on this machine, deliberately: +having a shell here is what makes someone an administrator, and there are no +roles to get subtly wrong. + +``` +sudo -u llmgw llmgw users create you@example.com --password 'change me now' +# prints a user id, usr-xxxxxxxx +sudo -u llmgw llmgw users promote usr-xxxxxxxx +``` + +`promote` is what grants `/admin`. Sign in at `http://:8080/login`, +change the password, and issue yourself a key from the profile page. + +### Check it + +``` +curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health # 200, with an empty body +sudo -u llmgw llmgw ctl status +``` + +`ctl status` will report no nodes and no models, which is correct — nothing has +been added yet. It is worth running now so its output is familiar before you +need it. + +## 4. Preparing a node + +**Do this before onboarding.** A node missing its driver or engine registers +healthy and fails on first placement, and the reason is three logs away. + +### The driver + +``` +sudo apt-get update +sudo apt-get install -y nvidia-driver-550 # or whatever your cards need +sudo reboot +nvidia-smi # must print your GPUs +``` + +`nvidia-smi` must be on `PATH` for the user the agent runs as, which is root. +Containerised driver installs often put it elsewhere; if so, note the path and +set `discovery.nvidia_smi` in the agent configuration rather than moving it. + +A node with no GPU at all is fine and is not an error. It reports zero cards and +can host CPU profiles, which is a good way to try llmgw before committing +hardware to it. + +### InfiniBand, if you have it + +``` +sudo apt-get install -y rdma-core ibverbs-utils infiniband-diags +ibstat # must list your HCAs with State: Active +ls /sys/class/infiniband +``` + +Fabric membership is configuration, not discovery, and there is no way around +that: OpenSM's default subnet prefix is the same on every fabric, so a matching +prefix proves nothing and a wrong guess produces a replica that starts, passes +health checks, and then hangs on its first all-reduce. You name the fabric when +you onboard the node, and two nodes are only in the same fabric if you said so. + +### An engine + +Pick what you will actually serve. For a first cluster, llama.cpp is the least +trouble and runs without a GPU. + +``` +# llama.cpp, from a release build +sudo apt-get install -y libgomp1 # it will not start without this +sudo mkdir -p /opt/llama.cpp +# unpack a llama.cpp release into /opt/llama.cpp +/opt/llama.cpp/llama-server --version +``` + +That `libgomp1` line is not padding. A `llama-server` binary missing it exists, +is executable, and cannot start — and the node reports `ready` throughout. + +For vLLM, a virtualenv on the node: + +``` +sudo apt-get install -y python3-venv +sudo python3 -m venv /opt/venvs/vllm +sudo /opt/venvs/vllm/bin/pip install vllm +/opt/venvs/vllm/bin/vllm --version +``` + +Check the engine still takes the flags llmgw builds for it, which drift between +releases: + +``` +llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm +``` + +### Weights + +Put them where the node can read them. There is no distribution mechanism yet; +`scp`, a shared filesystem, or the engine's own downloader are all valid, and +which you want depends on how many nodes you have. + +``` +sudo install -d -o root -g root /var/lib/llmgw-models +# then place your .gguf or model directory there +``` + +For a first model, something small: a 3B at `q4_k_m` is about 2 GB, runs on +CPU, and answers fast enough to prove the whole path works. + +## 5. Adding the node + +Two halves, because they need different access: building the machine needs the +hypervisor, and issuing its identity needs the certificate authority. Where one +person holds both — the common case — one command does it: + +``` +llmgw-add-node \ + --name llmgw-node01 --ip 10.0.0.141/24 \ + --server https://10.0.0.140:7443 \ + --cores 16 --memory 131072 --disk 200 \ + --fabric ib-a:mlx5_0,mlx5_1 \ + --llamacpp-bin /opt/llama.cpp/llama-server +``` + +**The VM name, the hostname, the node id and the certificate's common name are +all the same string.** They do not have to be, and every time they are not, +somebody spends an afternoon working out which of the four a message refers to. + +For a machine that already exists — bare metal, or a VM you built yourself — skip +provisioning and issue the identity only: + +``` +llmgw-onboard-node \ + --node llmgw-node01 --addr 10.0.0.141 \ + --server https://10.0.0.140:7443 \ + --fabric ib-a:mlx5_0,mlx5_1 \ + --llamacpp-bin /opt/llama.cpp/llama-server +``` + +Onboarding needs SSH from the CA host to the node. It waits for SSH rather than +failing on it, and learns the host key of a machine it has not seen — a new +machine spends its first minute regenerating host keys and restarting sshd. + +It also checks that a configured engine actually runs, and says so loudly when +it does not. That check exists because the failure is otherwise invisible. + +Within a few seconds: + +``` +llmgw ctl nodes +``` + +``` +NODE STATE GPUS MEM CLASSES FABRICS REPLICAS LAST SEEN +llmgw-node01 ready 8/8 free 503G h100-80g ib-a 0 2s +``` + +If it does not appear, the control plane's log says why. A certificate whose +common name does not match the claimed node id is refused by design — that check +is what makes a node's identity something it proves rather than something it +picks. + +Repeat for each node. Nodes in the same fabric must be given the same `--fabric` +name; nodes with no InfiniBand take no `--fabric` at all and are each their own +island, which is fine for single-node models. + +## 6. Setting the node's CPU model, if it is a VM + +Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2 +from the guest. Inference then falls back to scalar code and is not slightly +slower but unusable. Measured on one 8-core VM, same weights, same threads, the +only change being `--cpu host`: + +| Qwen2.5-Coder | kvm64 | host | +| --- | ---: | ---: | +| 1.5B q4_K_M | 5.80 | **39.09** tok/s | +| 3B q4_K_M | 2.90 | **20.51** | +| 7B q4_K_M | 1.33 | **9.88** | + +`llmgw-provision-node` sets `--cpu host` for you. A VM you built yourself +probably did not, and nothing about it looks broken — every number is simply +small. Check with `grep -qw avx2 /proc/cpuinfo` on the node. + +## 7. Adding a model + +Write the definition. This is the same shape a model takes in `server.yaml`, so +an entry moves between the two by being copied: + +```yaml +# qwen-coder-3b.yaml +name: qwen2.5-coder-3b-instruct +aliases: [coder] +source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf +routing_profile: chat +# What a client may use in one request. Worth stating: llama.cpp divides +# --ctx-size across its parallel slots, so a pool of 65536 over two slots is +# 32768 each, and nothing else will tell you that. +context_length: 32768 +max_output: 4096 +scaling: { min: 1, max: 1 } +profiles: + - name: cpu + engine: llamacpp + weight: 1 + priority: 100 + startup_timeout: 5m + min_memory_gb: 8 + args: ["--ctx-size", "65536"] +``` + +``` +llmgw ctl model add -f qwen-coder-3b.yaml +llmgw ctl models +``` + +``` +MODEL SERVING REPLICAS STATUS ALIASES +qwen2.5-coder-3b-instruct yes 1/1 ready coder +``` + +No restart, and it survives one: models added this way are stored, while +anything in `server.yaml` is declarative and returns on every start. + +For a GPU model, name the class and the width instead: + +```yaml +profiles: + - name: tp2 + engine: vllm + gpu_class: h100-80g # as `llmgw ctl nodes` reports it + tp: 2 + priority: 100 +``` + +If it does not place, ask: + +``` +llmgw ctl explain-placement qwen2.5-coder-3b-instruct +``` + +which names every candidate and why each was rejected, rather than leaving you +to infer it. + +## 8. The first request + +``` +curl http://:8080/v1/chat/completions \ + -H "Authorization: Bearer $LLMGW_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "coder", "messages": [{"role":"user","content":"hello"}]}' +``` + +Or open `/chat` in a browser and use it without a key at all — the page carries +your session, and the request still goes through admission, rate limits, budget +and accounting exactly as an API call does. + +## When something does not work + +**The node is `ready` but every replica fails.** Almost always a missing engine +or weights. `llmgw ctl replicas` shows the shard's error and the engine's last +output, which is the actual diagnosis. + +**A model will not place.** `llmgw ctl explain-placement ` names each +node and the reason: wrong GPU class, not enough free cards, too little memory, +no fabric, a label it does not carry. + +**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the +quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of +two. `llmgw ctl status` shows how full the cluster is. + +**Tool calls come back as text.** The model made the call and the engine did not +parse it. `llmgw ctl tool-check --endpoint http://:` distinguishes +that from a model ignoring its tools, which has a different fix. + +**Requests fail at exactly sixty seconds.** `timeouts.response_header` is a +liveness check for streaming and a total generation budget for non-streaming +requests. On a slow model, set `timeouts.generation` higher. + +## What to read next + +- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding + deliberately does not do +- `docs/onboarding.md` — giving people accounts and keys +- `llmgw-plan.md` — why the system is shaped the way it is + +## Known gaps, so you find them here rather than at 2am + +- **Weights are yours to distribute.** No cache, no fetch, no progress. +- **Engines are yours to install.** Onboarding verifies one runs; it installs + nothing. +- **One control plane.** No HA yet; it is a single process with a SQLite file. + Back up `/var/lib/llmgw/llmgw.db`. +- **Multi-node replicas have not been run on real InfiniBand.** The scheduling, + the fabric invariant and the rendezvous are exercised against simulated + hardware only. Single-node placement on real GPUs is the well-trodden path; + if you are the first to run a TP group across real cards, expect to find + something.