From c8da06988f3493687540aa4deb4e98841da59072 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Tue, 29 Sep 2026 18:13:05 +0200 Subject: [PATCH] Add a quick install guide: six steps from empty servers to a model install.md explains every choice and is long for someone who wants the commands. quick-install.md has only those, and what to check after each: the binary, server init, onboarding GPU nodes (with doctor first and the script fetched from the release tag, since it is not installed anywhere), a model from the mirror's explorer, keys and editors, and Caddy. Every button and path it names was checked against the pages. quickstart.md is now said to be the from-source, one-workstation path it is; README and the about page point at the new guide. Co-Authored-By: Claude Opus 5.5 (1M context) --- README.md | 8 +- docs/install.md | 3 +- docs/quick-install.md | 187 ++++++++++++++++++++++++++++++ docs/quickstart.md | 6 +- internal/web/templates/about.html | 2 +- 5 files changed, 198 insertions(+), 8 deletions(-) create mode 100644 docs/quick-install.md diff --git a/README.md b/README.md index 108519b6..9794ce1e 100644 --- a/README.md +++ b/README.md @@ -58,9 +58,11 @@ them. On such a machine, run `llmgw doctor --json` first and keep the report; ## Documentation -- [docs/install.md](docs/install.md) — installing from nothing: control plane, - nodes, engines, weights, first request -- [docs/quickstart.md](docs/quickstart.md) — the short path +- [docs/quick-install.md](docs/quick-install.md) — installing on servers, in + six steps: start here +- [docs/install.md](docs/install.md) — the same from nothing, with every option + explained: control plane, nodes, engines, weights, first request +- [docs/quickstart.md](docs/quickstart.md) — one workstation, from source - [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with `llmgw doctor`, and what to look at when a replica spanning nodes will not run diff --git a/docs/install.md b/docs/install.md index 566ba427..6f661be6 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,7 +1,8 @@ # Installing llmgw from nothing Written for whoever is standing up the cluster: you have servers, you have root -on them, and nothing is installed yet. By the end there is a control plane, a +on them, and nothing is installed yet. For only the commands, read +[quick-install.md](quick-install.md) first; this page explains them. By the end there is a control plane, a node in the fleet, and a model answering requests. It assumes Debian or Ubuntu and adjusts nothing for other distributions beyond diff --git a/docs/quick-install.md b/docs/quick-install.md new file mode 100644 index 00000000..ca1183b2 --- /dev/null +++ b/docs/quick-install.md @@ -0,0 +1,187 @@ +# Quick install + +From empty servers to a model answering requests, in six steps. This page +has only the commands and what to check after each; [install.md](install.md) +explains every choice. + +## What you need + +| | | +|---|---| +| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. | +| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). | +| **root** | on the control plane, and `sudo` without a password on every node. | +| **SSH** | from root on the control plane to each node. | +| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. | + +You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that +themselves once they join. + +## 1. Put llmgw on the control plane + +``` +VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \ + | grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4) +U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION +curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM +curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing +sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw +llmgw --version +``` + +`sha256sum` must print `OK`. If it does not, the download is damaged; do not +install it. + +## 2. Set up the control plane + +``` +sudo llmgw server init --admin you@example.com +``` + +This one command sets up everything on this machine, and takes a few minutes: + +- the service, its certificates and its configuration; +- your administrator account. **The password is printed once. Copy it now.** +- a model mirror, which downloads model files once for the whole cluster; +- this machine as a first node, and a small model (Qwen2.5 0.5B) on it. + +If nodes will reach this machine by a name or address it does not know itself +(a DNS name, a NAT address), add `--host` for each: + +``` +sudo llmgw server init --admin you@example.com --host llm.example.com +``` + +It is safe to run again. It keeps what exists and finishes what is missing. + +**Check:** open `http://:8080/chat`, sign in with the printed +password, and ask something. The small model answers. Change the password on +the account page. + +## 3. Add the GPU nodes + +**On each node**, check it first. Set `VERSION` and `U` as in step 1, then: + +``` +curl -fsSLO $U/llmgw-$VERSION-linux-amd64 +sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw +sudo llmgw doctor +``` + +Fix every `FAIL` line before going on. The fix is printed under each line. + +**On the control plane**, get the onboarding script once: + +``` +curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh +chmod +x onboard-node.sh +``` + +Check that root can reach the node over SSH. This must print `ok` without asking +for a password: + +``` +sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok' +``` + +Then add the node. `--node` is its name in the cluster; use its hostname. + +``` +sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \ + --server https://10.0.0.10:7443 --ssh-user ubuntu +``` + +Replace `10.0.0.21` with the node's address, `10.0.0.10` with the control +plane's, and `ubuntu` with the user you SSH in as. + +If the node has InfiniBand, name the fabric and its adapters, and give every +node on the same fabric the same name: + +``` + --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3 +``` + +**Check:** + +``` +llmgw ctl nodes +``` + +The node appears within seconds. It then installs its engines by itself: +llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page +(`/admin`) shows the progress under each node. If a node cannot run vLLM, it +says why on the same page. + +Repeat for each node. + +## 4. Add a model + +Open `/admin/mirror` and choose **Explore models**. It lists what is popular on +Hugging Face, showing only what your nodes can run. + +1. Click a model to read its card. +2. Pick one of its files, usually a `q4_k_m` for a first try, and click + **Fetch**. It is downloaded into the mirror once, for every node. +3. When the download finishes, click **serve** on its row in the table above. +4. In the form, choose how it runs: **always on**, or **on demand**, which + loads it on the first request and stops it after 15 minutes unused. Click + **Add to the catalog**. + +The model is placed on a node that fits it, and its status on `/admin` turns +`ready`. + +**Check:** the model appears in the list on `/chat`. Ask it something. + +## 5. Give people access + +**A key for a program:** each person makes their own on the account page +(`/profile`). The key is shown once. + +``` +curl http://:8080/v1/chat/completions \ + -H "Authorization: Bearer $LLMGW_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "", "messages": [{"role": "user", "content": "hello"}]}' +``` + +Any OpenAI client works. Only the base URL changes: +`http://:8080/v1`. + +**An editor:** the account page connects Continue, Codex or Cline. Pick the +tool and a model, and it gives a one-line command that writes the +configuration. + +**Other people's accounts:** they sign up at `/signup`, and you approve them on +the people page (`/admin/users`). + +## 6. Put it behind HTTPS + +Port 8080 is plain HTTP. Before users outside a trusted network use it, put a +reverse proxy in front with a real certificate. With Caddy, which gets the +certificate by itself: + +``` +sudo apt-get install -y caddy # Debian and Ubuntu +curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile +sudo systemctl reload caddy +``` + +Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go +through the proxy. There is an nginx configuration too: +[deploy/proxy/](../deploy/proxy/). + +## When something does not work + +| What you see | What to do | +|---|---| +| A node does not appear in `llmgw ctl nodes` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run `server init` again with `--host`. | +| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. | +| A model shows `not placed` | `llmgw ctl explain-placement ` names each node and why it was not used. | +| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. | +| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. | + +## Next + +- [install.md](install.md): the same steps with every option explained. +- [fabrics.md](fabrics.md): InfiniBand and NVLink. +- [onboarding.md](onboarding.md): accounts, projects, budgets and keys. diff --git a/docs/quickstart.md b/docs/quickstart.md index 5440b55b..5a16f7a8 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,8 +1,8 @@ # Quickstart: one machine, one model, ten minutes -Written for someone who wants to see llmgw answer a request on a single -workstation before reading anything else. No systemd, no `sudo`. For a real -cluster, see [install.md](install.md). +Written for a developer who wants to see llmgw answer a request on a single +workstation, built from source, with no systemd and no `sudo`. **To install +llmgw on servers, read [quick-install.md](quick-install.md) instead.** What you end up with: an OpenAI-compatible endpoint on `localhost:8080` in front of a real model served by llama.cpp. diff --git a/internal/web/templates/about.html b/internal/web/templates/about.html index 92849dbe..2b5713c3 100644 --- a/internal/web/templates/about.html +++ b/internal/web/templates/about.html @@ -203,7 +203,7 @@ footer a{color:var(--muted)} fabric, and a model split across machines never straddles two.

Download{{with .Release}} {{.}}{{end}} - Install guide + Install guide
$ sudo llmgw server init --admin you@example.com