Ilker ManapandClaude Opus 5.5
2a16c26575
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
179 lines
7.1 KiB
Markdown
179 lines
7.1 KiB
Markdown
# Quick install
|
|
|
|
From empty servers to a model answering requests, in six steps. This page
|
|
has only the commands and what to check after each; [install.md](install.md)
|
|
explains every choice.
|
|
|
|
## What you need
|
|
|
|
| | |
|
|
|---|---|
|
|
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
|
|
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
|
|
| **root** | on the control plane, and `sudo` without a password on every node. |
|
|
| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
|
|
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
|
|
|
|
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
|
|
themselves once they join.
|
|
|
|
## 1. Put llmgw on the control plane
|
|
|
|
```
|
|
VERSION=$(curl -fsSL https://gitea.mynodes.xyz/api/v1/repos/ilker/llmgw/releases/latest \
|
|
| grep -o '"tag_name":"[^"]*"' | cut -d'"' -f4)
|
|
U=https://gitea.mynodes.xyz/ilker/llmgw/releases/download/$VERSION
|
|
curl -fsSLO $U/llmgw-$VERSION-linux-amd64 # -linux-arm64 on ARM
|
|
curl -fsSL $U/checksums.txt | sha256sum -c --ignore-missing
|
|
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
|
|
llmgw --version
|
|
```
|
|
|
|
`sha256sum` must print `OK`. If it does not, the download is damaged; do not
|
|
install it.
|
|
|
|
## 2. Set up the control plane
|
|
|
|
```
|
|
sudo llmgw server init --admin you@example.com
|
|
```
|
|
|
|
This one command sets up everything on this machine, and takes a few minutes:
|
|
|
|
- the service, its certificates and its configuration;
|
|
- your administrator account. **The password is printed once. Copy it now.**
|
|
- a model mirror, which downloads model files once for the whole cluster;
|
|
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
|
|
|
|
If nodes will reach this machine by a name or address it does not know itself
|
|
(a DNS name, a NAT address), add `--host` for each:
|
|
|
|
```
|
|
sudo llmgw server init --admin you@example.com --host llm.example.com
|
|
```
|
|
|
|
It is safe to run again. It keeps what exists and finishes what is missing.
|
|
|
|
**Check:** open `http://<control-plane>:8080/chat`, sign in with the printed
|
|
password, and ask something. The small model answers. Change the password on
|
|
the account page.
|
|
|
|
## 3. Add the GPU nodes
|
|
|
|
On the control plane, for each node:
|
|
|
|
```
|
|
sudo llmgw node add ubuntu@10.0.0.21
|
|
```
|
|
|
|
Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
|
|
its address. That user needs `sudo` without a password on the node.
|
|
|
|
It logs in over SSH and, while you watch:
|
|
|
|
1. installs llmgw on the node, the release for its processor, checksum checked;
|
|
2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
|
|
comes with the command that fixes it. Fix it and run the same command again;
|
|
3. gives the node its certificate and configuration, and starts its agent;
|
|
4. waits for the node to join and install llama.cpp.
|
|
|
|
The node's name in the cluster is its hostname. vLLM installs by itself after
|
|
that, in a few more minutes on a GPU node (about 8 GB). The cluster page
|
|
(`/admin`) shows its progress, or why the node cannot run it.
|
|
|
|
**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
|
|
by the fabric they are on and asks for each fabric's name. Give every node on
|
|
the same fabric the same name. In a script, give it instead:
|
|
|
|
```
|
|
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
|
```
|
|
|
|
**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
|
|
own keys instead, log in to the control plane with `ssh -A` and keep your
|
|
agent through sudo:
|
|
|
|
```
|
|
sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
|
|
```
|
|
|
|
**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
|
|
same node is safe: it renews its certificate and configuration, and keeps what
|
|
runs there.
|
|
|
|
A machine retired for good is removed with `llmgw node remove <name>` once its
|
|
agent is stopped.
|
|
|
|
## 4. Add a model
|
|
|
|
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
|
Hugging Face, showing only what your nodes can run.
|
|
|
|
1. Click a model to read its card.
|
|
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
|
**Fetch**. It is downloaded into the mirror once, for every node.
|
|
3. When the download finishes, click **serve** on its row in the table above.
|
|
4. In the form, choose how it runs: **always on**, or **on demand**, which
|
|
loads it on the first request and stops it after 15 minutes unused. Click
|
|
**Add to the catalog**.
|
|
|
|
The model is placed on a node that fits it, and its status on `/admin` turns
|
|
`ready`.
|
|
|
|
**Check:** the model appears in the list on `/chat`. Ask it something.
|
|
|
|
## 5. Give people access
|
|
|
|
**A key for a program:** each person makes their own on the account page
|
|
(`/profile`). The key is shown once.
|
|
|
|
```
|
|
curl http://<control-plane>:8080/v1/chat/completions \
|
|
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "hello"}]}'
|
|
```
|
|
|
|
Any OpenAI client works. Only the base URL changes:
|
|
`http://<control-plane>:8080/v1`.
|
|
|
|
**An editor:** the account page connects Continue, Codex or Cline. Pick the
|
|
tool and a model, and it gives a one-line command that writes the
|
|
configuration.
|
|
|
|
**Other people's accounts:** they sign up at `/signup`, and you approve them on
|
|
the people page (`/admin/users`).
|
|
|
|
## 6. Put it behind HTTPS
|
|
|
|
Port 8080 is plain HTTP. Before users outside a trusted network use it, put a
|
|
reverse proxy in front with a real certificate. With Caddy, which gets the
|
|
certificate by itself:
|
|
|
|
```
|
|
sudo apt-get install -y caddy # Debian and Ubuntu
|
|
curl -fsSL $U/Caddyfile | sed 's/llm.example.com/YOUR.DOMAIN/' | sudo tee /etc/caddy/Caddyfile
|
|
sudo systemctl reload caddy
|
|
```
|
|
|
|
Keep 7443 (nodes) and 8081 (mirror) on the internal network. They never go
|
|
through the proxy. There is an nginx configuration too:
|
|
[deploy/proxy/](../deploy/proxy/).
|
|
|
|
## When something does not work
|
|
|
|
| What you see | What to do |
|
|
|---|---|
|
|
| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
|
|
| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
|
|
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
|
|
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
|
|
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |
|
|
| Anything else about a node | `sudo llmgw doctor` on it. `--json` writes everything it read, to send to whoever helps you. |
|
|
|
|
## Next
|
|
|
|
- [install.md](install.md): the same steps with every option explained.
|
|
- [fabrics.md](fabrics.md): InfiniBand and NVLink.
|
|
- [onboarding.md](onboarding.md): accounts, projects, budgets and keys.
|