Add a node with one command: llmgw node add, with a terminal screen

Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:

    sudo llmgw node add ubuntu@10.0.0.21

The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.

The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.

Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.

llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 19:33:47 +02:00
1 parent c8da06988f
commit 2a16c26575
17 files changed
+2056 -120

No files matched your search

+32 -41
View File
@@ -11,7 +11,7 @@ explains every choice.
| **One control plane machine** | Linux, x86-64 or ARM64. Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 580 or newer and a Turing GPU or newer (not V100). |
| **root** | on the control plane, and `sudo` without a password on every node. |
| **SSH** | from root on the control plane to each node. |
| **SSH** | from the control plane to each node, as a user with passwordless `sudo` there. |
| **Network** | nodes reach the control plane on 7443 and 8081; users reach it on 8080. The machines need internet access to download engines and models. |
You do not install llama.cpp, vLLM, Python or CUDA. The nodes do that
@@ -60,59 +60,49 @@ the account page.
## 3. Add the GPU nodes
**On each node**, check it first. Set `VERSION` and `U` as in step 1, then:
On the control plane, for each node:
```
curl -fsSLO $U/llmgw-$VERSION-linux-amd64
sudo install -m0755 llmgw-$VERSION-linux-amd64 /usr/local/bin/llmgw
sudo llmgw doctor
sudo llmgw node add ubuntu@10.0.0.21
```
Fix every `FAIL` line before going on. The fix is printed under each line.
Replace `ubuntu` with the user you log in to the node as, and `10.0.0.21` with
its address. That user needs `sudo` without a password on the node.
**On the control plane**, get the onboarding script once:
It logs in over SSH and, while you watch:
1. installs llmgw on the node, the release for its processor, checksum checked;
2. runs `llmgw doctor` there, and stops if the node has a problem. Each problem
comes with the command that fixes it. Fix it and run the same command again;
3. gives the node its certificate and configuration, and starts its agent;
4. waits for the node to join and install llama.cpp.
The node's name in the cluster is its hostname. vLLM installs by itself after
that, in a few more minutes on a GPU node (about 8 GB). The cluster page
(`/admin`) shows its progress, or why the node cannot run it.
**InfiniBand.** When the node's adapters are up, `node add` shows them grouped
by the fabric they are on and asks for each fabric's name. Give every node on
the same fabric the same name. In a script, give it instead:
```
curl -fsSLO https://gitea.mynodes.xyz/ilker/llmgw/raw/tag/$VERSION/deploy/bootstrap/onboard-node.sh
chmod +x onboard-node.sh
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
```
Check that root can reach the node over SSH. This must print `ok` without asking
for a password:
**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
own keys instead, log in to the control plane with `ssh -A` and keep your
agent through sudo:
```
sudo ssh ubuntu@10.0.0.21 'sudo true && echo ok'
sudo --preserve-env=SSH_AUTH_SOCK llmgw node add ubuntu@10.0.0.21
```
Then add the node. `--node` is its name in the cluster; use its hostname.
**Check:** `llmgw ctl nodes` lists the node. Running `node add` again on the
same node is safe: it renews its certificate and configuration, and keeps what
runs there.
```
sudo ./onboard-node.sh --node gpu01 --addr 10.0.0.21 \
--server https://10.0.0.10:7443 --ssh-user ubuntu
```
Replace `10.0.0.21` with the node's address, `10.0.0.10` with the control
plane's, and `ubuntu` with the user you SSH in as.
If the node has InfiniBand, name the fabric and its adapters, and give every
node on the same fabric the same name:
```
--fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
```
**Check:**
```
llmgw ctl nodes
```
The node appears within seconds. It then installs its engines by itself:
llama.cpp in about a minute, vLLM in a few more (about 8 GB). The cluster page
(`/admin`) shows the progress under each node. If a node cannot run vLLM, it
says why on the same page.
Repeat for each node.
A machine retired for good is removed with `llmgw node remove <name>` once its
agent is stopped.
## 4. Add a model
@@ -174,7 +164,8 @@ through the proxy. There is an nginx configuration too:
| What you see | What to do |
|---|---|
| A node does not appear in `llmgw ctl nodes` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443, or the control plane's address is not in its certificate: run `server init` again with `--host`. |
| `node add` stops at `join` | `sudo journalctl -u llmgw-agent -n 50` on the node. Usually it cannot reach port 7443 on the control plane. |
| `node add` says the address is not in the certificate | Run `sudo llmgw server init --host <that address>`, then `node add` again. |
| A node says it cannot run vLLM | The reason is on `/admin`: the driver is older than 580, the GPU is older than Turing, or there is less than 25 GB free. |
| A model shows `not placed` | `llmgw ctl explain-placement <model>` names each node and why it was not used. |
| A reply stops after 60 seconds behind nginx | Use the nginx configuration from the release. Its timeouts are set for slow models. |