Add a node with one command: llmgw node add, with a terminal screen

Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:

    sudo llmgw node add ubuntu@10.0.0.21

The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.

The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.

Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.

llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 19:33:47 +02:00
1 parent c8da06988f
commit 2a16c26575
17 files changed
+2056 -120

No files matched your search

+24 -42
View File
@@ -4,24 +4,16 @@ Written for whoever runs the cluster.
## The short version
Where one person holds both the hypervisor and the certificate authority —
which is the common case — one command does the whole thing in about
45 seconds:
On the control plane:
```
llmgw-add-node \
--name llmgw-node05 --ip 10.0.0.145/24 \
--server https://10.0.0.140:7443 \
--cores 8 --memory 32768 --disk 60 \
--llamacpp-bin /opt/llama.cpp/llama-server
sudo llmgw node add ubuntu@10.0.0.145
```
The VMID defaults to the last octet of the address, so `qm list` and
`llmgw ctl nodes` stay in the same order. `--provision-only` and
`--onboard-only` split it back into the two halves below when the two kinds
of access belong to different people.
The rest of this page is what that command does, and what it still does not.
That is all a machine that already exists needs — bare metal, or a VM you
built. [install.md](install.md#5-adding-the-node) says what it does. On
Proxmox, the machine itself can be built first by the script below, on the
hypervisor.
## The two halves
@@ -54,41 +46,31 @@ their working tree.
## 2. The identity
On the host that holds the certificate authority — this is the part that cannot
be delegated, because it issues the credential by which the control plane
believes the node is who it says it is:
On the control plane, which holds the certificate authority — this is the
part that cannot be delegated, because it issues the credential by which the
control plane believes the node is who it says it is:
```
llmgw-onboard-node \
--node llmgw-node04 \
--addr 10.0.0.144 \
--server https://10.0.0.140:7443 \
--fabric ib-b
sudo llmgw node add ilker@10.0.0.144 --fabric ib-b:mlx5_0,mlx5_1
```
It issues a client certificate whose common name is the node id, installs it
along with the CA root, trusts that root system-wide so other tooling on the
machine stops treating llmgw's certificates as strangers, installs the binary
and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
a few seconds.
It checks the machine with `llmgw doctor`, issues a client certificate whose
common name is the node id (the hostname), installs it along with the CA root,
trusts that root system-wide, installs the binary and the unit, starts the
agent, and waits for the node to join.
If it does not, the control plane's log says why. A certificate whose common
name does not match the claimed node id is refused by design — that check is
what makes a node's identity something it proves rather than something it picks.
If it does not join, the agent's log on the node says why. A certificate whose
common name does not match the claimed node id is refused by design — that
check is what makes a node's identity something it proves rather than
something it picks.
**It needs ssh from the CA host to the node.** Provisioning copies that host's
`authorized_keys` into the new machine, so a node built by the script above
already satisfies it; an older machine may need the key added.
**It needs SSH from the control plane to the node.** Provisioning copies the
hypervisor's `authorized_keys` into the new machine; with `ssh -A` to the
control plane and `sudo --preserve-env=SSH_AUTH_SOCK`, the same key that
reaches the hypervisor reaches the node.
It waits for ssh rather than failing on it, and learns the host key of a machine
it has not seen (`accept-new`, which still refuses a *changed* key — the risk
being guarded against is a substituted machine, not a new one). Both are needed
to run the halves back to back: for the first minute of its life a new machine
is still regenerating host keys and restarting sshd, so it refuses connections
it accepted a moment earlier.
It also checks that a configured engine actually runs, which is cheap here and
expensive later — see below.
`deploy/bootstrap/onboard-node.sh` is the script this replaced, for a machine
holding an authority but not a current llmgw.
## The CPU model matters, and its default is wrong