Ilker ManapandClaude Opus 5.5
2a16c26575
Adding a node was deploy/bootstrap/onboard-node.sh, which is not installed
anywhere and wanted the node's name, the control plane's address and a
dozen flags. It is now in the binary, run on the control plane:
sudo llmgw node add ubuntu@10.0.0.21
The node id is the machine's hostname, and the address its agent dials is
this machine's on the route to it, checked against the server certificate.
It installs llmgw there (the release for its architecture, fetched and
checked by the node; uname rather than dpkg, so RHEL works), runs doctor
there and stops on a problem before changing anything, issues and installs
the certificate, writes agent.yaml and the unit, starts the agent, and waits
for it to join and install llama.cpp. The authority is read first, so a
missing one is said before the node is touched.
The steps are events. In a terminal they draw a checklist (bubbletea) that
asks what a script cannot: which fabric each group of active adapters is on,
grouped by the subnet manager or RoCE network they report, and whether to go
on past a problem. Elsewhere, one line per step.
Found by running it on an empty VM: ssh's "Permanently added" warning on
stderr was read as the machine's hostname, so standard error is now kept
apart; and a node added again while cordoned never counted as joined, so
joining now means a heartbeat since the restart.
llmgw node remove forgets a lost node, which otherwise stays listed until
the control plane restarts. ctl nodes no longer lists vLLM as skipped on
every CPU node. The quick install, install and onboarding docs use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
210 lines
8.7 KiB
Markdown
210 lines
8.7 KiB
Markdown
# Adding a node
|
|
|
|
Written for whoever runs the cluster.
|
|
|
|
## The short version
|
|
|
|
On the control plane:
|
|
|
|
```
|
|
sudo llmgw node add ubuntu@10.0.0.145
|
|
```
|
|
|
|
That is all a machine that already exists needs — bare metal, or a VM you
|
|
built. [install.md](install.md#5-adding-the-node) says what it does. On
|
|
Proxmox, the machine itself can be built first by the script below, on the
|
|
hypervisor.
|
|
|
|
## The two halves
|
|
|
|
Two scripts, because the two halves of onboarding are done by different people
|
|
with different access: one needs the hypervisor, the other needs the
|
|
certificate authority.
|
|
|
|
## 1. The machine
|
|
|
|
On the Proxmox host:
|
|
|
|
```
|
|
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
|
|
```
|
|
|
|
Name it after the node id. **The VM name, the hostname, the node id and the
|
|
certificate's common name are all the same string.** They do not have to be, and
|
|
every time they are not, somebody spends an afternoon working out which of the
|
|
four a message is talking about.
|
|
|
|
It imports a cloud image and lets cloud-init do the rest, which is why a node
|
|
takes a minute rather than an afternoon. It refuses to touch an existing VMID:
|
|
adopting a machine is not a decision a provisioning script should make.
|
|
|
|
Releases carry static `linux/amd64` and `linux/arm64` binaries. Download one and
|
|
pass it with `--binary` rather than copying whatever happens to be on the
|
|
machine you are running onboarding from — otherwise whoever runs the script
|
|
quietly becomes the build server, and the fleet ends up running whatever was in
|
|
their working tree.
|
|
|
|
## 2. The identity
|
|
|
|
On the control plane, which holds the certificate authority — this is the
|
|
part that cannot be delegated, because it issues the credential by which the
|
|
control plane believes the node is who it says it is:
|
|
|
|
```
|
|
sudo llmgw node add ilker@10.0.0.144 --fabric ib-b:mlx5_0,mlx5_1
|
|
```
|
|
|
|
It checks the machine with `llmgw doctor`, issues a client certificate whose
|
|
common name is the node id (the hostname), installs it along with the CA root,
|
|
trusts that root system-wide, installs the binary and the unit, starts the
|
|
agent, and waits for the node to join.
|
|
|
|
If it does not join, the agent's log on the node says why. A certificate whose
|
|
common name does not match the claimed node id is refused by design — that
|
|
check is what makes a node's identity something it proves rather than
|
|
something it picks.
|
|
|
|
**It needs SSH from the control plane to the node.** Provisioning copies the
|
|
hypervisor's `authorized_keys` into the new machine; with `ssh -A` to the
|
|
control plane and `sudo --preserve-env=SSH_AUTH_SOCK`, the same key that
|
|
reaches the hypervisor reaches the node.
|
|
|
|
`deploy/bootstrap/onboard-node.sh` is the script this replaced, for a machine
|
|
holding an authority but not a current llmgw.
|
|
|
|
## The CPU model matters, and its default is wrong
|
|
|
|
Nodes are provisioned with `--cpu host`. Proxmox's default, `kvm64`, hides
|
|
AVX, AVX2, FMA and even SSE4.2 from the guest, and llama.cpp then loads its
|
|
scalar backend instead of the one built for the host's instruction set.
|
|
Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
|
|
threads, the only change being the CPU model:
|
|
|
|
| Qwen2.5-Coder | kvm64 tok/s | host tok/s |
|
|
| -------------- | ----------: | ---------: |
|
|
| 1.5B q4_K_M | 5.80 | 39.09 |
|
|
| 3B q4_K_M | 2.90 | 20.51 |
|
|
| 7B q4_K_M | 1.33 | 9.88 |
|
|
|
|
Prompt processing is worse still: 7.6 tokens a second against 143.8, a factor
|
|
of nineteen.
|
|
|
|
It is worth calling out because nothing looks broken. The node registers, the
|
|
engine starts, replies are correct — every number is simply small, and the
|
|
obvious conclusion is that CPU inference is hopeless rather than that the VM
|
|
cannot multiply efficiently.
|
|
|
|
The trade is live migration: a VM pinned to the host CPU cannot move to a
|
|
machine with a different one. For a node whose job is arithmetic that is the
|
|
right way round, and `--cpu` overrides it where it is not.
|
|
|
|
## Engines
|
|
|
|
The script installs none, and does not need to: once the agent is connected,
|
|
the control plane tells it which engines to have and it installs them itself
|
|
— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
|
|
on, SGLang where the control plane names a version. Each is checked to run
|
|
before the node offers it, so a node never advertises an engine that cannot
|
|
start. [install.md](install.md#engines) says what vLLM needs of the node.
|
|
|
|
`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
|
|
installed yourself. For the agent to use yours rather than install its own,
|
|
also set that engine's version to `""` under `engines` in `server.yaml`.
|
|
|
|
Weights are the other thing a node needs. llmgw fetches them for a model whose
|
|
`source` is a URL, one copy per node, which covers the simple case; anything
|
|
larger — a shared filesystem, a cache — stays the operator's arrangement, and
|
|
llmgw is content to find the files already there.
|
|
|
|
The `sim` engine needs neither, which is what makes it useful for exercising
|
|
everything else.
|
|
|
|
## Checking an engine after an upgrade
|
|
|
|
Engines rename and remove flags between versions, and when that happens the
|
|
failure arrives minutes into a model load, in the engine's own error message, on
|
|
a node nobody was watching.
|
|
|
|
```
|
|
llmgw ctl engine-check vllm
|
|
llmgw ctl engine-check llamacpp --help-output "$(llama-server --help 2>&1)"
|
|
```
|
|
|
|
It builds the widest command llmgw would ever produce — a multi-node one, since
|
|
a flag that only appears there is exactly the one nobody notices has gone — and
|
|
checks every flag against the engine's own `--help`. No GPU required: a CLI
|
|
prints its help on any machine.
|
|
|
|
Some programs do not give up their whole help for `--help`. vLLM prints a
|
|
summary of section names and expects `--help=all`; checking against the summary
|
|
reports every flag as missing, confidently and wrongly. `engine-check` knows
|
|
that one. If you add an engine, check that what you get back is the full list.
|
|
|
|
vLLM also refuses to do anything at all on a machine with no accelerator,
|
|
including print its help, unless `VLLM_TARGET_DEVICE` says what to assume.
|
|
`engine-check` sets it.
|
|
|
|
Give it the **whole** help. Truncating llama.cpp's 725 lines to 400 reports five
|
|
flags as missing that the running cluster uses; the command warns when the text
|
|
it was given does not document `--help` itself, which is the cheapest sign it
|
|
was cut short.
|
|
|
|
## Fabrics
|
|
|
|
`--fabric` is how a node says which InfiniBand fabric it is attached to. The
|
|
name is checked against what the adapters report — every port on one fabric
|
|
sees the same subnet managers — so two independent fabrics given one name are
|
|
placed as two, and the cluster page says so; give each fabric its own name.
|
|
The agent installs `infiniband-diags`, which that check needs, if it is missing.
|
|
|
|
Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
|
|
subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
|
|
and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
|
|
rest.
|
|
|
|
Omit it for a node with no fabric. Such a node is its own island: it can host
|
|
single-node replicas and will never be given a multi-node one.
|
|
|
|
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
|
|
explicitly. An adapter must never appear under two fabrics — the agent refuses
|
|
that configuration: a replica given adapters on two fabrics starts, passes its
|
|
health checks, and hangs on its first all-reduce.
|
|
|
|
## Where the driver is
|
|
|
|
```
|
|
--nvidia-smi /opt/nvidia/bin/nvidia-smi
|
|
--ib-root /sys/class/infiniband
|
|
```
|
|
|
|
Both default to the usual places. Containerized driver installs routinely put
|
|
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
|
|
GPUs — which looks exactly like a node that has none.
|
|
|
|
## Removing a node
|
|
|
|
```
|
|
llmgw ctl drain llmgw-node04 # move its replicas off, one at a time
|
|
llmgw ctl nodes # wait until it holds none
|
|
```
|
|
|
|
Then stop the agent. Draining brings each replacement up before stopping the
|
|
original, which is the difference between a rolling move and an outage.
|
|
|
|
The node's certificate stays valid until it expires. There is no revocation list
|
|
— if a machine is decommissioned rather than retired, reissue the authority or
|
|
accept that a certificate naming a node that no longer exists can still connect
|
|
and claim to be it.
|
|
|
|
## Testing without hardware
|
|
|
|
Every part of this works on machines with no GPUs and no InfiniBand. Point
|
|
`--nvidia-smi` at a script that prints the CSV discovery asks for, point
|
|
`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.
|
|
|
|
The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
|
|
does not serve until every worker has joined, and workers serve nothing. That
|
|
exercises the multi-node launch sequence, the fabric constraint, and failure and
|
|
rebuild — which is most of what goes wrong — without exercising NCCL, which is
|
|
the rest.
|