llmgw/docs/node-onboarding.md

# Adding a node

Written for whoever runs the cluster.

## The short version

Where one person holds both the hypervisor and the certificate authority —
which is the common case — one command does the whole thing in about
45 seconds:

```
llmgw-add-node \
  --name llmgw-node05 --ip 10.0.0.145/24 \
  --server https://10.0.0.140:7443 \
  --cores 8 --memory 32768 --disk 60 \
  --llamacpp-bin /opt/llama.cpp/llama-server
```

The VMID defaults to the last octet of the address, so `qm list` and
`llmgw ctl nodes` stay in the same order. `--provision-only` and
`--onboard-only` split it back into the two halves below when the two kinds
of access belong to different people.

The rest of this page is what that command does, and what it still does not.

## The two halves

Two scripts, because the two halves of onboarding are done by different people
with different access: one needs the hypervisor, the other needs the
certificate authority.

## 1. The machine

On the Proxmox host:

```
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
```

Name it after the node id. **The VM name, the hostname, the node id and the
certificate's common name are all the same string.** They do not have to be, and
every time they are not, somebody spends an afternoon working out which of the
four a message is talking about.

It imports a cloud image and lets cloud-init do the rest, which is why a node
takes a minute rather than an afternoon. It refuses to touch an existing VMID:
adopting a machine is not a decision a provisioning script should make.

Releases carry static `linux/amd64` and `linux/arm64` binaries. Download one and
pass it with `--binary` rather than copying whatever happens to be on the
machine you are running onboarding from — otherwise whoever runs the script
quietly becomes the build server, and the fleet ends up running whatever was in
their working tree.

## 2. The identity

On the host that holds the certificate authority — this is the part that cannot
be delegated, because it issues the credential by which the control plane
believes the node is who it says it is:

```
llmgw-onboard-node \
  --node llmgw-node04 \
  --addr 10.0.0.144 \
  --server https://10.0.0.140:7443 \
  --fabric ib-b
```

It issues a client certificate whose common name is the node id, installs it
along with the CA root, trusts that root system-wide so other tooling on the
machine stops treating llmgw's certificates as strangers, installs the binary
and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
a few seconds.

If it does not, the control plane's log says why. A certificate whose common
name does not match the claimed node id is refused by design — that check is
what makes a node's identity something it proves rather than something it picks.

**It needs ssh from the CA host to the node.** Provisioning copies that host's
`authorized_keys` into the new machine, so a node built by the script above
already satisfies it; an older machine may need the key added.

It waits for ssh rather than failing on it, and learns the host key of a machine
it has not seen (`accept-new`, which still refuses a *changed* key — the risk
being guarded against is a substituted machine, not a new one). Both are needed
to run the halves back to back: for the first minute of its life a new machine
is still regenerating host keys and restarting sshd, so it refuses connections
it accepted a moment earlier.

It also checks that a configured engine actually runs, which is cheap here and
expensive later — see below.

## The CPU model matters, and its default is wrong

Nodes are provisioned with `--cpu host`. Proxmox's default, `kvm64`, hides
AVX, AVX2, FMA and even SSE4.2 from the guest, and llama.cpp then loads its
scalar backend instead of the one built for the host's instruction set.
Measured on node05, an 8-core VM on a Xeon E5-2680 v4, same weights, same
threads, the only change being the CPU model:

| Qwen2.5-Coder  | kvm64 tok/s | host tok/s |
| -------------- | ----------: | ---------: |
| 1.5B q4_K_M    |        5.80 |      39.09 |
| 3B q4_K_M      |        2.90 |      20.51 |
| 7B q4_K_M      |        1.33 |       9.88 |

Prompt processing is worse still: 7.6 tokens a second against 143.8, a factor
of nineteen.

It is worth calling out because nothing looks broken. The node registers, the
engine starts, replies are correct — every number is simply small, and the
obvious conclusion is that CPU inference is hopeless rather than that the VM
cannot multiply efficiently.

The trade is live migration: a VM pinned to the host CPU cannot move to a
machine with a different one. For a node whose job is arithmetic that is the
right way round, and `--cpu` overrides it where it is not.

## Engines

The script installs none, and does not need to: once the agent is connected,
the control plane tells it which engines to have and it installs them itself
— llama.cpp on every node, vLLM on every node with an NVIDIA GPU it can run
on, SGLang where the control plane names a version. Each is checked to run
before the node offers it, so a node never advertises an engine that cannot
start. [install.md](install.md#engines) says what vLLM needs of the node.

`--llamacpp-bin`, `--vllm-bin` and `--sglang-python` are for an engine you
installed yourself. For the agent to use yours rather than install its own,
also set that engine's version to `""` under `engines` in `server.yaml`.

Weights are the other thing a node needs. llmgw fetches them for a model whose
`source` is a URL, one copy per node, which covers the simple case; anything
larger — a shared filesystem, a cache — stays the operator's arrangement, and
llmgw is content to find the files already there.

The `sim` engine needs neither, which is what makes it useful for exercising
everything else.

## Checking an engine after an upgrade

Engines rename and remove flags between versions, and when that happens the
failure arrives minutes into a model load, in the engine's own error message, on
a node nobody was watching.

```
llmgw ctl engine-check vllm
llmgw ctl engine-check llamacpp --help-output "$(llama-server --help 2>&1)"
```

It builds the widest command llmgw would ever produce — a multi-node one, since
a flag that only appears there is exactly the one nobody notices has gone — and
checks every flag against the engine's own `--help`. No GPU required: a CLI
prints its help on any machine.

Some programs do not give up their whole help for `--help`. vLLM prints a
summary of section names and expects `--help=all`; checking against the summary
reports every flag as missing, confidently and wrongly. `engine-check` knows
that one. If you add an engine, check that what you get back is the full list.

vLLM also refuses to do anything at all on a machine with no accelerator,
including print its help, unless `VLLM_TARGET_DEVICE` says what to assume.
`engine-check` sets it.

Give it the **whole** help. Truncating llama.cpp's 725 lines to 400 reports five
flags as missing that the running cluster uses; the command warns when the text
it was given does not document `--help` itself, which is the cheapest sign it
was cut short.

## Fabrics

`--fabric` is how a node says which InfiniBand fabric it is attached to. The
name is checked against what the adapters report — every port on one fabric
sees the same subnet managers — so two independent fabrics given one name are
placed as two, and the cluster page says so; give each fabric its own name.
The agent installs `infiniband-diags`, which that check needs, if it is missing.

Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
rest.

Omit it for a node with no fabric. Such a node is its own island: it can host
single-node replicas and will never be given a multi-node one.

Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
explicitly. An adapter must never appear under two fabrics — the agent refuses
that configuration: a replica given adapters on two fabrics starts, passes its
health checks, and hangs on its first all-reduce.

## Where the driver is

```
--nvidia-smi /opt/nvidia/bin/nvidia-smi
--ib-root /sys/class/infiniband
```

Both default to the usual places. Containerized driver installs routinely put
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
GPUs — which looks exactly like a node that has none.

## Removing a node

```
llmgw ctl drain llmgw-node04     # move its replicas off, one at a time
llmgw ctl nodes                  # wait until it holds none
```

Then stop the agent. Draining brings each replacement up before stopping the
original, which is the difference between a rolling move and an outage.

The node's certificate stays valid until it expires. There is no revocation list
— if a machine is decommissioned rather than retired, reissue the authority or
accept that a certificate naming a node that no longer exists can still connect
and claim to be it.

## Testing without hardware

Every part of this works on machines with no GPUs and no InfiniBand. Point
`--nvidia-smi` at a script that prints the CSV discovery asks for, point
`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.

The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
does not serve until every worker has joined, and workers serve nothing. That
exercises the multi-node launch sequence, the fabric constraint, and failure and
rebuild — which is most of what goes wrong — without exercising NCCL, which is
the rest.