# Adding a node

Written for whoever runs the cluster. Two scripts, because the two halves of
onboarding are done by different people with different access: one needs the
hypervisor, the other needs the certificate authority.

## 1. The machine

On the Proxmox host:

```
llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
```

Name it after the node id. **The VM name, the hostname, the node id and the
certificate's common name are all the same string.** They do not have to be, and
every time they are not, somebody spends an afternoon working out which of the
four a message is talking about.

It imports a cloud image and lets cloud-init do the rest, which is why a node
takes a minute rather than an afternoon. It refuses to touch an existing VMID:
adopting a machine is not a decision a provisioning script should make.

## 2. The identity

On the host that holds the certificate authority — this is the part that cannot
be delegated, because it issues the credential by which the control plane
believes the node is who it says it is:

```
llmgw-onboard-node \
  --node llmgw-node04 \
  --addr 10.0.0.144 \
  --server https://10.0.0.140:7443 \
  --fabric ib-b
```

It issues a client certificate whose common name is the node id, installs it
along with the CA root, trusts that root system-wide so other tooling on the
machine stops treating llmgw's certificates as strangers, installs the binary
and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
a few seconds.

If it does not, the control plane's log says why. A certificate whose common
name does not match the claimed node id is refused by design — that check is
what makes a node's identity something it proves rather than something it picks.

**It needs ssh from the CA host to the node.** Provisioning copies that host's
`authorized_keys` into the new machine, so a node built by the script above
already satisfies it; an older machine may need the key added.

## Fabrics

`--fabric` is how a node says which InfiniBand fabric it is attached to, and
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
every fabric, so a matching prefix proves nothing and a wrong guess produces a
replica that starts, passes health checks, and then hangs on its first
all-reduce.

Omit it for a node with no fabric. Such a node is its own island: it can host
single-node replicas and will never be given a multi-node one.

Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
explicitly. An adapter must never appear under two fabrics — the agent refuses
that configuration, because the consequence is the hang above.

## Where the driver is

```
--nvidia-smi /opt/nvidia/bin/nvidia-smi
--ib-root /sys/class/infiniband
```

Both default to the usual places. Containerized driver installs routinely put
`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
GPUs — which looks exactly like a node that has none.

## Removing a node

```
llmgw ctl drain llmgw-node04     # move its replicas off, one at a time
llmgw ctl nodes                  # wait until it holds none
```

Then stop the agent. Draining brings each replacement up before stopping the
original, which is the difference between a rolling move and an outage.

The node's certificate stays valid until it expires. There is no revocation list
— if a machine is decommissioned rather than retired, reissue the authority or
accept that a certificate naming a node that no longer exists can still connect
and claim to be it.

## Testing without hardware

Every part of this works on machines with no GPUs and no InfiniBand. Point
`--nvidia-smi` at a script that prints the CSV discovery asks for, point
`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.

The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
does not serve until every worker has joined, and workers serve nothing. That
exercises the multi-node launch sequence, the fabric constraint, and failure and
rebuild — which is most of what goes wrong — without exercising NCCL, which is
the rest.