# Adding a node Written for whoever runs the cluster. Two scripts, because the two halves of onboarding are done by different people with different access: one needs the hypervisor, the other needs the certificate authority. ## 1. The machine On the Proxmox host: ``` llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24 ``` Name it after the node id. **The VM name, the hostname, the node id and the certificate's common name are all the same string.** They do not have to be, and every time they are not, somebody spends an afternoon working out which of the four a message is talking about. It imports a cloud image and lets cloud-init do the rest, which is why a node takes a minute rather than an afternoon. It refuses to touch an existing VMID: adopting a machine is not a decision a provisioning script should make. Releases carry static `linux/amd64` and `linux/arm64` binaries. Download one and pass it with `--binary` rather than copying whatever happens to be on the machine you are running onboarding from — otherwise whoever runs the script quietly becomes the build server, and the fleet ends up running whatever was in their working tree. ## 2. The identity On the host that holds the certificate authority — this is the part that cannot be delegated, because it issues the credential by which the control plane believes the node is who it says it is: ``` llmgw-onboard-node \ --node llmgw-node04 \ --addr 10.0.0.144 \ --server https://10.0.0.140:7443 \ --fabric ib-b ``` It issues a client certificate whose common name is the node id, installs it along with the CA root, trusts that root system-wide so other tooling on the machine stops treating llmgw's certificates as strangers, installs the binary and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within a few seconds. If it does not, the control plane's log says why. A certificate whose common name does not match the claimed node id is refused by design — that check is what makes a node's identity something it proves rather than something it picks. **It needs ssh from the CA host to the node.** Provisioning copies that host's `authorized_keys` into the new machine, so a node built by the script above already satisfies it; an older machine may need the key added. ## What onboarding does not do It gives a machine an identity and an agent. It does **not** install inference engines or model weights, and a node missing either looks healthy in `llmgw ctl nodes` right up until a replica is placed on it and the shard dies with "executable file not found". So the node is not finished until: - the engines its catalog profiles name are installed (`vllm`, `sglang`, `llama-server`), at the paths the agent configuration gives - the weights are on disk, at the `source` each model declares Both are deliberately outside this script. Engines are a platform decision with versions and drivers attached, and distributing weights is its own problem — hundreds of gigabytes that want a shared filesystem or a cache, not an scp from a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight. The `sim` engine needs neither, which is what makes it useful for exercising everything else. ## Checking an engine after an upgrade Engines rename and remove flags between versions, and when that happens the failure arrives minutes into a model load, in the engine's own error message, on a node nobody was watching. ``` llmgw ctl engine-check vllm llmgw ctl engine-check llamacpp --help-output "$(llama-server --help 2>&1)" ``` It builds the widest command llmgw would ever produce — a multi-node one, since a flag that only appears there is exactly the one nobody notices has gone — and checks every flag against the engine's own `--help`. No GPU required: a CLI prints its help on any machine. Give it the **whole** help. Truncating llama.cpp's 725 lines to 400 reports five flags as missing that the running cluster uses; the command warns when the text it was given does not document `--help` itself, which is the cheapest sign it was cut short. ## Fabrics `--fabric` is how a node says which InfiniBand fabric it is attached to, and there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on every fabric, so a matching prefix proves nothing and a wrong guess produces a replica that starts, passes health checks, and then hangs on its first all-reduce. Omit it for a node with no fabric. Such a node is its own island: it can host single-node replicas and will never be given a multi-node one. Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them explicitly. An adapter must never appear under two fabrics — the agent refuses that configuration, because the consequence is the hang above. ## Where the driver is ``` --nvidia-smi /opt/nvidia/bin/nvidia-smi --ib-root /sys/class/infiniband ``` Both default to the usual places. Containerized driver installs routinely put `nvidia-smi` somewhere else, and an agent that cannot find it registers with no GPUs — which looks exactly like a node that has none. ## Removing a node ``` llmgw ctl drain llmgw-node04 # move its replicas off, one at a time llmgw ctl nodes # wait until it holds none ``` Then stop the agent. Draining brings each replacement up before stopping the original, which is the difference between a rolling move and an outage. The node's certificate stays valid until it expires. There is no revocation list — if a machine is decommissioned rather than retired, reissue the authority or accept that a certificate naming a node that no longer exists can still connect and claim to be it. ## Testing without hardware Every part of this works on machines with no GPUs and no InfiniBand. Point `--nvidia-smi` at a script that prints the CSV discovery asks for, point `--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog. The simulated engine rendezvouses for real: rank 0 binds the dist-init port and does not serve until every worker has joined, and workers serve nothing. That exercises the multi-node launch sequence, the fabric constraint, and failure and rebuild — which is most of what goes wrong — without exercising NCCL, which is the rest.