Prove the multi-node path on a simulated cluster

Phase 4 had been written and never run. The hardware is not coming, so the
cluster is simulated instead — four nodes across two fabrics, eight GPUs each,
built with the onboarding scripts from the previous commit.

`sim` is now an engine adapter, not just a standalone command. Its leader binds
a rendezvous port and does not serve until every worker has joined; its workers
serve nothing at all. That is the part stub processes could never exercise, and
it is why this path survived four phases untested: the command line was
checked, the sequence was not.

Measured on the cluster:

- a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on
  ib-b, four GPUs from each
- a request served through one of them
- only the leader listening — node02 had no port open at all, which is
  ServesHTTP working end to end rather than in a unit test
- node02 killed; at t+60s it was declared lost, the ib-a replica failed and was
  reaped; at t+80s a replacement was ready on ib-b, the fabric that could still
  host it, while node01 sat idle rather than being paired across fabrics

The invariant held under the one condition that tests it: when a replica could
not be rebuilt where it was, it moved to a fabric that could hold all of it
rather than spanning two.

What this does not prove is written into the plan next to what it does. The
GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine
speaks no NCCL — so the launch sequence is now exercised while the launch
arguments for vLLM and SGLang are still read from documentation rather than
from a running engine.

docs/node-onboarding.md is the procedure, including how to run all of this on
machines that have neither GPUs nor InfiniBand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

deploy/bootstrap/onboard-node.sh

@@ -31,6 +31,9 @@ usage: $0 --node ID --addr HOST --server URL [options]
                     a multi-node one.
   --mgmt-addr ADDR  address engines are reachable on (default: --addr)
   --mgmt-if NAME    interface NCCL bootstraps over (default eth0)
+  --nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
+                    driver installs routinely put it somewhere else.
+  --ib-root DIR     the sysfs InfiniBand directory (default /sys/class/infiniband)
   --pki DIR         where the authority lives (default /etc/llmgw/pki)
   --binary PATH     llmgw binary to install (default: the one running this)
   --ssh-user NAME   (default ilker)
@@ -39,6 +42,7 @@ USAGE
 }
 
 NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
+NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
 PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
 FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
 
@@ -50,6 +54,8 @@ while [ $# -gt 0 ]; do
     --fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
     --mgmt-addr) MGMT="$2"; shift 2;;
     --mgmt-if) MGMT_IF="$2"; shift 2;;
+    --nvidia-smi) NVIDIA_SMI="$2"; shift 2;;
+    --ib-root) IB_ROOT="$2"; shift 2;;
     --pki) PKI="$2"; shift 2;;
     --binary) BINARY="$2"; shift 2;;
     --ssh-user) SSH_USER="$2"; shift 2;;
@@ -127,6 +133,10 @@ reconnect_max: 30s
 $FABRICS
 
 discovery:
+  # Empty means look on PATH. Worth setting where the driver lives outside the
+  # default paths, which containerized installs routinely do.
+  nvidia_smi: "$NVIDIA_SMI"
+  ib_root: $IB_ROOT
   prefer_nvml: false
   skip_gpus: false
 

docs/node-onboarding.md

@@ -0,0 +1,103 @@
+# Adding a node
+
+Written for whoever runs the cluster. Two scripts, because the two halves of
+onboarding are done by different people with different access: one needs the
+hypervisor, the other needs the certificate authority.
+
+## 1. The machine
+
+On the Proxmox host:
+
+```
+llmgw-provision-node --vmid 144 --name llmgw-node04 --ip 10.0.0.144/24
+```
+
+Name it after the node id. **The VM name, the hostname, the node id and the
+certificate's common name are all the same string.** They do not have to be, and
+every time they are not, somebody spends an afternoon working out which of the
+four a message is talking about.
+
+It imports a cloud image and lets cloud-init do the rest, which is why a node
+takes a minute rather than an afternoon. It refuses to touch an existing VMID:
+adopting a machine is not a decision a provisioning script should make.
+
+## 2. The identity
+
+On the host that holds the certificate authority — this is the part that cannot
+be delegated, because it issues the credential by which the control plane
+believes the node is who it says it is:
+
+```
+llmgw-onboard-node \
+  --node llmgw-node04 \
+  --addr 10.0.0.144 \
+  --server https://10.0.0.140:7443 \
+  --fabric ib-b
+```
+
+It issues a client certificate whose common name is the node id, installs it
+along with the CA root, trusts that root system-wide so other tooling on the
+machine stops treating llmgw's certificates as strangers, installs the binary
+and the unit, and starts the agent. The node appears in `llmgw ctl nodes` within
+a few seconds.
+
+If it does not, the control plane's log says why. A certificate whose common
+name does not match the claimed node id is refused by design — that check is
+what makes a node's identity something it proves rather than something it picks.
+
+**It needs ssh from the CA host to the node.** Provisioning copies that host's
+`authorized_keys` into the new machine, so a node built by the script above
+already satisfies it; an older machine may need the key added.
+
+## Fabrics
+
+`--fabric` is how a node says which InfiniBand fabric it is attached to, and
+there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
+every fabric, so a matching prefix proves nothing and a wrong guess produces a
+replica that starts, passes health checks, and then hangs on its first
+all-reduce.
+
+Omit it for a node with no fabric. Such a node is its own island: it can host
+single-node replicas and will never be given a multi-node one.
+
+Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
+explicitly. An adapter must never appear under two fabrics — the agent refuses
+that configuration, because the consequence is the hang above.
+
+## Where the driver is
+
+```
+--nvidia-smi /opt/nvidia/bin/nvidia-smi
+--ib-root /sys/class/infiniband
+```
+
+Both default to the usual places. Containerized driver installs routinely put
+`nvidia-smi` somewhere else, and an agent that cannot find it registers with no
+GPUs — which looks exactly like a node that has none.
+
+## Removing a node
+
+```
+llmgw ctl drain llmgw-node04     # move its replicas off, one at a time
+llmgw ctl nodes                  # wait until it holds none
+```
+
+Then stop the agent. Draining brings each replacement up before stopping the
+original, which is the difference between a rolling move and an outage.
+
+The node's certificate stays valid until it expires. There is no revocation list
+— if a machine is decommissioned rather than retired, reissue the authority or
+accept that a certificate naming a node that no longer exists can still connect
+and claim to be it.
+
+## Testing without hardware
+
+Every part of this works on machines with no GPUs and no InfiniBand. Point
+`--nvidia-smi` at a script that prints the CSV discovery asks for, point
+`--ib-root` at a sysfs-shaped directory, and use `engine: sim` in the catalog.
+
+The simulated engine rendezvouses for real: rank 0 binds the dist-init port and
+does not serve until every worker has joined, and workers serve nothing. That
+exercises the multi-node launch sequence, the fabric constraint, and failure and
+rebuild — which is most of what goes wrong — without exercising NCCL, which is
+the rest.

llmgw-plan.md

@@ -1391,12 +1391,32 @@ The launch path is now built too:
       bounded by `rendezvous_timeout` so a head that never appears fails the
       shard instead of hanging it.
 
-- [ ] **Not verified against real engines.** Everything above is tested with
-      stub processes. The Ray sequencing, and whether the deployed vLLM version
-      wants `ray` or `mp`, can only be confirmed on real hardware.
+- [x] **Verified against a simulated cluster**, which is not the same as real
+      hardware and covers more than stub processes did. Four nodes across two
+      fabrics, eight GPUs each; `sim` is a real engine adapter whose leader
+      binds a rendezvous port and does not serve until every worker has joined,
+      and whose workers serve nothing at all.
 
-- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the
-  same fabric and never spreads it across two.
+      What that proves: placement honours the fabric constraint, the leader
+      starts first and workers wait for its port, only rank 0 is health-checked,
+      a 2-node replica comes up on each fabric, and losing half of one fails the
+      replica and rebuilds it **within a single fabric** rather than spanning
+      two.
+
+      What it does not prove: that a given vLLM version wants `ray` or `mp`,
+      that NCCL picks the right HCA, or that the generated environment makes a
+      real collective run. Those need hardware. The GPUs here are an inventory,
+      the fabric is a sysfs-shaped directory, and the engine speaks no NCCL.
+- [ ] **Confirm the engine flags on real hardware.** The launch *sequence* is
+      now exercised; the launch *arguments* for vLLM and SGLang are still read
+      from their documentation rather than from a running engine.
+
+- **Exit criterion (met on the simulated cluster):** a 2-node replica on each
+  fabric; killing a node rebuilds the replica within one fabric and never
+  spreads it across two. Measured: node02 lost at t+60s, the ib-a replica failed
+  and was reaped, a replacement was ready at t+80s on ib-b — the fabric that
+  could still host it — while node01 sat idle rather than being paired across
+  fabrics with a node from ib-b.
 
 ### Phase 5 — Production features (3 weeks)
 - [x] **SQLite store and migrations**, behind an interface, with per-dialect