Prove the multi-node path on a simulated cluster
Phase 4 had been written and never run. The hardware is not coming, so the cluster is simulated instead — four nodes across two fabrics, eight GPUs each, built with the onboarding scripts from the previous commit. `sim` is now an engine adapter, not just a standalone command. Its leader binds a rendezvous port and does not serve until every worker has joined; its workers serve nothing at all. That is the part stub processes could never exercise, and it is why this path survived four phases untested: the command line was checked, the sequence was not. Measured on the cluster: - a 2-node replica on each fabric, node01+node02 on ib-a and node03+node04 on ib-b, four GPUs from each - a request served through one of them - only the leader listening — node02 had no port open at all, which is ServesHTTP working end to end rather than in a unit test - node02 killed; at t+60s it was declared lost, the ib-a replica failed and was reaped; at t+80s a replacement was ready on ib-b, the fabric that could still host it, while node01 sat idle rather than being paired across fabrics The invariant held under the one condition that tests it: when a replica could not be rebuilt where it was, it moved to a fabric that could hold all of it rather than spanning two. What this does not prove is written into the plan next to what it does. The GPUs are an inventory, the fabric is a sysfs-shaped directory, and the engine speaks no NCCL — so the launch sequence is now exercised while the launch arguments for vLLM and SGLang are still read from documentation rather than from a running engine. docs/node-onboarding.md is the procedure, including how to run all of this on machines that have neither GPUs nor InfiniBand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c1ac2f022f
commit
c97b671ef9
3 files changed
+138
-5
No files matched your search
@@ -31,6 +31,9 @@ usage: $0 --node ID --addr HOST --server URL [options]
|
||||
a multi-node one.
|
||||
--mgmt-addr ADDR address engines are reachable on (default: --addr)
|
||||
--mgmt-if NAME interface NCCL bootstraps over (default eth0)
|
||||
--nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
|
||||
driver installs routinely put it somewhere else.
|
||||
--ib-root DIR the sysfs InfiniBand directory (default /sys/class/infiniband)
|
||||
--pki DIR where the authority lives (default /etc/llmgw/pki)
|
||||
--binary PATH llmgw binary to install (default: the one running this)
|
||||
--ssh-user NAME (default ilker)
|
||||
@@ -39,6 +42,7 @@ USAGE
|
||||
}
|
||||
|
||||
NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
|
||||
NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
|
||||
PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
|
||||
FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
|
||||
|
||||
@@ -50,6 +54,8 @@ while [ $# -gt 0 ]; do
|
||||
--fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
|
||||
--mgmt-addr) MGMT="$2"; shift 2;;
|
||||
--mgmt-if) MGMT_IF="$2"; shift 2;;
|
||||
--nvidia-smi) NVIDIA_SMI="$2"; shift 2;;
|
||||
--ib-root) IB_ROOT="$2"; shift 2;;
|
||||
--pki) PKI="$2"; shift 2;;
|
||||
--binary) BINARY="$2"; shift 2;;
|
||||
--ssh-user) SSH_USER="$2"; shift 2;;
|
||||
@@ -127,6 +133,10 @@ reconnect_max: 30s
|
||||
$FABRICS
|
||||
|
||||
discovery:
|
||||
# Empty means look on PATH. Worth setting where the driver lives outside the
|
||||
# default paths, which containerized installs routinely do.
|
||||
nvidia_smi: "$NVIDIA_SMI"
|
||||
ib_root: $IB_ROOT
|
||||
prefer_nvml: false
|
||||
skip_gpus: false
|
||||
|
||||
|
||||
Reference in new issue
Block a user