llmgw/docs/fabrics.md

InfiniBand fabrics and NVLink

For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both — several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what llmgw does with them, what it needs from you before it can, how to check a node in one command, and where to look when a replica spanning nodes will not run.

Read install.md first; this is the part of it that only matters once there is a fabric.

Where this stands. Every piece here was written against the real formats of sminfo, saquery, ibnetdiscover, nvidia-smi topo -m and nvidia-smi -q, and tested against them — but none of it has yet run on a site with the hardware. Run llmgw doctor --json on two or three nodes first and keep the reports: if a parser meets a format it does not know, that report is what fixes it, and the fix is a control plane upgrade, not one per node.

What llmgw does with them

Two kinds of interconnect. InfiniBand or RoCE joins nodes (scale-out). NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes is placed, in order of preference:

1. inside one multi-node NVLink domain, if any has enough usable nodes: tensor parallelism between its nodes then runs over NVLink;
2. on one physical InfiniBand or RoCE fabric, preferring nodes under one leaf switch and putting nodes with a degraded cable last.

It is never split across two fabrics or two NVLink domains. Within a node, the GPUs a shard gets are the set with the most NVLink pairs, then the nearest to the fabric's adapters, then on one NUMA node.

A fabric has a name and an identity. The name is what agent.yaml says; the identity is what the adapters report — on InfiniBand the subnet managers every port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name is how you address a fabric; the identity is how llmgw catches the name being wrong: two independent fabrics given one name are placed as two, and the cluster page says so.

Before a node joins

On every GPU node:

What	Why	Check
NVIDIA driver	everything	nvidia-smi
nvidia-fabricmanager, on an NVSwitch machine	without it CUDA does not start on those GPUs	systemctl is-active nvidia-fabricmanager
rdma-core, ib_uverbs loaded	user-space RDMA at all	ls /sys/class/infiniband, lsmod | grep ib_uverbs
nvidia_peermem loaded	GPUDirect RDMA; without it NCCL copies every transfer through host memory	lsmod | grep nvidia_peermem
infiniband-diags	tells llmgw which fabric a port is on; the agent installs it if missing	which sminfo

On an NVL72 or NVLink Switch system, also:

What	Why	Check
nvidia-imex running, every node of the domain in /etc/nvidia-imex/nodes_config.cfg	GPUs in different nodes share memory through it	systemctl is-active nvidia-imex
an IMEX channel (options nvidia NVreg_CreateImexChannel0=1)	an engine opens one	ls /dev/nvidia-caps-imex-channels
every GPU joined to its domain	the fabric manager does this	nvidia-smi -q | grep -A6 Fabric shows State: Completed

On each InfiniBand fabric:

• A subnet manager, and a standby. Without one, ports come up to INIT and stay there; without a standby, the fabric keeps its routes if the master stops but cannot recover from any change after. llmgw watches this but does not run one.
• Its own name in agent.yaml on every node attached to it. Optionally, a distinct subnet_prefix in each subnet manager's configuration, which makes GIDs unique across fabrics; llmgw does not need it.

Declaring the fabrics

In /etc/llmgw/agent.yaml, the adapters on each fabric, by name:

fabrics:
  - name: ib-compute
    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
  - name: ib-storage
    hcas: [mlx5_4]

or --fabric ib-compute:mlx5_0,mlx5_1,… when onboarding (node-onboarding.md). An adapter belongs to one fabric; a node may be on several. A replica on ib-compute is given only ib-compute's adapters (NCCL_IB_HCA).

The rate is measured, not taken from the file: a fabric's rate on a node is the slowest of its active adapters, because a collective runs at the speed of its slowest link.

The agent checks the declaration against the adapters on every heartbeat. An adapter that is missing, or whose port is down, takes that fabric off the node within seconds — the node stops being offered multi-node work on it — and returns it when the port comes back.

Checking a node: llmgw doctor

sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
sudo llmgw doctor --json > $(hostname)-doctor.json

It reads the node and says what is wrong and how to fix it, most serious first, and exits non-zero when anything fails:

FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
              → modprobe nvidia_peermem
OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
OK    agent   NCCL bootstraps over ens3 (10.1.0.21)

followed by each GPU's NVLink peers and nearest adapter, each port's state, rate and errors, and the NCCL environment a shard on each fabric would be given. --json writes every fact it read as well as what it concluded — send that file when the report is not enough.

Checking two nodes: llmgw ctl fabric-test

Every check on one node can pass while two nodes still cannot reach each other — a cable into the wrong switch, a partition key, a RoCE network with no route — and a replica spanning them then hangs on its first all-reduce. Before placing one, on the control plane:

llmgw ctl fabric-test gpu01 gpu02

gpu01 → gpu02 on ib-compute, 41s

RAIL              LINK      MEASURED    VERDICT  DETAIL
mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517

It runs ib_write_bw (perftest, installed on a node the first time) from each of the first node's adapters on the fabric to the same adapter on the second — mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the agents, so no SSH between nodes is needed. Each rail is graded against its link's rate: ok at 80% or more, slow at 50%, fail below that or when nothing got through. --fabric picks one when the nodes share several, --seconds the length of each rail's run. The traffic is real: run it when the fabric is quiet.

It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says the fabric carries traffic between these two nodes at its rate, which is what a first all-reduce needs and what nothing on one node can show.

Watching them

• Cluster page, Fabrics. Each declared fabric and what it physically is; a table of subnet managers per fabric — master, priority, standbys, the last change of master — and every disagreement: one name on two fabrics, two names on one, an adapter up but undeclared, ports up with no subnet manager, a fabric with no standby, a recent failover.
• /admin/fabrics. Each fabric's map, from ibnetdiscover run by one of its nodes every five minutes: switches in their tiers, adapters under their leaves, cables coloured by rate, degraded ones in red, rail alignment, and the cluster's own adapters live with their traffic. Click anything for its detail.
• /admin/nvlink. Each multi-node NVLink domain as a rack of trays: which GPUs are joined, which node's IMEX is down, which link runs degraded.
• A node's page. Its ports with state, rate, errors and what they are physically on; its GPU-by-GPU NVLink matrix; its load.
• Metrics, for alerts:

Metric	Alert when
llmgw_ib_port_active	a declared port is 0
llmgw_ib_port_counter_total{counter="symbol_error"} and link_downed	rising
llmgw_ib_port_rate_gbps	below its usual rate
llmgw_fabric_subnet_managers{state="standby"}	absent or 0
llmgw_fabric_master_sm	its sm_guid changes (a failover)
llmgw_fabric_problems{level="fail"}	above 0
llmgw_gpu_nvlink_joined	0
llmgw_imex_up	0

What an engine is given

For a shard on an InfiniBand or RoCE fabric:

Variable	Value
NCCL_IB_HCA	that fabric's adapters on this node, and no others
NCCL_IB_DISABLE	0 (1 for a single-node shard)
NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME	the node's management interface: mgmt_interface, or the one holding its management address
NCCL_IB_GID_INDEX	on RoCE only: the RoCE v2 entry with an IPv4 address
NCCL_MNNVL_ENABLE, NCCL_CUMEM_ENABLE	1, for a replica inside a multi-node NVLink domain

A profile's env overrides any of them. vLLM launches a multi-node replica with its own multiprocessing (engines.vllm.launch_mode: mp, the default): every node runs vllm serve with its rank and the leader's address, the workers headless. launch_mode: ray is there for a vLLM too old for that.

When a replica spanning nodes will not run

It	Look at
is not placed	llmgw ctl explain-placement <model>: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics
starts and hangs on its first collective	llmgw ctl fabric-test between two of its nodes; the cluster page's fabric problems; doctor on each node — a port down, the wrong interface, RoCE without a v2 GID
runs, slowly	nvidia_peermem; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe
worked yesterday	the subnet manager table for a failover; link_downed on the node's ports