llmgw/docs/fabrics.md

InfiniBand fabrics and NVLink

For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both — several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what llmgw does with them, what it needs from you before it can, how to check a node in one command, and where to look when a replica spanning nodes will not run.

Read install.md first; this is the part of it that only matters once there is a fabric.

Where this stands. Every piece here was written against the real formats of sminfo, saquery, ibnetdiscover, nvidia-smi topo -m and nvidia-smi -q, and tested against them — but none of it has yet run on a site with the hardware. Run llmgw doctor --json on two or three nodes first and keep the reports: if a parser meets a format it does not know, that report is what fixes it, and the fix is a control plane upgrade, not one per node.

What llmgw does with them

Two kinds of interconnect. InfiniBand or RoCE joins nodes (scale-out). NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes is placed, in order of preference:

1. inside one multi-node NVLink domain, if any has enough usable nodes: tensor parallelism between its nodes then runs over NVLink;
2. on one physical InfiniBand or RoCE fabric, preferring nodes under one leaf switch and putting nodes with a degraded cable last.

It is never split across two fabrics or two NVLink domains. Within a node, the GPUs a shard gets are the set with the most NVLink pairs, then the nearest to the fabric's adapters, then on one NUMA node.

A fabric has a name and an identity. The name is what agent.yaml says; the identity is what the adapters report — on InfiniBand the subnet managers every port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name is how you address a fabric; the identity is how llmgw catches the name being wrong: two independent fabrics given one name are placed as two, and the cluster page says so.

Before a node joins

On every GPU node:

What	Why	Check
NVIDIA driver	everything	nvidia-smi
nvidia-fabricmanager, on an NVSwitch machine	without it CUDA does not start on those GPUs	systemctl is-active nvidia-fabricmanager
rdma-core, ib_uverbs loaded	user-space RDMA at all	ls /sys/class/infiniband, lsmod | grep ib_uverbs
nvidia_peermem loaded	GPUDirect RDMA; without it NCCL copies every transfer through host memory	lsmod | grep nvidia_peermem
infiniband-diags	tells llmgw which fabric a port is on; the agent installs it if missing	which sminfo

On an NVL72 or NVLink Switch system, also:

What	Why	Check
nvidia-imex running, every node of the domain in /etc/nvidia-imex/nodes_config.cfg	GPUs in different nodes share memory through it	systemctl is-active nvidia-imex
an IMEX channel (options nvidia NVreg_CreateImexChannel0=1)	an engine opens one	ls /dev/nvidia-caps-imex-channels
every GPU joined to its domain	the fabric manager does this	nvidia-smi -q | grep -A6 Fabric shows State: Completed

On each InfiniBand fabric:

• A subnet manager, and a standby. Without one, ports come up to INIT and stay there; without a standby, the fabric keeps its routes if the master stops but cannot recover from any change after. llmgw watches this but does not run one.
• Its own name in agent.yaml on every node attached to it. Optionally, a distinct subnet_prefix in each subnet manager's configuration, which makes GIDs unique across fabrics; llmgw does not need it.

Declaring the fabrics

In /etc/llmgw/agent.yaml, the adapters on each fabric, by name:

fabrics:
  - name: ib-compute
    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
  - name: ib-storage
    hcas: [mlx5_4]

or --fabric ib-compute:mlx5_0,mlx5_1,… when onboarding (node-onboarding.md). An adapter belongs to one fabric; a node may be on several. A replica on ib-compute is given only ib-compute's adapters (NCCL_IB_HCA).

The rate is measured, not taken from the file: a fabric's rate on a node is the slowest of its active adapters, because a collective runs at the speed of its slowest link.

The agent checks the declaration against the adapters on every heartbeat. An adapter that is missing, or whose port is down, takes that fabric off the node within seconds — the node stops being offered multi-node work on it — and returns it when the port comes back.

Checking a node: llmgw doctor

sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
sudo llmgw doctor --json > $(hostname)-doctor.json

It reads the node and says what is wrong and how to fix it, most serious first, and exits non-zero when anything fails:

FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
              → modprobe nvidia_peermem
OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
OK    agent   NCCL bootstraps over ens3 (10.1.0.21)

followed by each GPU's NVLink peers and nearest adapter, each port's state, rate and errors, and the NCCL environment a shard on each fabric would be given. --json writes every fact it read as well as what it concluded — send that file when the report is not enough.

Watching them

• Cluster page, Fabrics. Each declared fabric and what it physically is; a table of subnet managers per fabric — master, priority, standbys, the last change of master — and every disagreement: one name on two fabrics, two names on one, an adapter up but undeclared, ports up with no subnet manager, a fabric with no standby, a recent failover.
• /admin/fabrics. Each fabric's map, from ibnetdiscover run by one of its nodes every five minutes: switches in their tiers, adapters under their leaves, cables coloured by rate, degraded ones in red, rail alignment, and the cluster's own adapters live with their traffic. Click anything for its detail.
• /admin/nvlink. Each multi-node NVLink domain as a rack of trays: which GPUs are joined, which node's IMEX is down, which link runs degraded.
• A node's page. Its ports with state, rate, errors and what they are physically on; its GPU-by-GPU NVLink matrix; its load.
• Metrics, for alerts:

Metric	Alert when
llmgw_ib_port_active	a declared port is 0
llmgw_ib_port_counter_total{counter="symbol_error"} and link_downed	rising
llmgw_ib_port_rate_gbps	below its usual rate
llmgw_fabric_subnet_managers{state="standby"}	absent or 0
llmgw_fabric_master_sm	its sm_guid changes (a failover)
llmgw_fabric_problems{level="fail"}	above 0
llmgw_gpu_nvlink_joined	0
llmgw_imex_up	0

What an engine is given

For a shard on an InfiniBand or RoCE fabric:

Variable	Value
NCCL_IB_HCA	that fabric's adapters on this node, and no others
NCCL_IB_DISABLE	0 (1 for a single-node shard)
NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME	the node's management interface: mgmt_interface, or the one holding its management address
NCCL_IB_GID_INDEX	on RoCE only: the RoCE v2 entry with an IPv4 address
NCCL_MNNVL_ENABLE, NCCL_CUMEM_ENABLE	1, for a replica inside a multi-node NVLink domain

A profile's env overrides any of them. vLLM launches a multi-node replica with its own multiprocessing (engines.vllm.launch_mode: mp, the default): every node runs vllm serve with its rank and the leader's address, the workers headless. launch_mode: ray is there for a vLLM too old for that.

When a replica spanning nodes will not run

It	Look at
is not placed	llmgw ctl explain-placement <model>: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics
starts and hangs on its first collective	the cluster page's fabric problems; doctor on each node — a port down, the wrong interface, RoCE without a v2 GID
runs, slowly	nvidia_peermem; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe
worked yesterday	the subnet manager table for a failover; link_downed on the node's ports