# InfiniBand fabrics and NVLink

For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
llmgw does with them, what it needs from you before it can, how to check a node
in one command, and where to look when a replica spanning nodes will not run.

Read [install.md](install.md) first; this is the part of it that only matters
once there is a fabric.

> **Where this stands.** Every piece here was written against the real formats
> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
> and keep the reports: if a parser meets a format it does not know, that report
> is what fixes it, and the fix is a control plane upgrade, not one per node.

## What llmgw does with them

**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
is placed, in order of preference:

1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
   tensor parallelism between its nodes then runs over NVLink;
2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
   leaf switch and putting nodes with a degraded cable last.

It is never split across two fabrics or two NVLink domains. Within a node, the
GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
the fabric's adapters, then on one NUMA node.

**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
identity is what the adapters report — on InfiniBand the subnet managers every
port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
is how you address a fabric; the identity is how llmgw catches the name being
wrong: two independent fabrics given one name are placed as two, and the
cluster page says so.

## Before a node joins

On every GPU node:

| What | Why | Check |
|---|---|---|
| NVIDIA driver | everything | `nvidia-smi` |
| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |

On an NVL72 or NVLink Switch system, also:

| What | Why | Check |
|---|---|---|
| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |

On each InfiniBand fabric:

- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
  stay there; without a standby, the fabric keeps its routes if the master
  stops but cannot recover from any change after. llmgw watches this but does
  not run one.
- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
  distinct `subnet_prefix` in each subnet manager's configuration, which makes
  GIDs unique across fabrics; llmgw does not need it.

## Declaring the fabrics

In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:

```yaml
fabrics:
  - name: ib-compute
    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
  - name: ib-storage
    hcas: [mlx5_4]
```

or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
An adapter belongs to one fabric; a node may be on several. A replica on
`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).

The rate is measured, not taken from the file: a fabric's rate on a node is the
slowest of its active adapters, because a collective runs at the speed of its
slowest link.

The agent checks the declaration against the adapters on every heartbeat. An
adapter that is missing, or whose port is down, takes that fabric off the node
within seconds — the node stops being offered multi-node work on it — and
returns it when the port comes back.

## Checking a node: `llmgw doctor`

```
sudo llmgw doctor                 # uses /etc/llmgw/agent.yaml when present
sudo llmgw doctor --json > $(hostname)-doctor.json
```

It reads the node and says what is wrong and how to fix it, most serious first,
and exits non-zero when anything fails:

```
FAIL  nvlink  this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
              → systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
WARN  fabric  nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
              → modprobe nvidia_peermem
OK    gpu     8 GPUs: 8× NVIDIA H100 80GB HBM3
OK    nvlink  NVLink groups: [0 1 2 3 4 5 6 7]
OK    fabric  mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
OK    fabric  subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
OK    agent   fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
OK    agent   NCCL bootstraps over ens3 (10.1.0.21)
```

followed by each GPU's NVLink peers and nearest adapter, each port's state,
rate and errors, and the NCCL environment a shard on each fabric would be given.
`--json` writes every fact it read as well as what it concluded — send that
file when the report is not enough.

## Checking two nodes: `llmgw ctl fabric-test`

Every check on one node can pass while two nodes still cannot reach each other
— a cable into the wrong switch, a partition key, a RoCE network with no route
— and a replica spanning them then hangs on its first all-reduce. Before
placing one, on the control plane:

```
llmgw ctl fabric-test gpu01 gpu02
```

```
gpu01 → gpu02 on ib-compute, 41s

RAIL              LINK      MEASURED    VERDICT  DETAIL
mlx5_0 → mlx5_0   400 Gb/s  388.2 Gb/s  ok       97% of 400 Gb/s
mlx5_1 → mlx5_1   400 Gb/s  241.0 Gb/s  slow     60% of 400 Gb/s: check the cable, and the card's PCIe width and NUMA node
mlx5_2 → mlx5_2   400 Gb/s  0.0 Gb/s    fail     gpu01: ib_write_bw: Couldn't connect to 10.1.0.22:18517
```

It runs `ib_write_bw` (perftest, installed on a node the first time) from each
of the first node's adapters on the fabric to the same adapter on the second —
mlx5_0 to mlx5_0, the way NCCL pairs them — one rail at a time, through the
agents, so no SSH between nodes is needed. Each rail is graded against its
link's rate: **ok** at 80% or more, **slow** at 50%, **fail** below that or when
nothing got through. `--fabric` picks one when the nodes share several,
`--seconds` the length of each rail's run. The traffic is real: run it when the
fabric is quiet.

It measures RDMA between adapters, not GPUDirect or NCCL itself: a pass says
the fabric carries traffic between these two nodes at its rate, which is what
a first all-reduce needs and what nothing on one node can show.

## Watching them

- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
  a table of subnet managers per fabric — master, priority, standbys, the last
  change of master — and every disagreement: one name on two fabrics, two names
  on one, an adapter up but undeclared, ports up with no subnet manager, a
  fabric with no standby, a recent failover.
- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
  its nodes every five minutes: switches in their tiers, adapters under their
  leaves, cables coloured by rate, degraded ones in red, rail alignment, and
  the cluster's own adapters live with their traffic. Click anything for its
  detail.
- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
  GPUs are joined, which node's IMEX is down, which link runs degraded.
- **A node's page.** Its ports with state, rate, errors and what they are
  physically on; its GPU-by-GPU NVLink matrix; its load.
- **Metrics**, for alerts:

| Metric | Alert when |
|---|---|
| `llmgw_ib_port_active` | a declared port is 0 |
| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
| `llmgw_ib_port_rate_gbps` | below its usual rate |
| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
| `llmgw_fabric_problems{level="fail"}` | above 0 |
| `llmgw_gpu_nvlink_joined` | 0 |
| `llmgw_imex_up` | 0 |

## What an engine is given

For a shard on an InfiniBand or RoCE fabric:

| Variable | Value |
|---|---|
| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |

A profile's `env` overrides any of them. vLLM launches a multi-node replica
with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
every node runs `vllm serve` with its rank and the leader's address, the
workers headless. `launch_mode: ray` is there for a vLLM too old for that.

## When a replica spanning nodes will not run

| It | Look at |
|---|---|
| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
| starts and hangs on its first collective | `llmgw ctl fabric-test` between two of its nodes; the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |