Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable weights; it now covers the rest: telling independent fabrics apart, their subnet managers and maps, NVLink on every build and across nodes, the multi-node launch fixes and vLLM's new default, doctor, node load, editors and the Responses API, the chat and mirror pages, a person's own use, the admin pages, the new metrics, and how to upgrade. docs/fabrics.md is new, for a site with InfiniBand or NVLink: what llmgw does with them, what each node and fabric needs first, declaring fabrics, checking a node with doctor, what the pages and metrics show and which to alert on, what an engine is given, and where to look when a replica spanning nodes will not run. It says plainly that none of it has yet run on such hardware. install.md, node-onboarding.md, onboarding.md, the README and the example agent configuration no longer say the portable binary cannot see NVLink or that a fabric's name cannot be checked, and describe the account page and connecting an editor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed
1 parent
f989e67e1a
commit
80e0dd252f
10 files changed
+440
-67
No files matched your search
+177
@@ -0,0 +1,177 @@
|
||||
# InfiniBand fabrics and NVLink
|
||||
|
||||
For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
|
||||
— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
|
||||
llmgw does with them, what it needs from you before it can, how to check a node
|
||||
in one command, and where to look when a replica spanning nodes will not run.
|
||||
|
||||
Read [install.md](install.md) first; this is the part of it that only matters
|
||||
once there is a fabric.
|
||||
|
||||
> **Where this stands.** Every piece here was written against the real formats
|
||||
> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
|
||||
> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
|
||||
> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
|
||||
> and keep the reports: if a parser meets a format it does not know, that report
|
||||
> is what fixes it, and the fix is a control plane upgrade, not one per node.
|
||||
|
||||
## What llmgw does with them
|
||||
|
||||
**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
|
||||
NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
|
||||
rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
|
||||
is placed, in order of preference:
|
||||
|
||||
1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
|
||||
tensor parallelism between its nodes then runs over NVLink;
|
||||
2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
|
||||
leaf switch and putting nodes with a degraded cable last.
|
||||
|
||||
It is never split across two fabrics or two NVLink domains. Within a node, the
|
||||
GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
|
||||
the fabric's adapters, then on one NUMA node.
|
||||
|
||||
**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
|
||||
identity is what the adapters report — on InfiniBand the subnet managers every
|
||||
port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
|
||||
is how you address a fabric; the identity is how llmgw catches the name being
|
||||
wrong: two independent fabrics given one name are placed as two, and the
|
||||
cluster page says so.
|
||||
|
||||
## Before a node joins
|
||||
|
||||
On every GPU node:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| NVIDIA driver | everything | `nvidia-smi` |
|
||||
| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
|
||||
| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
|
||||
| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
|
||||
| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
|
||||
|
||||
On an NVL72 or NVLink Switch system, also:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
|
||||
| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
|
||||
| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
|
||||
|
||||
On each InfiniBand fabric:
|
||||
|
||||
- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
|
||||
stay there; without a standby, the fabric keeps its routes if the master
|
||||
stops but cannot recover from any change after. llmgw watches this but does
|
||||
not run one.
|
||||
- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
|
||||
distinct `subnet_prefix` in each subnet manager's configuration, which makes
|
||||
GIDs unique across fabrics; llmgw does not need it.
|
||||
|
||||
## Declaring the fabrics
|
||||
|
||||
In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
|
||||
|
||||
```yaml
|
||||
fabrics:
|
||||
- name: ib-compute
|
||||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
|
||||
- name: ib-storage
|
||||
hcas: [mlx5_4]
|
||||
```
|
||||
|
||||
or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
|
||||
An adapter belongs to one fabric; a node may be on several. A replica on
|
||||
`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
|
||||
|
||||
The rate is measured, not taken from the file: a fabric's rate on a node is the
|
||||
slowest of its active adapters, because a collective runs at the speed of its
|
||||
slowest link.
|
||||
|
||||
The agent checks the declaration against the adapters on every heartbeat. An
|
||||
adapter that is missing, or whose port is down, takes that fabric off the node
|
||||
within seconds — the node stops being offered multi-node work on it — and
|
||||
returns it when the port comes back.
|
||||
|
||||
## Checking a node: `llmgw doctor`
|
||||
|
||||
```
|
||||
sudo llmgw doctor # uses /etc/llmgw/agent.yaml when present
|
||||
sudo llmgw doctor --json > $(hostname)-doctor.json
|
||||
```
|
||||
|
||||
It reads the node and says what is wrong and how to fix it, most serious first,
|
||||
and exits non-zero when anything fails:
|
||||
|
||||
```
|
||||
FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
|
||||
→ systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
|
||||
WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
|
||||
→ modprobe nvidia_peermem
|
||||
OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3
|
||||
OK nvlink NVLink groups: [0 1 2 3 4 5 6 7]
|
||||
OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
|
||||
OK fabric subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
|
||||
OK agent fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
|
||||
OK agent NCCL bootstraps over ens3 (10.1.0.21)
|
||||
```
|
||||
|
||||
followed by each GPU's NVLink peers and nearest adapter, each port's state,
|
||||
rate and errors, and the NCCL environment a shard on each fabric would be given.
|
||||
`--json` writes every fact it read as well as what it concluded — send that
|
||||
file when the report is not enough.
|
||||
|
||||
## Watching them
|
||||
|
||||
- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
|
||||
a table of subnet managers per fabric — master, priority, standbys, the last
|
||||
change of master — and every disagreement: one name on two fabrics, two names
|
||||
on one, an adapter up but undeclared, ports up with no subnet manager, a
|
||||
fabric with no standby, a recent failover.
|
||||
- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
|
||||
its nodes every five minutes: switches in their tiers, adapters under their
|
||||
leaves, cables coloured by rate, degraded ones in red, rail alignment, and
|
||||
the cluster's own adapters live with their traffic. Click anything for its
|
||||
detail.
|
||||
- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
|
||||
GPUs are joined, which node's IMEX is down, which link runs degraded.
|
||||
- **A node's page.** Its ports with state, rate, errors and what they are
|
||||
physically on; its GPU-by-GPU NVLink matrix; its load.
|
||||
- **Metrics**, for alerts:
|
||||
|
||||
| Metric | Alert when |
|
||||
|---|---|
|
||||
| `llmgw_ib_port_active` | a declared port is 0 |
|
||||
| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
|
||||
| `llmgw_ib_port_rate_gbps` | below its usual rate |
|
||||
| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
|
||||
| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
|
||||
| `llmgw_fabric_problems{level="fail"}` | above 0 |
|
||||
| `llmgw_gpu_nvlink_joined` | 0 |
|
||||
| `llmgw_imex_up` | 0 |
|
||||
|
||||
## What an engine is given
|
||||
|
||||
For a shard on an InfiniBand or RoCE fabric:
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
|
||||
| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
|
||||
| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
|
||||
| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
|
||||
| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
|
||||
|
||||
A profile's `env` overrides any of them. vLLM launches a multi-node replica
|
||||
with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
|
||||
every node runs `vllm serve` with its rank and the leader's address, the
|
||||
workers headless. `launch_mode: ray` is there for a vLLM too old for that.
|
||||
|
||||
## When a replica spanning nodes will not run
|
||||
|
||||
| It | Look at |
|
||||
|---|---|
|
||||
| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
|
||||
| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
|
||||
| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
|
||||
| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |
|
||||
Reference in new issue
Block a user