Document what changed since v0.9.2, and fabrics for the sites that have them

The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.

docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.

install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 10:14:58 +02:00
1 parent f989e67e1a
commit 80e0dd252f
10 files changed
+440 -67

No files matched your search

+177
View File
@@ -0,0 +1,177 @@
# InfiniBand fabrics and NVLink
For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
llmgw does with them, what it needs from you before it can, how to check a node
in one command, and where to look when a replica spanning nodes will not run.
Read [install.md](install.md) first; this is the part of it that only matters
once there is a fabric.
> **Where this stands.** Every piece here was written against the real formats
> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
> and keep the reports: if a parser meets a format it does not know, that report
> is what fixes it, and the fix is a control plane upgrade, not one per node.
## What llmgw does with them
**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
is placed, in order of preference:
1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
tensor parallelism between its nodes then runs over NVLink;
2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
leaf switch and putting nodes with a degraded cable last.
It is never split across two fabrics or two NVLink domains. Within a node, the
GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
the fabric's adapters, then on one NUMA node.
**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
identity is what the adapters report — on InfiniBand the subnet managers every
port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
is how you address a fabric; the identity is how llmgw catches the name being
wrong: two independent fabrics given one name are placed as two, and the
cluster page says so.
## Before a node joins
On every GPU node:
| What | Why | Check |
|---|---|---|
| NVIDIA driver | everything | `nvidia-smi` |
| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
On an NVL72 or NVLink Switch system, also:
| What | Why | Check |
|---|---|---|
| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
On each InfiniBand fabric:
- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
stay there; without a standby, the fabric keeps its routes if the master
stops but cannot recover from any change after. llmgw watches this but does
not run one.
- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
distinct `subnet_prefix` in each subnet manager's configuration, which makes
GIDs unique across fabrics; llmgw does not need it.
## Declaring the fabrics
In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
```yaml
fabrics:
- name: ib-compute
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
- name: ib-storage
hcas: [mlx5_4]
```
or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
An adapter belongs to one fabric; a node may be on several. A replica on
`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
The rate is measured, not taken from the file: a fabric's rate on a node is the
slowest of its active adapters, because a collective runs at the speed of its
slowest link.
The agent checks the declaration against the adapters on every heartbeat. An
adapter that is missing, or whose port is down, takes that fabric off the node
within seconds — the node stops being offered multi-node work on it — and
returns it when the port comes back.
## Checking a node: `llmgw doctor`
```
sudo llmgw doctor # uses /etc/llmgw/agent.yaml when present
sudo llmgw doctor --json > $(hostname)-doctor.json
```
It reads the node and says what is wrong and how to fix it, most serious first,
and exits non-zero when anything fails:
```
FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
→ systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
→ modprobe nvidia_peermem
OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3
OK nvlink NVLink groups: [0 1 2 3 4 5 6 7]
OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
OK fabric subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
OK agent fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
OK agent NCCL bootstraps over ens3 (10.1.0.21)
```
followed by each GPU's NVLink peers and nearest adapter, each port's state,
rate and errors, and the NCCL environment a shard on each fabric would be given.
`--json` writes every fact it read as well as what it concluded — send that
file when the report is not enough.
## Watching them
- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
a table of subnet managers per fabric — master, priority, standbys, the last
change of master — and every disagreement: one name on two fabrics, two names
on one, an adapter up but undeclared, ports up with no subnet manager, a
fabric with no standby, a recent failover.
- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
its nodes every five minutes: switches in their tiers, adapters under their
leaves, cables coloured by rate, degraded ones in red, rail alignment, and
the cluster's own adapters live with their traffic. Click anything for its
detail.
- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
GPUs are joined, which node's IMEX is down, which link runs degraded.
- **A node's page.** Its ports with state, rate, errors and what they are
physically on; its GPU-by-GPU NVLink matrix; its load.
- **Metrics**, for alerts:
| Metric | Alert when |
|---|---|
| `llmgw_ib_port_active` | a declared port is 0 |
| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
| `llmgw_ib_port_rate_gbps` | below its usual rate |
| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
| `llmgw_fabric_problems{level="fail"}` | above 0 |
| `llmgw_gpu_nvlink_joined` | 0 |
| `llmgw_imex_up` | 0 |
## What an engine is given
For a shard on an InfiniBand or RoCE fabric:
| Variable | Value |
|---|---|
| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
A profile's `env` overrides any of them. vLLM launches a multi-node replica
with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
every node runs `vllm serve` with its rank and the leader's address, the
workers headless. `launch_mode: ray` is there for a vLLM too old for that.
## When a replica spanning nodes will not run
| It | Look at |
|---|---|
| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |
+33 -36
View File
@@ -318,32 +318,26 @@ hardware to it.
### NVLink, if your GPUs have it
The published binary cannot see NVLink topology. NVML is the only source of it,
its bindings need cgo, and the portable binary is built without cgo on purpose
so it runs on a node whose libc is older than the builder's.
The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
joins, each card's NUMA node, and which network adapter is nearest each. A
tensor-parallel group is then the set of cards with the most NVLink between
them — on a PCIe box where bridges pair specific cards, a group straddling two
bridges falls back to PCIe and runs several times slower, and nothing else
would report it.
Without that topology, a tensor-parallel group takes the lowest free GPU
indices. On an SXM board with NVSwitch that is correct, because every card is
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
straddling two bridges falls back to PCIe and runs several times slower — and
nothing reports it, because from the outside it is simply a replica that is
slow.
For those nodes, install the `-nvml` build published beside the portable one:
On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
checks it:
```
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
sudo llmgw doctor
```
It is dynamically linked, so it needs a libc no older than the builder's and the
NVIDIA driver present. Check it took effect with:
```
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
```
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
flat zero everywhere means the topology is unknown rather than absent.
A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.
### InfiniBand, if you have it
@@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active
ls /sys/class/infiniband
```
The agent checks what you declare against what the node has. An adapter that is
not there, or that has no ACTIVE port — physically linked but with no subnet
manager, which survives a reboot and looks like a software problem — takes that
fabric out of service on this node, loudly, in the log. Without that check the
replica starts, passes its health checks, and hangs on its first all-reduce.
Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
node, and llmgw checks the name against what the adapters report: every port on
one fabric sees the same subnet managers, so two independent fabrics given one
name are placed as two, and the cluster page says so. An adapter that is not
there, or whose port goes down, takes that fabric off the node within seconds,
and gives it back when the port returns.
Fabric membership is configuration, not discovery, and there is no way around
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
prefix proves nothing and a wrong guess produces a replica that starts, passes
health checks, and then hangs on its first all-reduce. You name the fabric when
you onboard the node, and two nodes are only in the same fabric if you said so.
[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
fabric pages show, and what to look at when a replica spanning nodes will not
run. Run `llmgw doctor` on the node before it joins.
### An engine
@@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher.
and what belongs on which port
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
checking a node, and what to look at when a replica spanning nodes will not run
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
deliberately does not do
- `docs/onboarding.md` — giving people accounts and keys
@@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher.
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
well-trodden path.
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
the fabric invariant and the rendezvous are exercised against simulated
hardware only. Single-node placement on real GPUs is the well-trodden path;
if you are the first to run a TP group across real cards, expect to find
something.
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
scheduling, the fabric identity, the maps and the launch are written against
the real tools' formats and tested against them, not yet run on hardware that
has them. Single-node placement on real GPUs is the well-trodden path; if you
are the first to run a TP group across real cards, run `llmgw doctor --json`
on each node first and keep the reports.
+12 -6
View File
@@ -178,18 +178,24 @@ was cut short.
## Fabrics
`--fabric` is how a node says which InfiniBand fabric it is attached to, and
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
every fabric, so a matching prefix proves nothing and a wrong guess produces a
replica that starts, passes health checks, and then hangs on its first
all-reduce.
`--fabric` is how a node says which InfiniBand fabric it is attached to. The
name is checked against what the adapters report — every port on one fabric
sees the same subnet managers — so two independent fabrics given one name are
placed as two, and the cluster page says so; give each fabric its own name.
The agent installs `infiniband-diags`, which that check needs, if it is missing.
Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
rest.
Omit it for a node with no fabric. Such a node is its own island: it can host
single-node replicas and will never be given a multi-node one.
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
explicitly. An adapter must never appear under two fabrics — the agent refuses
that configuration, because the consequence is the hang above.
that configuration: a replica given adapters on two fabrics starts, passes its
health checks, and hangs on its first all-reduce.
## Where the driver is
+26 -3
View File
@@ -47,9 +47,32 @@ llmgw users approve ayse@example.com --project Acme/ml/training --store … #
```
That is the whole operator loop. From then on they sign in at `/login`, and
`/profile` is where they create keys, see when each was last used, and revoke the
ones they are done with. A key is shown once on the page that creates it and
never again.
their account page (`/profile`) is where they create keys, see when each was
last used, revoke the ones they are done with, and see their own use over the
last thirty days against every budget that applies to them. A key is shown once
on the page that creates it and never again.
### Connecting an editor
The account page's **Connect your editor** makes a key for Continue, Codex or
Cline and shows what the tool needs with that key, this gateway's address and a
model already in it:
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
so their `config.yaml` is not touched.
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
`codex --profile llmgw`; their default model stays as it was. Codex speaks the
Responses API, which the gateway translates at `/v1/responses`.
- **Cline** — four settings to paste into its own screen, since it keeps them
in VS Code's secret storage where no file reaches.
For Continue and Codex there is a one-line command,
`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
signing in and holds no secret — the key comes from the environment — and it
keeps a copy of anything it replaces. Revoking the key disconnects the tool.
How well an editor's agent mode works depends on the model: it needs one that
calls tools well, which the coder models do and small general ones often do not.
**This gateway sends no email.** There is no "your account was approved" message
and no password reset, so tell people what to expect: they sign up, they tell
+3 -2
View File
@@ -793,8 +793,9 @@ IBRoot is the sysfs InfiniBand directory.
`bool`
PreferNVML uses the NVML backend when this binary has one. It is the only
source of NVLink topology.
PreferNVML uses the NVML backend when this binary has one. NVLink
topology is read from nvidia-smi topo -m on every build; NVML, where it
is used, is the more exact source of the peers.
### `skip_gpus`