Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable weights; it now covers the rest: telling independent fabrics apart, their subnet managers and maps, NVLink on every build and across nodes, the multi-node launch fixes and vLLM's new default, doctor, node load, editors and the Responses API, the chat and mirror pages, a person's own use, the admin pages, the new metrics, and how to upgrade. docs/fabrics.md is new, for a site with InfiniBand or NVLink: what llmgw does with them, what each node and fabric needs first, declaring fabrics, checking a node with doctor, what the pages and metrics show and which to alert on, what an engine is given, and where to look when a replica spanning nodes will not run. It says plainly that none of it has yet run on such hardware. install.md, node-onboarding.md, onboarding.md, the README and the example agent configuration no longer say the portable binary cannot see NVLink or that a fabric's name cannot be checked, and describe the account page and connecting an editor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f989e67e1a
commit
80e0dd252f
10 files changed
+440
-67
No files matched your search
+177
@@ -0,0 +1,177 @@
|
||||
# InfiniBand fabrics and NVLink
|
||||
|
||||
For a site whose GPU nodes are joined by InfiniBand or RoCE, by NVLink, or both
|
||||
— several independent fabrics, NVL72 racks, eight-GPU HGX boards. It says what
|
||||
llmgw does with them, what it needs from you before it can, how to check a node
|
||||
in one command, and where to look when a replica spanning nodes will not run.
|
||||
|
||||
Read [install.md](install.md) first; this is the part of it that only matters
|
||||
once there is a fabric.
|
||||
|
||||
> **Where this stands.** Every piece here was written against the real formats
|
||||
> of `sminfo`, `saquery`, `ibnetdiscover`, `nvidia-smi topo -m` and
|
||||
> `nvidia-smi -q`, and tested against them — but none of it has yet run on a
|
||||
> site with the hardware. Run `llmgw doctor --json` on two or three nodes first
|
||||
> and keep the reports: if a parser meets a format it does not know, that report
|
||||
> is what fixes it, and the fix is a control plane upgrade, not one per node.
|
||||
|
||||
## What llmgw does with them
|
||||
|
||||
**Two kinds of interconnect.** InfiniBand or RoCE joins nodes (scale-out).
|
||||
NVLink joins GPUs — inside one machine on an HGX board, or across the nodes of a
|
||||
rack on an NVL72 or an NVLink Switch system (scale-up). A replica spanning nodes
|
||||
is placed, in order of preference:
|
||||
|
||||
1. inside one **multi-node NVLink domain**, if any has enough usable nodes:
|
||||
tensor parallelism between its nodes then runs over NVLink;
|
||||
2. on one **physical InfiniBand or RoCE fabric**, preferring nodes under one
|
||||
leaf switch and putting nodes with a degraded cable last.
|
||||
|
||||
It is never split across two fabrics or two NVLink domains. Within a node, the
|
||||
GPUs a shard gets are the set with the most NVLink pairs, then the nearest to
|
||||
the fabric's adapters, then on one NUMA node.
|
||||
|
||||
**A fabric has a name and an identity.** The name is what `agent.yaml` says; the
|
||||
identity is what the adapters report — on InfiniBand the subnet managers every
|
||||
port on one fabric sees, on RoCE the IP network of its RoCE v2 address. The name
|
||||
is how you address a fabric; the identity is how llmgw catches the name being
|
||||
wrong: two independent fabrics given one name are placed as two, and the
|
||||
cluster page says so.
|
||||
|
||||
## Before a node joins
|
||||
|
||||
On every GPU node:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| NVIDIA driver | everything | `nvidia-smi` |
|
||||
| `nvidia-fabricmanager`, on an NVSwitch machine | without it CUDA does not start on those GPUs | `systemctl is-active nvidia-fabricmanager` |
|
||||
| `rdma-core`, `ib_uverbs` loaded | user-space RDMA at all | `ls /sys/class/infiniband`, `lsmod \| grep ib_uverbs` |
|
||||
| `nvidia_peermem` loaded | GPUDirect RDMA; without it NCCL copies every transfer through host memory | `lsmod \| grep nvidia_peermem` |
|
||||
| `infiniband-diags` | tells llmgw which fabric a port is on; the agent installs it if missing | `which sminfo` |
|
||||
|
||||
On an NVL72 or NVLink Switch system, also:
|
||||
|
||||
| What | Why | Check |
|
||||
|---|---|---|
|
||||
| `nvidia-imex` running, every node of the domain in `/etc/nvidia-imex/nodes_config.cfg` | GPUs in different nodes share memory through it | `systemctl is-active nvidia-imex` |
|
||||
| an IMEX channel (`options nvidia NVreg_CreateImexChannel0=1`) | an engine opens one | `ls /dev/nvidia-caps-imex-channels` |
|
||||
| every GPU joined to its domain | the fabric manager does this | `nvidia-smi -q \| grep -A6 Fabric` shows `State: Completed` |
|
||||
|
||||
On each InfiniBand fabric:
|
||||
|
||||
- **A subnet manager, and a standby.** Without one, ports come up to `INIT` and
|
||||
stay there; without a standby, the fabric keeps its routes if the master
|
||||
stops but cannot recover from any change after. llmgw watches this but does
|
||||
not run one.
|
||||
- **Its own name** in `agent.yaml` on every node attached to it. Optionally, a
|
||||
distinct `subnet_prefix` in each subnet manager's configuration, which makes
|
||||
GIDs unique across fabrics; llmgw does not need it.
|
||||
|
||||
## Declaring the fabrics
|
||||
|
||||
In `/etc/llmgw/agent.yaml`, the adapters on each fabric, by name:
|
||||
|
||||
```yaml
|
||||
fabrics:
|
||||
- name: ib-compute
|
||||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3, mlx5_6, mlx5_7, mlx5_8, mlx5_9]
|
||||
- name: ib-storage
|
||||
hcas: [mlx5_4]
|
||||
```
|
||||
|
||||
or `--fabric ib-compute:mlx5_0,mlx5_1,…` when onboarding ([node-onboarding.md](node-onboarding.md)).
|
||||
An adapter belongs to one fabric; a node may be on several. A replica on
|
||||
`ib-compute` is given only `ib-compute`'s adapters (`NCCL_IB_HCA`).
|
||||
|
||||
The rate is measured, not taken from the file: a fabric's rate on a node is the
|
||||
slowest of its active adapters, because a collective runs at the speed of its
|
||||
slowest link.
|
||||
|
||||
The agent checks the declaration against the adapters on every heartbeat. An
|
||||
adapter that is missing, or whose port is down, takes that fabric off the node
|
||||
within seconds — the node stops being offered multi-node work on it — and
|
||||
returns it when the port comes back.
|
||||
|
||||
## Checking a node: `llmgw doctor`
|
||||
|
||||
```
|
||||
sudo llmgw doctor # uses /etc/llmgw/agent.yaml when present
|
||||
sudo llmgw doctor --json > $(hostname)-doctor.json
|
||||
```
|
||||
|
||||
It reads the node and says what is wrong and how to fix it, most serious first,
|
||||
and exits non-zero when anything fails:
|
||||
|
||||
```
|
||||
FAIL nvlink this machine has NVSwitch but nvidia-fabricmanager is inactive: CUDA will not initialise on its GPUs
|
||||
→ systemctl enable --now nvidia-fabricmanager (its version must match the driver's)
|
||||
WARN fabric nvidia_peermem is not loaded: GPUDirect RDMA is off, and NCCL copies every transfer through host memory
|
||||
→ modprobe nvidia_peermem
|
||||
OK gpu 8 GPUs: 8× NVIDIA H100 80GB HBM3
|
||||
OK nvlink NVLink groups: [0 1 2 3 4 5 6 7]
|
||||
OK fabric mlx5_0/1 ACTIVE at 400 Gb/s (InfiniBand), subnet manager 0x248a07030030f0c0 (LID 1, SMINFO_MASTER)
|
||||
OK fabric subnet managers of 0x248a07030030f0c0's fabric: 0x248a07030030f0c0 (master, priority 15), 0x248a07030030f1d4 (standby, priority 14)
|
||||
OK agent fabric ib-compute: mlx5_0,mlx5_1,… usable at 400 Gb/s
|
||||
OK agent NCCL bootstraps over ens3 (10.1.0.21)
|
||||
```
|
||||
|
||||
followed by each GPU's NVLink peers and nearest adapter, each port's state,
|
||||
rate and errors, and the NCCL environment a shard on each fabric would be given.
|
||||
`--json` writes every fact it read as well as what it concluded — send that
|
||||
file when the report is not enough.
|
||||
|
||||
## Watching them
|
||||
|
||||
- **Cluster page, Fabrics.** Each declared fabric and what it physically is;
|
||||
a table of subnet managers per fabric — master, priority, standbys, the last
|
||||
change of master — and every disagreement: one name on two fabrics, two names
|
||||
on one, an adapter up but undeclared, ports up with no subnet manager, a
|
||||
fabric with no standby, a recent failover.
|
||||
- **`/admin/fabrics`.** Each fabric's map, from `ibnetdiscover` run by one of
|
||||
its nodes every five minutes: switches in their tiers, adapters under their
|
||||
leaves, cables coloured by rate, degraded ones in red, rail alignment, and
|
||||
the cluster's own adapters live with their traffic. Click anything for its
|
||||
detail.
|
||||
- **`/admin/nvlink`.** Each multi-node NVLink domain as a rack of trays: which
|
||||
GPUs are joined, which node's IMEX is down, which link runs degraded.
|
||||
- **A node's page.** Its ports with state, rate, errors and what they are
|
||||
physically on; its GPU-by-GPU NVLink matrix; its load.
|
||||
- **Metrics**, for alerts:
|
||||
|
||||
| Metric | Alert when |
|
||||
|---|---|
|
||||
| `llmgw_ib_port_active` | a declared port is 0 |
|
||||
| `llmgw_ib_port_counter_total{counter="symbol_error"}` and `link_downed` | rising |
|
||||
| `llmgw_ib_port_rate_gbps` | below its usual rate |
|
||||
| `llmgw_fabric_subnet_managers{state="standby"}` | absent or 0 |
|
||||
| `llmgw_fabric_master_sm` | its `sm_guid` changes (a failover) |
|
||||
| `llmgw_fabric_problems{level="fail"}` | above 0 |
|
||||
| `llmgw_gpu_nvlink_joined` | 0 |
|
||||
| `llmgw_imex_up` | 0 |
|
||||
|
||||
## What an engine is given
|
||||
|
||||
For a shard on an InfiniBand or RoCE fabric:
|
||||
|
||||
| Variable | Value |
|
||||
|---|---|
|
||||
| `NCCL_IB_HCA` | that fabric's adapters on this node, and no others |
|
||||
| `NCCL_IB_DISABLE` | `0` (`1` for a single-node shard) |
|
||||
| `NCCL_SOCKET_IFNAME`, `GLOO_SOCKET_IFNAME` | the node's management interface: `mgmt_interface`, or the one holding its management address |
|
||||
| `NCCL_IB_GID_INDEX` | on RoCE only: the RoCE v2 entry with an IPv4 address |
|
||||
| `NCCL_MNNVL_ENABLE`, `NCCL_CUMEM_ENABLE` | `1`, for a replica inside a multi-node NVLink domain |
|
||||
|
||||
A profile's `env` overrides any of them. vLLM launches a multi-node replica
|
||||
with its own multiprocessing (`engines.vllm.launch_mode: mp`, the default):
|
||||
every node runs `vllm serve` with its rank and the leader's address, the
|
||||
workers headless. `launch_mode: ray` is there for a vLLM too old for that.
|
||||
|
||||
## When a replica spanning nodes will not run
|
||||
|
||||
| It | Look at |
|
||||
|---|---|
|
||||
| is not placed | `llmgw ctl explain-placement <model>`: a fabric split by identity, a node whose IMEX is down, a node refused because its adapters are cabled to two fabrics |
|
||||
| starts and hangs on its first collective | the cluster page's fabric problems; `doctor` on each node — a port down, the wrong interface, RoCE without a v2 GID |
|
||||
| runs, slowly | `nvidia_peermem`; the map for a degraded cable; the node page's NVLink matrix for a group that crossed PCIe |
|
||||
| worked yesterday | the subnet manager table for a failover; `link_downed` on the node's ports |
|
||||
+33
-36
@@ -318,32 +318,26 @@ hardware to it.
|
||||
|
||||
### NVLink, if your GPUs have it
|
||||
|
||||
The published binary cannot see NVLink topology. NVML is the only source of it,
|
||||
its bindings need cgo, and the portable binary is built without cgo on purpose
|
||||
so it runs on a node whose libc is older than the builder's.
|
||||
The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
|
||||
joins, each card's NUMA node, and which network adapter is nearest each. A
|
||||
tensor-parallel group is then the set of cards with the most NVLink between
|
||||
them — on a PCIe box where bridges pair specific cards, a group straddling two
|
||||
bridges falls back to PCIe and runs several times slower, and nothing else
|
||||
would report it.
|
||||
|
||||
Without that topology, a tensor-parallel group takes the lowest free GPU
|
||||
indices. On an SXM board with NVSwitch that is correct, because every card is
|
||||
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
|
||||
straddling two bridges falls back to PCIe and runs several times slower — and
|
||||
nothing reports it, because from the outside it is simply a replica that is
|
||||
slow.
|
||||
|
||||
For those nodes, install the `-nvml` build published beside the portable one:
|
||||
On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
|
||||
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
|
||||
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
|
||||
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
|
||||
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
|
||||
checks it:
|
||||
|
||||
```
|
||||
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
|
||||
sudo llmgw doctor
|
||||
```
|
||||
|
||||
It is dynamically linked, so it needs a libc no older than the builder's and the
|
||||
NVIDIA driver present. Check it took effect with:
|
||||
|
||||
```
|
||||
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
|
||||
```
|
||||
|
||||
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
|
||||
flat zero everywhere means the topology is unknown rather than absent.
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
@@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active
|
||||
ls /sys/class/infiniband
|
||||
```
|
||||
|
||||
The agent checks what you declare against what the node has. An adapter that is
|
||||
not there, or that has no ACTIVE port — physically linked but with no subnet
|
||||
manager, which survives a reboot and looks like a software problem — takes that
|
||||
fabric out of service on this node, loudly, in the log. Without that check the
|
||||
replica starts, passes its health checks, and hangs on its first all-reduce.
|
||||
Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
|
||||
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
|
||||
node, and llmgw checks the name against what the adapters report: every port on
|
||||
one fabric sees the same subnet managers, so two independent fabrics given one
|
||||
name are placed as two, and the cluster page says so. An adapter that is not
|
||||
there, or whose port goes down, takes that fabric off the node within seconds,
|
||||
and gives it back when the port returns.
|
||||
|
||||
Fabric membership is configuration, not discovery, and there is no way around
|
||||
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
|
||||
prefix proves nothing and a wrong guess produces a replica that starts, passes
|
||||
health checks, and then hangs on its first all-reduce. You name the fabric when
|
||||
you onboard the node, and two nodes are only in the same fabric if you said so.
|
||||
[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
|
||||
fabric pages show, and what to look at when a replica spanning nodes will not
|
||||
run. Run `llmgw doctor` on the node before it joins.
|
||||
|
||||
### An engine
|
||||
|
||||
@@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
and what belongs on which port
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
checking a node, and what to look at when a replica spanning nodes will not run
|
||||
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
|
||||
deliberately does not do
|
||||
- `docs/onboarding.md` — giving people accounts and keys
|
||||
@@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
well-trodden path.
|
||||
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
|
||||
the fabric invariant and the rendezvous are exercised against simulated
|
||||
hardware only. Single-node placement on real GPUs is the well-trodden path;
|
||||
if you are the first to run a TP group across real cards, expect to find
|
||||
something.
|
||||
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
|
||||
scheduling, the fabric identity, the maps and the launch are written against
|
||||
the real tools' formats and tested against them, not yet run on hardware that
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
+12
-6
@@ -178,18 +178,24 @@ was cut short.
|
||||
|
||||
## Fabrics
|
||||
|
||||
`--fabric` is how a node says which InfiniBand fabric it is attached to, and
|
||||
there is no way around stating it. OpenSM's default subnet prefix is `fe80::` on
|
||||
every fabric, so a matching prefix proves nothing and a wrong guess produces a
|
||||
replica that starts, passes health checks, and then hangs on its first
|
||||
all-reduce.
|
||||
`--fabric` is how a node says which InfiniBand fabric it is attached to. The
|
||||
name is checked against what the adapters report — every port on one fabric
|
||||
sees the same subnet managers — so two independent fabrics given one name are
|
||||
placed as two, and the cluster page says so; give each fabric its own name.
|
||||
The agent installs `infiniband-diags`, which that check needs, if it is missing.
|
||||
|
||||
Before a node with a fabric joins, run `sudo llmgw doctor` on it: it checks the
|
||||
subnet manager, GPUDirect RDMA, the fabric manager and IMEX on NVLink machines,
|
||||
and the declared fabrics against the adapters. [fabrics.md](fabrics.md) has the
|
||||
rest.
|
||||
|
||||
Omit it for a node with no fabric. Such a node is its own island: it can host
|
||||
single-node replicas and will never be given a multi-node one.
|
||||
|
||||
Two adapters per fabric is the default; `--fabric ib-b:mlx5_2,mlx5_3` names them
|
||||
explicitly. An adapter must never appear under two fabrics — the agent refuses
|
||||
that configuration, because the consequence is the hang above.
|
||||
that configuration: a replica given adapters on two fabrics starts, passes its
|
||||
health checks, and hangs on its first all-reduce.
|
||||
|
||||
## Where the driver is
|
||||
|
||||
|
||||
+26
-3
@@ -47,9 +47,32 @@ llmgw users approve ayse@example.com --project Acme/ml/training --store … #
|
||||
```
|
||||
|
||||
That is the whole operator loop. From then on they sign in at `/login`, and
|
||||
`/profile` is where they create keys, see when each was last used, and revoke the
|
||||
ones they are done with. A key is shown once on the page that creates it and
|
||||
never again.
|
||||
their account page (`/profile`) is where they create keys, see when each was
|
||||
last used, revoke the ones they are done with, and see their own use over the
|
||||
last thirty days against every budget that applies to them. A key is shown once
|
||||
on the page that creates it and never again.
|
||||
|
||||
### Connecting an editor
|
||||
|
||||
The account page's **Connect your editor** makes a key for Continue, Codex or
|
||||
Cline and shows what the tool needs with that key, this gateway's address and a
|
||||
model already in it:
|
||||
|
||||
- **Continue** — an assistant file of its own, `~/.continue/assistants/llmgw.yaml`,
|
||||
so their `config.yaml` is not touched.
|
||||
- **Codex** — a provider and a profile in `~/.codex/config.toml`, run with
|
||||
`codex --profile llmgw`; their default model stays as it was. Codex speaks the
|
||||
Responses API, which the gateway translates at `/v1/responses`.
|
||||
- **Cline** — four settings to paste into its own screen, since it keeps them
|
||||
in VS Code's secret storage where no file reaches.
|
||||
|
||||
For Continue and Codex there is a one-line command,
|
||||
`curl -fsSL <gateway>/setup/<tool>?model=… | sh`. Its script is served without
|
||||
signing in and holds no secret — the key comes from the environment — and it
|
||||
keeps a copy of anything it replaces. Revoking the key disconnects the tool.
|
||||
|
||||
How well an editor's agent mode works depends on the model: it needs one that
|
||||
calls tools well, which the coder models do and small general ones often do not.
|
||||
|
||||
**This gateway sends no email.** There is no "your account was approved" message
|
||||
and no password reset, so tell people what to expect: they sign up, they tell
|
||||
|
||||
+3
-2
@@ -793,8 +793,9 @@ IBRoot is the sysfs InfiniBand directory.
|
||||
|
||||
`bool`
|
||||
|
||||
PreferNVML uses the NVML backend when this binary has one. It is the only
|
||||
source of NVLink topology.
|
||||
PreferNVML uses the NVML backend when this binary has one. NVLink
|
||||
topology is read from nvidia-smi topo -m on every build; NVML, where it
|
||||
is used, is the more exact source of the peers.
|
||||
|
||||
### `skip_gpus`
|
||||
|
||||
|
||||
Reference in new issue
Block a user