Document what changed since v0.9.2, and fabrics for the sites that have them

The changelog's unreleased section had stopped at refusing unloadable
weights; it now covers the rest: telling independent fabrics apart,
their subnet managers and maps, NVLink on every build and across nodes,
the multi-node launch fixes and vLLM's new default, doctor, node load,
editors and the Responses API, the chat and mirror pages, a person's own
use, the admin pages, the new metrics, and how to upgrade.

docs/fabrics.md is new, for a site with InfiniBand or NVLink: what
llmgw does with them, what each node and fabric needs first, declaring
fabrics, checking a node with doctor, what the pages and metrics show
and which to alert on, what an engine is given, and where to look when
a replica spanning nodes will not run. It says plainly that none of it
has yet run on such hardware.

install.md, node-onboarding.md, onboarding.md, the README and the
example agent configuration no longer say the portable binary cannot
see NVLink or that a fabric's name cannot be checked, and describe the
account page and connecting an editor.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 10:14:58 +02:00
1 parent f989e67e1a
commit 80e0dd252f
10 files changed
+440 -67

No files matched your search

+33 -36
View File
@@ -318,32 +318,26 @@ hardware to it.
### NVLink, if your GPUs have it
The published binary cannot see NVLink topology. NVML is the only source of it,
its bindings need cgo, and the portable binary is built without cgo on purpose
so it runs on a node whose libc is older than the builder's.
The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
joins, each card's NUMA node, and which network adapter is nearest each. A
tensor-parallel group is then the set of cards with the most NVLink between
them — on a PCIe box where bridges pair specific cards, a group straddling two
bridges falls back to PCIe and runs several times slower, and nothing else
would report it.
Without that topology, a tensor-parallel group takes the lowest free GPU
indices. On an SXM board with NVSwitch that is correct, because every card is
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
straddling two bridges falls back to PCIe and runs several times slower — and
nothing reports it, because from the outside it is simply a replica that is
slow.
For those nodes, install the `-nvml` build published beside the portable one:
On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
checks it:
```
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
sudo llmgw doctor
```
It is dynamically linked, so it needs a libc no older than the builder's and the
NVIDIA driver present. Check it took effect with:
```
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
```
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
flat zero everywhere means the topology is unknown rather than absent.
A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.
### InfiniBand, if you have it
@@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active
ls /sys/class/infiniband
```
The agent checks what you declare against what the node has. An adapter that is
not there, or that has no ACTIVE port — physically linked but with no subnet
manager, which survives a reboot and looks like a software problem — takes that
fabric out of service on this node, loudly, in the log. Without that check the
replica starts, passes its health checks, and hangs on its first all-reduce.
Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
node, and llmgw checks the name against what the adapters report: every port on
one fabric sees the same subnet managers, so two independent fabrics given one
name are placed as two, and the cluster page says so. An adapter that is not
there, or whose port goes down, takes that fabric off the node within seconds,
and gives it back when the port returns.
Fabric membership is configuration, not discovery, and there is no way around
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
prefix proves nothing and a wrong guess produces a replica that starts, passes
health checks, and then hangs on its first all-reduce. You name the fabric when
you onboard the node, and two nodes are only in the same fabric if you said so.
[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
fabric pages show, and what to look at when a replica spanning nodes will not
run. Run `llmgw doctor` on the node before it joins.
### An engine
@@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher.
and what belongs on which port
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
each one assumes about this system
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
checking a node, and what to look at when a replica spanning nodes will not run
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
deliberately does not do
- `docs/onboarding.md` — giving people accounts and keys
@@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher.
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
well-trodden path.
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
the fabric invariant and the rendezvous are exercised against simulated
hardware only. Single-node placement on real GPUs is the well-trodden path;
if you are the first to run a TP group across real cards, expect to find
something.
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
scheduling, the fabric identity, the maps and the launch are written against
the real tools' formats and tested against them, not yet run on hardware that
has them. Single-node placement on real GPUs is the well-trodden path; if you
are the first to run a TP group across real cards, run `llmgw doctor --json`
on each node first and keep the reports.