Document what changed since v0.9.2, and fabrics for the sites that have them
The changelog's unreleased section had stopped at refusing unloadable weights; it now covers the rest: telling independent fabrics apart, their subnet managers and maps, NVLink on every build and across nodes, the multi-node launch fixes and vLLM's new default, doctor, node load, editors and the Responses API, the chat and mirror pages, a person's own use, the admin pages, the new metrics, and how to upgrade. docs/fabrics.md is new, for a site with InfiniBand or NVLink: what llmgw does with them, what each node and fabric needs first, declaring fabrics, checking a node with doctor, what the pages and metrics show and which to alert on, what an engine is given, and where to look when a replica spanning nodes will not run. It says plainly that none of it has yet run on such hardware. install.md, node-onboarding.md, onboarding.md, the README and the example agent configuration no longer say the portable binary cannot see NVLink or that a fabric's name cannot be checked, and describe the account page and connecting an editor. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f989e67e1a
commit
80e0dd252f
10 files changed
+440
-67
No files matched your search
+33
-36
@@ -318,32 +318,26 @@ hardware to it.
|
||||
|
||||
### NVLink, if your GPUs have it
|
||||
|
||||
The published binary cannot see NVLink topology. NVML is the only source of it,
|
||||
its bindings need cgo, and the portable binary is built without cgo on purpose
|
||||
so it runs on a node whose libc is older than the builder's.
|
||||
The agent reads the topology from `nvidia-smi topo -m`: which cards NVLink
|
||||
joins, each card's NUMA node, and which network adapter is nearest each. A
|
||||
tensor-parallel group is then the set of cards with the most NVLink between
|
||||
them — on a PCIe box where bridges pair specific cards, a group straddling two
|
||||
bridges falls back to PCIe and runs several times slower, and nothing else
|
||||
would report it.
|
||||
|
||||
Without that topology, a tensor-parallel group takes the lowest free GPU
|
||||
indices. On an SXM board with NVSwitch that is correct, because every card is
|
||||
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
|
||||
straddling two bridges falls back to PCIe and runs several times slower — and
|
||||
nothing reports it, because from the outside it is simply a replica that is
|
||||
slow.
|
||||
|
||||
For those nodes, install the `-nvml` build published beside the portable one:
|
||||
On an NVSwitch machine, `nvidia-fabricmanager` has to be running, at the
|
||||
driver's version, or CUDA will not start on its GPUs. On an NVL72 rack or an
|
||||
NVLink Switch system, the GPUs of several nodes form one domain, and a replica
|
||||
can run tensor parallelism across its nodes — which needs `nvidia-imex` and an
|
||||
IMEX channel as well. [fabrics.md](fabrics.md) has the list, and `llmgw doctor`
|
||||
checks it:
|
||||
|
||||
```
|
||||
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
|
||||
sudo llmgw doctor
|
||||
```
|
||||
|
||||
It is dynamically linked, so it needs a libc no older than the builder's and the
|
||||
NVIDIA driver present. Check it took effect with:
|
||||
|
||||
```
|
||||
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
|
||||
```
|
||||
|
||||
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
|
||||
flat zero everywhere means the topology is unknown rather than absent.
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
@@ -353,17 +347,17 @@ ibstat # must list your HCAs with State: Active
|
||||
ls /sys/class/infiniband
|
||||
```
|
||||
|
||||
The agent checks what you declare against what the node has. An adapter that is
|
||||
not there, or that has no ACTIVE port — physically linked but with no subnet
|
||||
manager, which survives a reboot and looks like a software problem — takes that
|
||||
fabric out of service on this node, loudly, in the log. Without that check the
|
||||
replica starts, passes its health checks, and hangs on its first all-reduce.
|
||||
Each fabric needs a subnet manager, and should have a standby; `nvidia_peermem`
|
||||
should be loaded for GPUDirect RDMA. You name each fabric when you onboard the
|
||||
node, and llmgw checks the name against what the adapters report: every port on
|
||||
one fabric sees the same subnet managers, so two independent fabrics given one
|
||||
name are placed as two, and the cluster page says so. An adapter that is not
|
||||
there, or whose port goes down, takes that fabric off the node within seconds,
|
||||
and gives it back when the port returns.
|
||||
|
||||
Fabric membership is configuration, not discovery, and there is no way around
|
||||
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
|
||||
prefix proves nothing and a wrong guess produces a replica that starts, passes
|
||||
health checks, and then hangs on its first all-reduce. You name the fabric when
|
||||
you onboard the node, and two nodes are only in the same fabric if you said so.
|
||||
[fabrics.md](fabrics.md) covers declaring several fabrics, what the cluster and
|
||||
fabric pages show, and what to look at when a replica spanning nodes will not
|
||||
run. Run `llmgw doctor` on the node before it joins.
|
||||
|
||||
### An engine
|
||||
|
||||
@@ -1048,6 +1042,8 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
and what belongs on which port
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/fabrics.md` — InfiniBand fabrics and NVLink: what llmgw needs of them,
|
||||
checking a node, and what to look at when a replica spanning nodes will not run
|
||||
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
|
||||
deliberately does not do
|
||||
- `docs/onboarding.md` — giving people accounts and keys
|
||||
@@ -1069,8 +1065,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
well-trodden path.
|
||||
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
|
||||
the fabric invariant and the rendezvous are exercised against simulated
|
||||
hardware only. Single-node placement on real GPUs is the well-trodden path;
|
||||
if you are the first to run a TP group across real cards, expect to find
|
||||
something.
|
||||
- **Multi-node replicas have not been run on real InfiniBand or NVLink.** The
|
||||
scheduling, the fabric identity, the maps and the launch are written against
|
||||
the real tools' formats and tested against them, not yet run on hardware that
|
||||
has them. Single-node placement on real GPUs is the well-trodden path; if you
|
||||
are the first to run a TP group across real cards, run `llmgw doctor --json`
|
||||
on each node first and keep the reports.
|
||||
Reference in new issue
Block a user