Stop trusting hardware facts that were never checked
CI / test (push) Successful in 1m32s

Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.

A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.

An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.

Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.

And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.

A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 17:29:15 +02:00
1 parent 3a4cc0c719
commit b619546516
7 files changed
+426 -2

No files matched your search

+35
View File
@@ -222,6 +222,35 @@ A node with no GPU at all is fine and is not an error. It reports zero cards and
can host CPU profiles, which is a good way to try llmgw before committing
hardware to it.
### NVLink, if your GPUs have it
The published binary cannot see NVLink topology. NVML is the only source of it,
its bindings need cgo, and the portable binary is built without cgo on purpose
so it runs on a node whose libc is older than the builder's.
Without that topology, a tensor-parallel group takes the lowest free GPU
indices. On an SXM board with NVSwitch that is correct, because every card is
equidistant. On a PCIe box where NVLink bridges pair specific cards, a group
straddling two bridges falls back to PCIe and runs several times slower — and
nothing reports it, because from the outside it is simply a replica that is
slow.
For those nodes, install the `-nvml` build published beside the portable one:
```
sudo install -m0755 llmgw-$VERSION-linux-amd64-nvml /usr/local/bin/llmgw
```
It is dynamically linked, so it needs a libc no older than the builder's and the
NVIDIA driver present. Check it took effect with:
```
llmgw ctl nodes --json | grep -c nvlink_peers # 0 means the topology is unknown
```
`llmgw_gpu_nvlink_peers` in the metrics says the same thing across a fleet: a
flat zero everywhere means the topology is unknown rather than absent.
### InfiniBand, if you have it
```
@@ -230,6 +259,12 @@ ibstat # must list your HCAs with State: Active
ls /sys/class/infiniband
```
The agent checks what you declare against what the node has. An adapter that is
not there, or that has no ACTIVE port — physically linked but with no subnet
manager, which survives a reboot and looks like a software problem — takes that
fabric out of service on this node, loudly, in the log. Without that check the
replica starts, passes its health checks, and hangs on its first all-reduce.
Fabric membership is configuration, not discovery, and there is no way around
that: OpenSM's default subnet prefix is the same on every fabric, so a matching
prefix proves nothing and a wrong guess produces a replica that starts, passes