A card in the wrong PCIe slot is said so
CI / test (push) Successful in 9m17s

At the user's asking: we know what a card's link could be, so a card
linked slower is reported. Each agent reads, apart from the GPU query so
that a driver not knowing a field costs nothing else, the fastest PCIe
generation card and slot allow together, the card's own, and the lanes in
use and the card's most (nvidia-smi; NVML in the nvml build). The current
generation is not compared: it drops while a card idles. A card on fewer
lanes, or a slot of an older generation, is a node warning with the GB/s
it carries against what it could — on the dashboard, the console and in
llmgw doctor, which points at lspci's LnkCap and LnkSta. MIG instances are
said of their card. GPU proto fields 23-26.

The PoC's Quadro RTX 6000 reads Gen3 of Gen3, x16 of x16, idle at Gen1:
nothing to say.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 08:49:16 +01:00
1 parent 6a870a8c15
commit 721705b29d
13 files changed
+279 -3

No files matched your search

+8
View File
@@ -707,6 +707,14 @@ func (a *API) Nodes() []NodeView {
}
}
v.MemBandwidthGBs = n.Host.MemBandwidthGBs
for _, g := range n.GPUs {
if g.MIG != nil {
continue // said of the card it is carved from
}
if p := g.PCIe.Problem(); p != "" {
v.Warnings = append(v.Warnings, fmt.Sprintf("GPU %d (%s) %s", g.Index, g.Name, p))
}
}
if h := n.Host; h.OSID != "" {
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
+14
View File
@@ -735,3 +735,17 @@ func TestRemovingANodeRevokesItsCertificate(t *testing.T) {
t.Error("the node was removed though its certificate was not revoked")
}
}
func TestANodesCardInTheWrongSlotIsAWarning(t *testing.T) {
n := gpuNode("g1", 2)
n.GPUs[0].PCIe = discovery.PCIeLink{Gen: 5, GPUGen: 5, Width: 16, WidthMax: 16}
n.GPUs[1].PCIe = discovery.PCIeLink{Gen: 5, GPUGen: 5, Width: 8, WidthMax: 16}
api, _ := setup(t, testModels(), n)
var warned []string
for _, v := range api.Nodes() {
warned = append(warned, v.Warnings...)
}
if len(warned) != 1 || !strings.Contains(warned[0], "GPU 1 (NVIDIA H100 80GB HBM3) linked at x8 of its x16 lanes") {
t.Errorf("warnings %q", warned)
}
}