A card in the wrong PCIe slot is said so
At the user's asking: we know what a card's link could be, so a card linked slower is reported. Each agent reads, apart from the GPU query so that a driver not knowing a field costs nothing else, the fastest PCIe generation card and slot allow together, the card's own, and the lanes in use and the card's most (nvidia-smi; NVML in the nvml build). The current generation is not compared: it drops while a card idles. A card on fewer lanes, or a slot of an older generation, is a node warning with the GB/s it carries against what it could — on the dashboard, the console and in llmgw doctor, which points at lspci's LnkCap and LnkSta. MIG instances are said of their card. GPU proto fields 23-26. The PoC's Quadro RTX 6000 reads Gen3 of Gen3, x16 of x16, idle at Gen1: nothing to say. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6a870a8c15
commit
721705b29d
13 files changed
+279
-3
No files matched your search
@@ -1053,6 +1053,27 @@ A node with no location is placed as before, as a rack of its own.
|
||||
See [facility-aware-placement.md](facility-aware-placement.md) for power caps
|
||||
and holds.
|
||||
|
||||
### A card in the wrong slot
|
||||
|
||||
An x16 card in a slot wired x8 or x4, behind a riser, or in a slot of an
|
||||
older PCIe generation than the card's, links slower than it was built to and
|
||||
works otherwise. A model on the card still decodes at the card's own memory
|
||||
speed, but it loads slower, a replica split over several cards by PCIe is
|
||||
slower, and so is anything that spills to the host. Each card's agent reads
|
||||
how its link stands — the fastest generation card and slot allow together,
|
||||
and the lanes in use — against what the card could do, and the node says so
|
||||
among its warnings (the dashboard, the console's Nodes, `llmgw doctor`):
|
||||
|
||||
```
|
||||
GPU 1 (NVIDIA H100 80GB HBM3) linked at x8 of its x16 lanes: 31.5 GB/s where it
|
||||
could have 63.0. Check the slot (wired fewer lanes, or an older generation), a
|
||||
riser, or the BIOS's PCIe setting
|
||||
```
|
||||
|
||||
The generation a card reports now drops while it idles, to save power; what is
|
||||
compared is the fastest it may reach, which does not. `lspci -vv -s BUS-ID`
|
||||
shows the same: `LnkCap` is the card's, `LnkSta` the link's.
|
||||
|
||||
## 6. Setting the node's CPU model, if it is a VM
|
||||
|
||||
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
||||
|
||||
Reference in new issue
Block a user