Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors

Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 16:02:07 +02:00
1 parent ff554c7062
commit 82a63cfc6e
16 files changed
+885 -11

No files matched your search

+1 -1
View File
@@ -55,7 +55,7 @@
<div class="gpu-map">
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
{{end}}</div>
<p class="note">{{.Legend}} The figure is the one coloured by; the rest of a card is under the pointer.</p>
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
{{end}}
{{with .Efficiency}}
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>