Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors

Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 16:02:07 +02:00
1 parent ff554c7062
commit 82a63cfc6e
16 files changed
+885 -11

No files matched your search

+2 -2
View File
@@ -35,7 +35,7 @@
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
{{with .Load}}{{if .GPUs}}
<h2>GPUs</h2>
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by</th></tr>
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
{{range .GPUs}}<tr>
<td class="num">{{.Index}}</td>
{{/* From the last sample; a dash where the driver or an older agent did
@@ -46,7 +46,7 @@
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}</td>
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
</tr>{{end}}</table>
{{end}}{{end}}
{{end}}