Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors
Every minute the agent reads nvidia-smi's full report, its NVLink error counters and the kernel's XID lines. They are published per card, shown on the node page and the GPU map, charted in Grafana, and five alerts watch them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ff554c7062
commit
82a63cfc6e
16 files changed
+885
-11
No files matched your search
@@ -55,7 +55,7 @@
|
||||
<div class="gpu-map">
|
||||
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
|
||||
{{end}}</div>
|
||||
<p class="note">{{.Legend}} The figure is the one coloured by; the rest of a card is under the pointer.</p>
|
||||
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
|
||||
{{end}}
|
||||
{{with .Efficiency}}
|
||||
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
|
||||
|
||||
Reference in new issue
Block a user