Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors

Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 16:02:07 +02:00
1 parent ff554c7062
commit 82a63cfc6e
16 files changed
+885 -11

No files matched your search

@@ -855,6 +855,59 @@
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU wear",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"id": 24,
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
"gridPos": {
"x": 0,
"y": 66,
"w": 24,
"h": 7
},
"targets": [
{
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
"legendFormat": "XID {{node}} gpu {{gpu}}",
"refId": "A"
},
{
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
"refId": "B"
},
{
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
"refId": "C"
},
{
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
"refId": "D"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
}
]
}
+54
View File
@@ -224,6 +224,60 @@ groups:
card may be computing on corrupted memory. Drain the node
(llmgw ctl drain) and have the card looked at.
- alert: GPUXidError
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
description: >-
The driver reported an error on the card (its code is
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
94/95 contained and uncontained ECC errors: those need the card
looked at. Others are often an engine's own fault, and pass.
- alert: GPUOutOfSpareRows
expr: llmgw_gpu_remap_failure == 1
for: 1m
labels: { severity: critical }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
description: >-
The card cannot remap another failing row. Drain the node and
have the card replaced.
- alert: GPUNeedsReset
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
description: >-
Rows remapped or pages retired take effect only after the card is
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
it back.
- alert: GPUPCIeRetriesClimbing
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
description: >-
The card's link to the host is retrying. A riser, the slot or a
cable is the usual cause, and transfers to the card slow down.
- alert: GPUNVLinkErrorsClimbing
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
description: >-
Errors on the card's NVLink links: tensor-parallel replicas on it
slow down, and a link that keeps failing takes a replica with it.
- alert: GPUHot
expr: llmgw_gpu_temperature_celsius > 87
for: 10m