How much of each GPU works, beside whether anything does
CI / test (push) Failing after 6m42s

The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.

Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-31 20:11:26 +01:00
1 parent 7bfe70c8df
commit c866b3a8ee
34 files changed
+954 -52

No files matched your search

+46 -1
View File
@@ -504,7 +504,7 @@
{
"type": "timeseries",
"title": "GPU busy, by card",
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
"description": "Share of each sample period in which any kernel ran (nvidia-smi's utilization.gpu), sampled every heartbeat. Not how much of the chip: one SM busy reads 100%. The SMs panel and power against its limit say how much.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
@@ -908,6 +908,51 @@
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU SMs and tensor cores active, by card",
"description": "From the cards' profiling counters: NVML's GPM on Hopper and later, or a dcgm-exporter on the node. SMs active is the share of time the SMs had work, averaged over all of them; the gap to busy is chip a running kernel leaves idle. Tensor cores stay low in decode at batch one; prefill and batches are where they rise. Empty for cards nothing can profile.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 73,
"w": 24,
"h": 7
},
"id": 25,
"targets": [
{
"expr": "llmgw_gpu_sm_active_percent",
"legendFormat": "{{node}} gpu {{gpu}} SMs",
"refId": "A"
},
{
"expr": "llmgw_gpu_tensor_active_percent",
"legendFormat": "{{node}} gpu {{gpu}} tensor",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
}
]
}