How much of each GPU works, beside whether anything does

The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.

Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-31 20:11:26 +01:00
1 parent 7bfe70c8df
commit c866b3a8ee
34 files changed
+954 -52

No files matched your search

+5 -1
View File
@@ -77,7 +77,11 @@
"node.chart.engine_memory": "Engine memory, by model",
"node.chart.engine_memory_note": "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use.",
"node.chart.gpu_busy": "GPU busy, by card",
"node.chart.gpu_busy_note": "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking.",
"node.chart.gpu_busy_note": "Of each sample period, the share a kernel was running — not how much of the chip it used: one SM busy reads 100%. A card held by a replica and idle here is one nobody is asking.",
"node.chart.gpu_power_load": "GPU power load, by card",
"node.chart.gpu_power_load_note": "Draw against the card's power limit. Busy says a kernel ran; this rises with how much of the chip it ran on.",
"node.chart.gpu_sm_active": "SMs active, by card",
"node.chart.gpu_sm_active_note": "The share of time the card's SMs had work, from its profiling counters. When busy reads 90 and this 60, a third of the chip sits idle while a kernel runs.",
"node.chart.gpu_memory": "GPU memory used, by card",
"node.chart.gpu_memory_note": "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's.",
"node.chart.gpu_temp": "GPU temperature, °C",