The busy figure is nvidia-smi's utilization.gpu: the share of time any kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy while the SMs had work 69% of the time and issued on 18% of their cycles. Busy is now said to be that, and beside it: each card's draw against its power limit (efficiency table, fleet and node charts, the console's GPU work chart), and where the profiling counters can be read, SMs active, SM occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's GPM on Hopper and later or a dcgm-exporter on the node (discovery.dcgm_exporter). The GPU map colours by work by default and keeps busy as a mode. A MIG instance is never published with its card's profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7bfe70c8df
commit
c866b3a8ee
34 files changed
+954
-52
No files matched your search
@@ -340,6 +340,36 @@ type GPUSensor struct {
|
||||
// Wear is what wears the card out, read every minute; nil from an agent
|
||||
// that does not read it.
|
||||
Wear *GPUWear `json:"wear,omitempty"`
|
||||
// Profile is how much of the chip the work uses, which UtilPercent does
|
||||
// not say; nil where nothing on the node can read it.
|
||||
Profile *GPUProfile `json:"profile,omitempty"`
|
||||
}
|
||||
|
||||
// GPUProfile is a card's profiling counters, in percent; -1 is not reported.
|
||||
// UtilPercent is the share of time any kernel ran, and a kernel on one SM of
|
||||
// a hundred counts in full; these say how much of the chip it ran on.
|
||||
type GPUProfile struct {
|
||||
// SMActive is the share of time the SMs had work, averaged over all of
|
||||
// them; SMOccupancy how full those SMs were with warps.
|
||||
SMActive float64 `json:"sm_active"`
|
||||
SMOccupancy float64 `json:"sm_occupancy"`
|
||||
// TensorActive is the tensor cores, DRAMActive the memory interface.
|
||||
TensorActive float64 `json:"tensor_active"`
|
||||
DRAMActive float64 `json:"dram_active"`
|
||||
// Source is what read them: "gpm" (NVML, Hopper and later) or "dcgm".
|
||||
Source string `json:"source,omitempty"`
|
||||
}
|
||||
|
||||
// PowerLoad is the card's draw as a share of its power limit, from 0 to 1,
|
||||
// or -1 without both figures. Unlike UtilPercent it rises with how much of
|
||||
// the chip is working, not only with whether anything is: a card reading
|
||||
// 92% busy drew 239 W of 260 W in decode and 112 W when the busy figure was
|
||||
// a kernel or two between requests.
|
||||
func (g GPUSensor) PowerLoad() float64 {
|
||||
if g.PowerW <= 0 || g.PowerLimitW <= 0 {
|
||||
return -1
|
||||
}
|
||||
return min(g.PowerW/g.PowerLimitW, 1)
|
||||
}
|
||||
|
||||
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
|
||||
|
||||
Reference in new issue
Block a user