The busy figure is nvidia-smi's utilization.gpu: the share of time any kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy while the SMs had work 69% of the time and issued on 18% of their cycles. Busy is now said to be that, and beside it: each card's draw against its power limit (efficiency table, fleet and node charts, the console's GPU work chart), and where the profiling counters can be read, SMs active, SM occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's GPM on Hopper and later or a dcgm-exporter on the node (discovery.dcgm_exporter). The GPU map colours by work by default and keeps busy as a mode. A MIG instance is never published with its card's profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7bfe70c8df
commit
c866b3a8ee
34 files changed
+954
-52
No files matched your search
@@ -28,6 +28,24 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **How much of each GPU works, beside whether anything does.** The busy
|
||||
figure is nvidia-smi's `utilization.gpu` — the share of time any kernel
|
||||
ran, which a kernel on one SM fills — and it is now said to be that: on
|
||||
the dashboard, the node page, the metric's help and Grafana. Beside it,
|
||||
each card's draw against its power limit (the efficiency table, the fleet
|
||||
and node charts, the console's GPU work chart), and where the card's
|
||||
profiling counters can be read, its SMs, SM occupancy, tensor cores and
|
||||
memory interface: `llmgw_gpu_sm_active_percent`,
|
||||
`llmgw_gpu_sm_occupancy_percent`, `llmgw_gpu_tensor_active_percent`,
|
||||
`llmgw_gpu_dram_active_percent`. The agent reads them from NVML's GPM on
|
||||
Hopper and later, and from a dcgm-exporter on the node elsewhere
|
||||
(`discovery.dcgm_exporter`, default `http://127.0.0.1:9400/metrics`, `off`
|
||||
for none). The GPU map colours by this "work" by default — SMs active
|
||||
where read, power against the limit otherwise — and keeps busy as a mode.
|
||||
On cuda01 decode read 92% busy with the SMs at work 69% of the time.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A model of several files showed one file's download as the whole.** A node fetching a set reported the part it
|
||||
|
||||
Reference in new issue
Block a user