The busy figure is nvidia-smi's utilization.gpu: the share of time any kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy while the SMs had work 69% of the time and issued on 18% of their cycles. Busy is now said to be that, and beside it: each card's draw against its power limit (efficiency table, fleet and node charts, the console's GPU work chart), and where the profiling counters can be read, SMs active, SM occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's GPM on Hopper and later or a dcgm-exporter on the node (discovery.dcgm_exporter). The GPU map colours by work by default and keeps busy as a mode. A MIG instance is never published with its card's profile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
7bfe70c8df
commit
c866b3a8ee
34 files changed
+954
-52
No files matched your search
@@ -11019,6 +11019,30 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"inventory.GPUProfile": {
|
||||
"type": "object",
|
||||
"description": "GPUProfile is a card's profiling counters, in percent; -1 is not reported. UtilPercent is the share of time any kernel ran, and a kernel on one SM of a hundred counts in full; these say how much of the chip it ran on.",
|
||||
"properties": {
|
||||
"sm_active": {
|
||||
"type": "number",
|
||||
"description": "SMActive is the share of time the SMs had work, averaged over all of them; SMOccupancy how full those SMs were with warps."
|
||||
},
|
||||
"sm_occupancy": {
|
||||
"type": "number"
|
||||
},
|
||||
"tensor_active": {
|
||||
"type": "number",
|
||||
"description": "TensorActive is the tensor cores, DRAMActive the memory interface."
|
||||
},
|
||||
"dram_active": {
|
||||
"type": "number"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is what read them: \"gpm\" (NVML, Hopper and later) or \"dcgm\"."
|
||||
}
|
||||
}
|
||||
},
|
||||
"inventory.GPUSensor": {
|
||||
"type": "object",
|
||||
"description": "GPUSensor is one card's heat and power, and what is throttling it.",
|
||||
@@ -11066,6 +11090,10 @@
|
||||
"wear": {
|
||||
"$ref": "#/components/schemas/inventory.GPUWear",
|
||||
"description": "Wear is what wears the card out, read every minute; nil from an agent that does not read it."
|
||||
},
|
||||
"profile": {
|
||||
"$ref": "#/components/schemas/inventory.GPUProfile",
|
||||
"description": "Profile is how much of the chip the work uses, which UtilPercent does not say; nil where nothing on the node can read it."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+20
-1
@@ -2019,12 +2019,31 @@ dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
|
||||
to run it, which is not the control plane: monitoring that dies with the thing
|
||||
it monitors tells you nothing at the moment you need it.
|
||||
|
||||
Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
|
||||
Three figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
|
||||
rather than zero when the driver does not report it — a flat zero across a
|
||||
fleet means the measurement is missing, not that the fleet is idle. And
|
||||
`llmgw_gpu_nvlink_peers` at zero everywhere means the topology is unknown,
|
||||
which is what a binary built without the nvml tag reports.
|
||||
|
||||
The third is what "busy" means. `llmgw_gpu_utilization_percent` is
|
||||
nvidia-smi's `utilization.gpu`: the share of time *any* kernel ran, however
|
||||
little of the chip it ran on — a kernel on one SM of 142 reads 100%. A Quadro
|
||||
RTX 6000 decoding read 92% busy while its SMs had work 69% of the time and
|
||||
issued an instruction on 18% of their cycles. How much of the chip is at work
|
||||
is in `llmgw_gpu_sm_active_percent`, `llmgw_gpu_sm_occupancy_percent`,
|
||||
`llmgw_gpu_tensor_active_percent` and `llmgw_gpu_dram_active_percent`, from
|
||||
the card's profiling counters. The agent reads them from NVML's GPM on
|
||||
Hopper and later, with nothing to install; on older cards it reads them from
|
||||
a [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter) on the node, at
|
||||
`http://127.0.0.1:9400/metrics` unless `discovery.dcgm_exporter` in
|
||||
agent.yaml says otherwise (`off` reads none). Where neither is there, the
|
||||
dashboard's GPU map and the console colour cards by their draw against the
|
||||
power limit (`llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts`), which
|
||||
rises with how much of the chip works rather than with whether anything
|
||||
does. Low tensor-core activity in decode is not waste: a quantised
|
||||
matrix-vector product at batch one runs on the integer units, and it is
|
||||
prefill and batches that use the tensor cores.
|
||||
|
||||
### Token by token, and SLOs
|
||||
|
||||
Beside time to the first token, every streamed answer is timed per token:
|
||||
|
||||
@@ -234,6 +234,7 @@ rather than repeated.
|
||||
| `ib_root` | `string` | [Options](#options) |
|
||||
| `prefer_nvml` | `bool` | [Options](#options) |
|
||||
| `skip_gpus` | `bool` | [Options](#options) |
|
||||
| `dcgm_exporter` | `string` | [Options](#options) |
|
||||
| `repo` | `string` | [Origin](#origin) |
|
||||
| `revision` | `string` | [Origin](#origin) |
|
||||
| `files` | `[]OriginFile` | [Origin](#origin) |
|
||||
@@ -2182,6 +2183,15 @@ is used, is the more exact source of the peers.
|
||||
SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
||||
found to have none, so this is for one whose driver is broken.
|
||||
|
||||
### `dcgm_exporter`
|
||||
|
||||
`string`
|
||||
|
||||
DCGMExporter is the dcgm-exporter metrics URL read for how much of
|
||||
each card's chip is in use, on cards NVML's GPM does not cover (before
|
||||
Hopper). Empty is http://127.0.0.1:9400/metrics; "off" reads none. A
|
||||
node without one costs a refused connection every five minutes.
|
||||
|
||||
## Origin
|
||||
|
||||
### `repo`
|
||||
|
||||
Reference in new issue
Block a user