Sample GPU utilisation and memory every heartbeat, and chart them
They were read once, at agent start, and published unchanged since. Each sample now carries utilisation, memory and bandwidth in use, the SM clock and uncorrected ECC errors. The node page charts each card, the cluster page the fleet by class, Grafana has a GPU row, and three GPU alerts are added. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
dc483d8c68
commit
bd36798b15
19 files changed
+781
-33
No files matched your search
@@ -35,13 +35,18 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{with .Load}}{{if .GPUs}}
|
||||
<h2>GPUs</h2>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
|
||||
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by</th></tr>
|
||||
{{range .GPUs}}<tr>
|
||||
<td class="num">{{.Index}}</td>
|
||||
{{/* From the last sample; a dash where the driver or an older agent did
|
||||
not say, rather than a zero nobody measured. */}}
|
||||
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} <span class="note">memory {{printf "%.0f" .MemUtilPercent}}%</span>{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{printf "%.1f" (gib .MemoryUsedMB)}} <span class="note">of {{printf "%.0f" (gib .MemoryTotalMB)}} GB</span></span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
|
||||
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
|
||||
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
|
||||
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
{{end}}{{end}}
|
||||
{{end}}
|
||||
@@ -120,6 +125,16 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
|
||||
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}}
|
||||
{{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
|
||||
</div>
|
||||
{{if $.Node.GPUTotal}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}}
|
||||
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
|
||||
</div>
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}}
|
||||
{{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
|
||||
</div>
|
||||
{{end}}
|
||||
<div class="chart-row">
|
||||
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}}
|
||||
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
|
||||
|
||||
Reference in new issue
Block a user