Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 372ef64889 Count the GPU time each owner's models held, on the usage page
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m48s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 16:07:59 +02:00
Ilker ManapandClaude Opus 5.5 82a63cfc6e Read what wears a GPU out: XIDs, remapped rows, PCIe and NVLink errors
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 16:02:07 +02:00
Ilker ManapandClaude Opus 5.5 ff554c7062 Show what each model gets from its GPUs, per card and per watt-hour
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 15:56:42 +02:00
Ilker ManapandClaude Opus 5.5 b6a802c03b Map the fleet's GPUs, and name the cards held and idle
The cluster page draws every card coloured by utilisation, memory or
temperature, and lists replicas whose every card is under 5% busy.
llmgw_gpu_assigned says which replica, model and owner hold each card.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 15:54:58 +02:00
Ilker ManapandClaude Opus 5.5 bd36798b15 Sample GPU utilisation and memory every heartbeat, and chart them
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m52s
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 15:42:17 +02:00
31 changed files with 2356 additions and 33 deletions

No files matched your search

+51
View File
@@ -24,6 +24,57 @@ APIs change between minors without a deprecation period.
From the field report in issue #2, after a deployment on GPU nodes.
### GPU metrics that move, and charts of them
A GPU's utilisation and memory in use were read once, when the agent
started, and published from then on: a card busy since looked as idle as it
was at registration, and one idle since as busy. They are now sampled with
the temperatures and power every heartbeat (ten seconds at most), with the
memory bandwidth in use, the SM clock and uncorrected ECC errors besides.
An agent too old to send them still has its registration figures published,
and a figure the driver does not give is left out rather than shown as zero.
- **New series**: `llmgw_gpu_memory_utilization_percent`,
`llmgw_gpu_sm_clock_mhz`; `llmgw_gpu_utilization_percent`,
`llmgw_gpu_memory_used_bytes` and `llmgw_gpu_ecc_errors` now from the
sample.
- **The node page** shows each card's utilisation, memory, clock and ECC
errors in its table, and charts of utilisation, memory, temperature and
power, one line per card.
- **The cluster page** charts the fleet's GPUs by class: average utilisation
and memory in use.
- **Grafana** (`deploy/monitoring/grafana/llmgw-overview.json`) has a GPU row:
utilisation, memory, memory bandwidth, temperature, power against the
limit, clock, and what holds cards back.
- **Alerts**: `GPUThermalThrottling` (ten minutes slowed by heat),
`GPUUncorrectedECCErrors`, `GPUHot` (above 87 °C for ten minutes).
- **The cluster page's GPU map**: every card a square, a row per node,
coloured by how busy, how full or how hot it is (`?gpumap=util|memory|temp`),
with what holds it and its figures under the pointer; free cards dashed.
- **Cards held and doing nothing**: the replicas whose every card reads under
5% busy, with when they were last asked — the first place to find room.
- **What each model gets from its cards**, on the cluster page and in
Grafana: output tokens a second per GPU and per watt-hour, with the cards'
draw and average utilisation, over the last quarter hour.
- **What wears a card out**, read every minute: XID errors from the kernel's
log (counted from when the agent started), memory rows remapped or pages
retired and whether either waits for a reset, a card out of spare rows,
PCIe link retries and NVLink errors. Published as `llmgw_gpu_xid_errors`,
`llmgw_gpu_last_xid`, `llmgw_gpu_remapped_rows`, `llmgw_gpu_remap_pending`,
`llmgw_gpu_remap_failure`, `llmgw_gpu_retired_pages`,
`llmgw_gpu_retirement_pending`, `llmgw_gpu_pcie_replays`,
`llmgw_gpu_nvlink_errors`; said in words on the node page; a red outline
on the GPU map; a Grafana panel; and alerts — `GPUXidError`,
`GPUOutOfSpareRows`, `GPUNeedsReset`, `GPUPCIeRetriesClimbing`,
`GPUNVLinkErrorsClimbing`.
- **GPU time by owner** on the usage page: the cards each owner's models
held over the window, charted and summed into GPU-hours, whether or not the
models were asked anything — the time a GPU quota is about. From the
metrics store, as far back as it keeps history.
- **`llmgw_gpu_assigned{node,gpu,model,replica,owner}`**: which replica holds
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### A GPU placement no longer runs on the CPU unnoticed
llama.cpp placed on a GPU, with its CUDA backend unable to load (cuBLAS not
+211 -3
View File
@@ -1550,7 +1550,36 @@ type GPUSensor struct {
PowerLimitW float64 `protobuf:"fixed64,5,opt,name=power_limit_w,json=powerLimitW,proto3" json:"power_limit_w,omitempty"`
// throttle names what is holding the clocks down now: sw_power_cap,
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
Throttle []string `protobuf:"bytes,6,rep,name=throttle,proto3" json:"throttle,omitempty"`
// How busy the card is, sampled with the rest, rather than read once at
// registration. -1 is not reported by the driver. memory_total_mb is set
// by every agent that sends these, so zero there is an older agent.
UtilizationPercent float64 `protobuf:"fixed64,7,opt,name=utilization_percent,json=utilizationPercent,proto3" json:"utilization_percent,omitempty"`
MemoryUtilizationPercent float64 `protobuf:"fixed64,8,opt,name=memory_utilization_percent,json=memoryUtilizationPercent,proto3" json:"memory_utilization_percent,omitempty"`
MemoryUsedMb float64 `protobuf:"fixed64,9,opt,name=memory_used_mb,json=memoryUsedMb,proto3" json:"memory_used_mb,omitempty"`
MemoryTotalMb float64 `protobuf:"fixed64,10,opt,name=memory_total_mb,json=memoryTotalMb,proto3" json:"memory_total_mb,omitempty"`
SmClockMhz float64 `protobuf:"fixed64,11,opt,name=sm_clock_mhz,json=smClockMhz,proto3" json:"sm_clock_mhz,omitempty"`
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
EccUncorrected float64 `protobuf:"fixed64,12,opt,name=ecc_uncorrected,json=eccUncorrected,proto3" json:"ecc_uncorrected,omitempty"`
// What wears the card out, read every minute; has_wear says these are
// present. Counts of -1 are not reported for this card.
HasWear bool `protobuf:"varint,13,opt,name=has_wear,json=hasWear,proto3" json:"has_wear,omitempty"`
PcieReplays int64 `protobuf:"varint,14,opt,name=pcie_replays,json=pcieReplays,proto3" json:"pcie_replays,omitempty"`
RetiredPagesSbe int64 `protobuf:"varint,15,opt,name=retired_pages_sbe,json=retiredPagesSbe,proto3" json:"retired_pages_sbe,omitempty"`
RetiredPagesDbe int64 `protobuf:"varint,16,opt,name=retired_pages_dbe,json=retiredPagesDbe,proto3" json:"retired_pages_dbe,omitempty"`
RetirementPending bool `protobuf:"varint,17,opt,name=retirement_pending,json=retirementPending,proto3" json:"retirement_pending,omitempty"`
RemappedRowsCorrectable int64 `protobuf:"varint,18,opt,name=remapped_rows_correctable,json=remappedRowsCorrectable,proto3" json:"remapped_rows_correctable,omitempty"`
RemappedRowsUncorrectable int64 `protobuf:"varint,19,opt,name=remapped_rows_uncorrectable,json=remappedRowsUncorrectable,proto3" json:"remapped_rows_uncorrectable,omitempty"`
RemapPending bool `protobuf:"varint,20,opt,name=remap_pending,json=remapPending,proto3" json:"remap_pending,omitempty"`
RemapFailure bool `protobuf:"varint,21,opt,name=remap_failure,json=remapFailure,proto3" json:"remap_failure,omitempty"`
NvlinkCrcErrors int64 `protobuf:"varint,22,opt,name=nvlink_crc_errors,json=nvlinkCrcErrors,proto3" json:"nvlink_crc_errors,omitempty"`
NvlinkReplayErrors int64 `protobuf:"varint,23,opt,name=nvlink_replay_errors,json=nvlinkReplayErrors,proto3" json:"nvlink_replay_errors,omitempty"`
NvlinkRecoveryErrors int64 `protobuf:"varint,24,opt,name=nvlink_recovery_errors,json=nvlinkRecoveryErrors,proto3" json:"nvlink_recovery_errors,omitempty"`
// xid_errors are the driver's error events since the agent started.
XidErrors int64 `protobuf:"varint,25,opt,name=xid_errors,json=xidErrors,proto3" json:"xid_errors,omitempty"`
LastXid int32 `protobuf:"varint,26,opt,name=last_xid,json=lastXid,proto3" json:"last_xid,omitempty"`
LastXidText string `protobuf:"bytes,27,opt,name=last_xid_text,json=lastXidText,proto3" json:"last_xid_text,omitempty"`
LastXidUnix int64 `protobuf:"varint,28,opt,name=last_xid_unix,json=lastXidUnix,proto3" json:"last_xid_unix,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -1627,6 +1656,160 @@ func (x *GPUSensor) GetThrottle() []string {
return nil
}
func (x *GPUSensor) GetUtilizationPercent() float64 {
if x != nil {
return x.UtilizationPercent
}
return 0
}
func (x *GPUSensor) GetMemoryUtilizationPercent() float64 {
if x != nil {
return x.MemoryUtilizationPercent
}
return 0
}
func (x *GPUSensor) GetMemoryUsedMb() float64 {
if x != nil {
return x.MemoryUsedMb
}
return 0
}
func (x *GPUSensor) GetMemoryTotalMb() float64 {
if x != nil {
return x.MemoryTotalMb
}
return 0
}
func (x *GPUSensor) GetSmClockMhz() float64 {
if x != nil {
return x.SmClockMhz
}
return 0
}
func (x *GPUSensor) GetEccUncorrected() float64 {
if x != nil {
return x.EccUncorrected
}
return 0
}
func (x *GPUSensor) GetHasWear() bool {
if x != nil {
return x.HasWear
}
return false
}
func (x *GPUSensor) GetPcieReplays() int64 {
if x != nil {
return x.PcieReplays
}
return 0
}
func (x *GPUSensor) GetRetiredPagesSbe() int64 {
if x != nil {
return x.RetiredPagesSbe
}
return 0
}
func (x *GPUSensor) GetRetiredPagesDbe() int64 {
if x != nil {
return x.RetiredPagesDbe
}
return 0
}
func (x *GPUSensor) GetRetirementPending() bool {
if x != nil {
return x.RetirementPending
}
return false
}
func (x *GPUSensor) GetRemappedRowsCorrectable() int64 {
if x != nil {
return x.RemappedRowsCorrectable
}
return 0
}
func (x *GPUSensor) GetRemappedRowsUncorrectable() int64 {
if x != nil {
return x.RemappedRowsUncorrectable
}
return 0
}
func (x *GPUSensor) GetRemapPending() bool {
if x != nil {
return x.RemapPending
}
return false
}
func (x *GPUSensor) GetRemapFailure() bool {
if x != nil {
return x.RemapFailure
}
return false
}
func (x *GPUSensor) GetNvlinkCrcErrors() int64 {
if x != nil {
return x.NvlinkCrcErrors
}
return 0
}
func (x *GPUSensor) GetNvlinkReplayErrors() int64 {
if x != nil {
return x.NvlinkReplayErrors
}
return 0
}
func (x *GPUSensor) GetNvlinkRecoveryErrors() int64 {
if x != nil {
return x.NvlinkRecoveryErrors
}
return 0
}
func (x *GPUSensor) GetXidErrors() int64 {
if x != nil {
return x.XidErrors
}
return 0
}
func (x *GPUSensor) GetLastXid() int32 {
if x != nil {
return x.LastXid
}
return 0
}
func (x *GPUSensor) GetLastXidText() string {
if x != nil {
return x.LastXidText
}
return ""
}
func (x *GPUSensor) GetLastXidUnix() int64 {
if x != nil {
return x.LastXidUnix
}
return 0
}
// ShardLoad is one engine process group's use.
type ShardLoad struct {
state protoimpl.MessageState `protogen:"open.v1"`
@@ -3999,14 +4182,39 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
" \x01(\x01R\tcpuPowerW\x12 \n" +
"\finlet_temp_c\x18\v \x01(\x01R\n" +
"inletTempC\x12'\n" +
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xb5\x01\n" +
"\x04gpus\x18\f \x03(\v2\x13.llmgw.v1.GPUSensorR\x04gpus\"\xde\b\n" +
"\tGPUSensor\x12\x14\n" +
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x15\n" +
"\x06temp_c\x18\x02 \x01(\x01R\x05tempC\x12\"\n" +
"\rmemory_temp_c\x18\x03 \x01(\x01R\vmemoryTempC\x12\x17\n" +
"\apower_w\x18\x04 \x01(\x01R\x06powerW\x12\"\n" +
"\rpower_limit_w\x18\x05 \x01(\x01R\vpowerLimitW\x12\x1a\n" +
"\bthrottle\x18\x06 \x03(\tR\bthrottle\"m\n" +
"\bthrottle\x18\x06 \x03(\tR\bthrottle\x12/\n" +
"\x13utilization_percent\x18\a \x01(\x01R\x12utilizationPercent\x12<\n" +
"\x1amemory_utilization_percent\x18\b \x01(\x01R\x18memoryUtilizationPercent\x12$\n" +
"\x0ememory_used_mb\x18\t \x01(\x01R\fmemoryUsedMb\x12&\n" +
"\x0fmemory_total_mb\x18\n" +
" \x01(\x01R\rmemoryTotalMb\x12 \n" +
"\fsm_clock_mhz\x18\v \x01(\x01R\n" +
"smClockMhz\x12'\n" +
"\x0fecc_uncorrected\x18\f \x01(\x01R\x0eeccUncorrected\x12\x19\n" +
"\bhas_wear\x18\r \x01(\bR\ahasWear\x12!\n" +
"\fpcie_replays\x18\x0e \x01(\x03R\vpcieReplays\x12*\n" +
"\x11retired_pages_sbe\x18\x0f \x01(\x03R\x0fretiredPagesSbe\x12*\n" +
"\x11retired_pages_dbe\x18\x10 \x01(\x03R\x0fretiredPagesDbe\x12-\n" +
"\x12retirement_pending\x18\x11 \x01(\bR\x11retirementPending\x12:\n" +
"\x19remapped_rows_correctable\x18\x12 \x01(\x03R\x17remappedRowsCorrectable\x12>\n" +
"\x1bremapped_rows_uncorrectable\x18\x13 \x01(\x03R\x19remappedRowsUncorrectable\x12#\n" +
"\rremap_pending\x18\x14 \x01(\bR\fremapPending\x12#\n" +
"\rremap_failure\x18\x15 \x01(\bR\fremapFailure\x12*\n" +
"\x11nvlink_crc_errors\x18\x16 \x01(\x03R\x0fnvlinkCrcErrors\x120\n" +
"\x14nvlink_replay_errors\x18\x17 \x01(\x03R\x12nvlinkReplayErrors\x124\n" +
"\x16nvlink_recovery_errors\x18\x18 \x01(\x03R\x14nvlinkRecoveryErrors\x12\x1d\n" +
"\n" +
"xid_errors\x18\x19 \x01(\x03R\txidErrors\x12\x19\n" +
"\blast_xid\x18\x1a \x01(\x05R\alastXid\x12\"\n" +
"\rlast_xid_text\x18\x1b \x01(\tR\vlastXidText\x12\"\n" +
"\rlast_xid_unix\x18\x1c \x01(\x03R\vlastXidUnix\"m\n" +
"\tShardLoad\x12\x19\n" +
"\bshard_id\x18\x01 \x01(\tR\ashardId\x12\x1b\n" +
"\tcpu_cores\x18\x02 \x01(\x01R\bcpuCores\x12(\n" +
+29
View File
@@ -304,6 +304,35 @@ message GPUSensor {
// throttle names what is holding the clocks down now: sw_power_cap,
// hw_slowdown, sw_thermal, hw_thermal, power_brake.
repeated string throttle = 6;
// How busy the card is, sampled with the rest, rather than read once at
// registration. -1 is not reported by the driver. memory_total_mb is set
// by every agent that sends these, so zero there is an older agent.
double utilization_percent = 7;
double memory_utilization_percent = 8;
double memory_used_mb = 9;
double memory_total_mb = 10;
double sm_clock_mhz = 11;
// ecc_uncorrected counts uncorrected ECC errors since the driver loaded.
double ecc_uncorrected = 12;
// What wears the card out, read every minute; has_wear says these are
// present. Counts of -1 are not reported for this card.
bool has_wear = 13;
int64 pcie_replays = 14;
int64 retired_pages_sbe = 15;
int64 retired_pages_dbe = 16;
bool retirement_pending = 17;
int64 remapped_rows_correctable = 18;
int64 remapped_rows_uncorrectable = 19;
bool remap_pending = 20;
bool remap_failure = 21;
int64 nvlink_crc_errors = 22;
int64 nvlink_replay_errors = 23;
int64 nvlink_recovery_errors = 24;
// xid_errors are the driver's error events since the agent started.
int64 xid_errors = 25;
int32 last_xid = 26;
string last_xid_text = 27;
int64 last_xid_unix = 28;
}
// ShardLoad is one engine process group's use.
@@ -487,6 +487,427 @@
"defaults": {},
"overrides": []
}
},
{
"type": "row",
"title": "GPUs",
"id": 14,
"gridPos": {
"x": 0,
"y": 30,
"w": 24,
"h": 1
},
"collapsed": false,
"panels": []
},
{
"type": "timeseries",
"title": "GPU busy, by card",
"description": "Share of each sample period a kernel was running, sampled every heartbeat.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 31,
"w": 12,
"h": 7
},
"id": 15,
"targets": [
{
"expr": "llmgw_gpu_utilization_percent",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU memory in use, by card",
"description": "Framebuffer in use. Engines reserve most of it at start.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 12,
"y": 31,
"w": 12,
"h": 7
},
"id": 16,
"targets": [
{
"expr": "llmgw_gpu_memory_used_bytes / llmgw_gpu_memory_total_bytes",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU memory bandwidth busy, by card",
"description": "Decode is bound by memory: high here with compute low is a card waiting on memory.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 38,
"w": 12,
"h": 7
},
"id": 17,
"targets": [
{
"expr": "llmgw_gpu_memory_utilization_percent",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"min": 0,
"max": 100
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU temperature",
"description": "Throttling usually starts in the high 80s.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 12,
"y": 38,
"w": 12,
"h": 7
},
"id": 18,
"targets": [
{
"expr": "llmgw_gpu_temperature_celsius",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "celsius"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU power against its limit",
"description": "1 is the enforced limit; a card held there is power-capped.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 45,
"w": 12,
"h": 7
},
"id": 19,
"targets": [
{
"expr": "llmgw_gpu_power_watts / llmgw_gpu_power_limit_watts",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1.1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU clock",
"description": "A clock held low under load is the card throttled; see the next panel for why.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 12,
"y": 45,
"w": 12,
"h": 7
},
"id": 20,
"targets": [
{
"expr": "llmgw_gpu_sm_clock_mhz",
"legendFormat": "{{node}} gpu {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "rotmhz"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "What holds GPUs back",
"description": "Cards throttled now, by reason, and uncorrected ECC errors since the driver loaded.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 52,
"w": 24,
"h": 7
},
"id": 21,
"targets": [
{
"expr": "sum by (reason) (llmgw_gpu_throttled)",
"legendFormat": "throttled: {{reason}}",
"refId": "A"
},
{
"expr": "sum by (node, gpu) (llmgw_gpu_ecc_errors)",
"legendFormat": "ECC errors {{node}} gpu {{gpu}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "Output tokens/s per GPU, by model",
"description": "What each model gets out of each card it holds.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 0,
"y": 59,
"w": 12,
"h": 7
},
"id": 22,
"targets": [
{
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) / count by (model) (llmgw_gpu_assigned)",
"legendFormat": "{{model}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "Output tokens per watt-hour, by model",
"description": "Tokens for the energy the model's cards draw.",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"gridPos": {
"x": 12,
"y": 59,
"w": 12,
"h": 7
},
"id": 23,
"targets": [
{
"expr": "sum by (model) (rate(llmgw_tokens_total{direction=\"completion\"}[5m])) * 3600 / sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)",
"legendFormat": "{{model}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"type": "timeseries",
"title": "GPU wear",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"id": 24,
"description": "What wears cards out: XID errors in the last 15 minutes, memory rows remapped, PCIe retries an hour. Anything above zero here is worth a look on that card's node page.",
"gridPos": {
"x": 0,
"y": 66,
"w": 24,
"h": 7
},
"targets": [
{
"expr": "increase(llmgw_gpu_xid_errors[15m]) > 0",
"legendFormat": "XID {{node}} gpu {{gpu}}",
"refId": "A"
},
{
"expr": "sum by (node, gpu) (llmgw_gpu_remapped_rows) > 0",
"legendFormat": "rows remapped {{node}} gpu {{gpu}}",
"refId": "B"
},
{
"expr": "increase(llmgw_gpu_pcie_replays[1h]) > 0",
"legendFormat": "PCIe retries/h {{node}} gpu {{gpu}}",
"refId": "C"
},
{
"expr": "increase(llmgw_gpu_nvlink_errors[1h]) > 0",
"legendFormat": "NVLink {{kind}}/h {{node}} gpu {{gpu}}",
"refId": "D"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
}
]
}
+88
View File
@@ -200,6 +200,94 @@ groups:
is skipped by placement rather than failing, so this is quiet until
somebody wonders why a model will not land there.
- name: llmgw-gpus
rules:
- alert: GPUThermalThrottling
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
description: >-
Ten minutes held back by temperature: every replica on the card is
slower than its measurements say. Usually airflow, a fan, or the
room — the node page shows the card's temperature over time.
- alert: GPUUncorrectedECCErrors
expr: llmgw_gpu_ecc_errors > 0
for: 1m
labels: { severity: critical }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
description: >-
Uncorrected memory errors since the driver loaded: what runs on the
card may be computing on corrupted memory. Drain the node
(llmgw ctl drain) and have the card looked at.
- alert: GPUXidError
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
description: >-
The driver reported an error on the card (its code is
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
94/95 contained and uncontained ECC errors: those need the card
looked at. Others are often an engine's own fault, and pass.
- alert: GPUOutOfSpareRows
expr: llmgw_gpu_remap_failure == 1
for: 1m
labels: { severity: critical }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
description: >-
The card cannot remap another failing row. Drain the node and
have the card replaced.
- alert: GPUNeedsReset
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
description: >-
Rows remapped or pages retired take effect only after the card is
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
it back.
- alert: GPUPCIeRetriesClimbing
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
description: >-
The card's link to the host is retrying. A riser, the slot or a
cable is the usual cause, and transfers to the card slow down.
- alert: GPUNVLinkErrorsClimbing
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
for: 0m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
description: >-
Errors on the card's NVLink links: tensor-parallel replicas on it
slow down, and a link that keeps failing takes a replica with it.
- alert: GPUHot
expr: llmgw_gpu_temperature_celsius > 87
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
description: >-
Ten minutes above 87 °C, where most data-centre cards begin to
throttle.
- name: monitoring-host
rules:
- alert: MonitoringDiskFilling
+28 -2
View File
@@ -104,6 +104,8 @@ type Agent struct {
// sensors reads heat and power: CPU and board from hwmon and RAPL, GPUs
// from nvidia-smi.
sensors *discovery.SensorSampler
// wear reads what wears a GPU out: remapped rows, PCIe retries, XIDs.
wear *discovery.HealthSampler
// fetchWatchers are the shards waiting on each source, so progress on one
// download reaches every shard that asked for it. A fetch is single-flight
@@ -150,6 +152,7 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
provKick: make(chan struct{}, 1),
load: discovery.NewLoadSampler(),
sensors: discovery.NewSensorSampler(),
wear: discovery.NewHealthSampler(),
}
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
return a, nil
@@ -656,11 +659,34 @@ func (a *Agent) sampleLoad() *llmgwv1.NodeLoad {
if a.sensors != nil {
se := a.sensors.Sample(context.Background())
out.CpuTempC, out.CpuPowerW, out.InletTempC = se.CPUTempC, se.CPUPowerW, se.InletTempC
var wear map[int]discovery.GPUWear
if a.wear != nil && len(se.GPUs) > 0 {
wear = map[int]discovery.GPUWear{}
for _, w := range a.wear.Sample(context.Background()) {
wear[w.Index] = w
}
}
for _, g := range se.GPUs {
out.Gpus = append(out.Gpus, &llmgwv1.GPUSensor{
gs := &llmgwv1.GPUSensor{
Index: int32(g.Index), TempC: g.TempC, MemoryTempC: g.MemoryTempC,
PowerW: g.PowerW, PowerLimitW: g.PowerLimitW, Throttle: g.Throttle,
})
UtilizationPercent: g.UtilPercent, MemoryUtilizationPercent: g.MemUtilPercent,
MemoryUsedMb: g.MemoryUsedMB, MemoryTotalMb: g.MemoryTotalMB,
SmClockMhz: g.SMClockMHz, EccUncorrected: g.ECCUncorrected,
}
if w, ok := wear[g.Index]; ok {
gs.HasWear = true
gs.PcieReplays, gs.RetiredPagesSbe, gs.RetiredPagesDbe, gs.RetirementPending =
w.PCIeReplays, w.RetiredSBE, w.RetiredDBE, w.RetirePending
gs.RemappedRowsCorrectable, gs.RemappedRowsUncorrectable, gs.RemapPending, gs.RemapFailure =
w.RemappedCorrectable, w.RemappedUncorrectable, w.RemapPending, w.RemapFailure
gs.NvlinkCrcErrors, gs.NvlinkReplayErrors, gs.NvlinkRecoveryErrors = w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery
gs.XidErrors, gs.LastXid, gs.LastXidText = w.XIDs, int32(w.LastXID), w.LastXIDText
if !w.LastXIDAt.IsZero() {
gs.LastXidUnix = w.LastXIDAt.Unix()
}
}
out.Gpus = append(out.Gpus, gs)
}
}
return out
+280
View File
@@ -0,0 +1,280 @@
package discovery
import (
"context"
"encoding/xml"
"os/exec"
"regexp"
"strconv"
"strings"
"sync"
"time"
)
// GPUWear is what wears a card out or takes it down, beyond temperature:
// read every minute, from nvidia-smi and the kernel's log. The early signs of
// a card failing are here — a growing count of rows its memory had to remap,
// PCIe link retries, XID errors — long before a replica on it fails.
//
// A count of -1 is not reported: a card without row remapping (before
// Ampere) has no such count, which is not the same as none.
type GPUWear struct {
Index int
// PCIeReplays are link-level retries on the card's PCIe link: a few are
// noise, a count that keeps climbing is a riser, slot or cable.
PCIeReplays int64
// RetiredSBE and RetiredDBE are memory pages retired for single and
// double bit errors (cards before Ampere); RetirePending is a page
// waiting for a reset to be retired.
RetiredSBE, RetiredDBE int64
RetirePending bool
// RemappedCorrectable and RemappedUncorrectable are memory rows remapped
// (Ampere and later). RemapPending waits for a reset; RemapFailure is a
// card out of spare rows — it needs replacing.
RemappedCorrectable, RemappedUncorrectable int64
RemapPending, RemapFailure bool
// NVLink errors summed over the card's links, by kind.
NVLinkCRC, NVLinkReplay, NVLinkRecovery int64
// XIDs are the driver's error events for this card since the agent
// started, and the last of them.
XIDs int64
LastXID int
LastXIDText string
LastXIDAt time.Time
}
// healthEvery is how often health is read: it changes over hours, and the
// kernel log and nvidia-smi's full report are the heavier reads.
const healthEvery = time.Minute
// HealthSampler reads GPUWear, keeping the XID counts between reads.
type HealthSampler struct {
run func(ctx context.Context, name string, args ...string) ([]byte, error)
now func() time.Time
mu sync.Mutex
at time.Time
last []GPUWear
since time.Time
xids map[int]*GPUWear
started bool
}
// NewHealthSampler reads the machine's own nvidia-smi and kernel log.
func NewHealthSampler() *HealthSampler {
return &HealthSampler{now: time.Now, xids: map[int]*GPUWear{},
run: func(ctx context.Context, name string, args ...string) ([]byte, error) {
return exec.CommandContext(ctx, name, args...).Output() //nolint:gosec // fixed tools
}}
}
// Sample is the cards' health, read again once healthEvery has passed. Nil
// on a machine without nvidia-smi.
func (h *HealthSampler) Sample(ctx context.Context) []GPUWear {
h.mu.Lock()
defer h.mu.Unlock()
now := h.now()
if !h.at.IsZero() && now.Sub(h.at) < healthEvery {
return h.last
}
h.at = now
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
defer cancel()
report, err := h.run(ctx, "nvidia-smi", "-q", "-x")
if err != nil {
h.last = nil
return nil
}
cards, bus := parseHealthReport(report)
if out, err := h.run(ctx, "nvidia-smi", "nvlink", "-e"); err == nil {
for i, e := range parseNVLinkErrors(string(out)) {
if i < len(cards) {
cards[i].NVLinkCRC, cards[i].NVLinkReplay, cards[i].NVLinkRecovery = e[0], e[1], e[2]
}
}
}
h.readXIDs(ctx, now, bus)
for i := range cards {
if x := h.xids[i]; x != nil {
cards[i].XIDs, cards[i].LastXID, cards[i].LastXIDText, cards[i].LastXIDAt = x.XIDs, x.LastXID, x.LastXIDText, x.LastXIDAt
}
}
h.last = cards
return cards
}
// readXIDs adds the XID events logged since the last read. The first read
// starts from now: what happened before the agent started was before, and
// counting the whole log would raise every alert on every restart.
func (h *HealthSampler) readXIDs(ctx context.Context, now time.Time, bus map[string]int) {
if !h.started {
h.started, h.since = true, now
return
}
since := h.since
h.since = now
out, err := h.run(ctx, "journalctl", "-k", "--no-pager", "-o", "short-unix",
"--since", "@"+strconv.FormatInt(since.Unix(), 10), "-g", "NVRM: Xid")
if err != nil {
// No journal, or nothing matched (journalctl exits 1 for that).
return
}
for _, e := range parseXIDs(string(out)) {
i, ok := bus[e.bus]
if !ok {
continue
}
x := h.xids[i]
if x == nil {
x = &GPUWear{Index: i}
h.xids[i] = x
}
x.XIDs++
x.LastXID, x.LastXIDText = e.code, e.text
x.LastXIDAt = now
if e.at > 0 {
x.LastXIDAt = time.Unix(int64(e.at), 0)
}
}
}
// healthLog is the part of nvidia-smi's XML report health is read from.
type healthLog struct {
GPUs []struct {
ID string `xml:"id,attr"`
PCI struct {
Replay string `xml:"replay_counter"`
} `xml:"pci"`
Retired struct {
SBE string `xml:"multiple_single_bit_retirement>retired_count"`
DBE string `xml:"double_bit_retirement>retired_count"`
Pending string `xml:"pending_retirement"`
// Older drivers name it pending_blacklist.
Blacklist string `xml:"pending_blacklist"`
} `xml:"retired_pages"`
Remapped struct {
Corr string `xml:"remapped_row_corr"`
Unc string `xml:"remapped_row_unc"`
Pending string `xml:"remapped_row_pending"`
Failure string `xml:"remapped_row_failure"`
} `xml:"remapped_rows"`
} `xml:"gpu"`
}
// parseHealthReport reads each card's health, in index order, and maps each
// card's PCI bus ("3b:00", as the kernel names it) to its index.
func parseHealthReport(report []byte) ([]GPUWear, map[string]int) {
var log healthLog
if err := xml.Unmarshal(report, &log); err != nil {
return nil, nil
}
bus := map[string]int{}
out := make([]GPUWear, 0, len(log.GPUs))
for i, g := range log.GPUs {
bus[busOf(g.ID)] = i
out = append(out, GPUWear{
Index: i,
PCIeReplays: count(g.PCI.Replay),
RetiredSBE: count(g.Retired.SBE),
RetiredDBE: count(g.Retired.DBE),
RetirePending: yes(g.Retired.Pending) || yes(g.Retired.Blacklist),
RemappedCorrectable: count(g.Remapped.Corr),
RemappedUncorrectable: count(g.Remapped.Unc),
RemapPending: yes(g.Remapped.Pending),
RemapFailure: yes(g.Remapped.Failure),
})
}
return out, bus
}
// busOf is a PCI address as the kernel's XID lines carry it: nvidia-smi
// says "00000000:3B:00.0", the kernel "0000:3b:00".
func busOf(id string) string {
id = strings.ToLower(strings.TrimSpace(id))
if i := strings.LastIndex(id, "."); i >= 0 {
id = id[:i]
}
parts := strings.Split(id, ":")
if len(parts) >= 2 {
return parts[len(parts)-2] + ":" + parts[len(parts)-1]
}
return id
}
func count(s string) int64 {
n, err := strconv.ParseInt(strings.TrimSpace(s), 10, 64)
if err != nil {
return -1
}
return n
}
func yes(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "yes") }
// nvlinkError matches a line of `nvidia-smi nvlink -e`, whose wording varies
// by driver: "Link 0: Replay Errors: 0", "Link 0: CRC Errors: 0",
// "Link 3: Data CRC Errors: 2".
var (
nvlinkGPU = regexp.MustCompile(`^GPU (\d+):`)
nvlinkError = regexp.MustCompile(`Link \d+:\s*(.*?)\s*Errors?\s*:\s*(\d+)`)
)
// parseNVLinkErrors sums each card's link errors by kind: CRC, replay,
// recovery.
func parseNVLinkErrors(out string) map[int][3]int64 {
res := map[int][3]int64{}
gpu := -1
for _, line := range strings.Split(out, "\n") {
if m := nvlinkGPU.FindStringSubmatch(strings.TrimSpace(line)); m != nil {
gpu, _ = strconv.Atoi(m[1])
continue
}
m := nvlinkError.FindStringSubmatch(line)
if m == nil || gpu < 0 {
continue
}
n, _ := strconv.ParseInt(m[2], 10, 64)
e := res[gpu]
kind := strings.ToLower(m[1])
switch {
case strings.Contains(kind, "crc"):
e[0] += n
case strings.Contains(kind, "replay"):
e[1] += n
case strings.Contains(kind, "recovery"):
e[2] += n
}
res[gpu] = e
}
return res
}
type xidEvent struct {
bus string
code int
text string
at float64
}
// xidLine matches the driver's report of an XID:
// "NVRM: Xid (PCI:0000:3b:00): 79, pid=..., GPU has fallen off the bus."
var xidLine = regexp.MustCompile(`NVRM: Xid \(PCI:([0-9a-fA-F:.]+)\): (\d+),\s*(.*)$`)
// parseXIDs reads journalctl's short-unix lines: a timestamp, then the
// message.
func parseXIDs(out string) []xidEvent {
var evs []xidEvent
for _, line := range strings.Split(out, "\n") {
m := xidLine.FindStringSubmatch(line)
if m == nil {
continue
}
code, _ := strconv.Atoi(m[2])
e := xidEvent{bus: busOf(m[1]), code: code, text: strings.TrimSpace(m[3])}
if f := strings.Fields(line); len(f) > 0 {
e.at, _ = strconv.ParseFloat(f[0], 64)
}
evs = append(evs, e)
}
return evs
}
+101
View File
@@ -0,0 +1,101 @@
package discovery
import (
"context"
"errors"
"strings"
"testing"
"time"
)
const healthXML = `<?xml version="1.0" ?>
<nvidia_smi_log>
<gpu id="00000000:3B:00.0">
<pci><replay_counter>0</replay_counter></pci>
<retired_pages>
<multiple_single_bit_retirement><retired_count>N/A</retired_count></multiple_single_bit_retirement>
<double_bit_retirement><retired_count>N/A</retired_count></double_bit_retirement>
<pending_retirement>N/A</pending_retirement>
</retired_pages>
<remapped_rows>
<remapped_row_corr>2</remapped_row_corr>
<remapped_row_unc>1</remapped_row_unc>
<remapped_row_pending>Yes</remapped_row_pending>
<remapped_row_failure>No</remapped_row_failure>
</remapped_rows>
</gpu>
<gpu id="00000000:5E:00.0">
<pci><replay_counter>4021</replay_counter></pci>
<retired_pages>
<multiple_single_bit_retirement><retired_count>3</retired_count></multiple_single_bit_retirement>
<double_bit_retirement><retired_count>1</retired_count></double_bit_retirement>
<pending_blacklist>Yes</pending_blacklist>
</retired_pages>
<remapped_rows>N/A</remapped_rows>
</gpu>
</nvidia_smi_log>`
const nvlinkOut = `GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-aaa)
Link 0: Replay Errors: 1
Link 0: Recovery Errors: 0
Link 0: CRC Errors: 5
Link 1: Data CRC Errors: 2
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-bbb)
Link 0: Replay Errors: 0
`
func TestAGPUsWearAndFailuresAreReadAndCountedFromWhenTheAgentStarted(t *testing.T) {
now := time.Unix(1_800_000_000, 0)
journal := ""
h := &HealthSampler{now: func() time.Time { return now }, xids: map[int]*GPUWear{},
run: func(_ context.Context, name string, args ...string) ([]byte, error) {
switch {
case name == "nvidia-smi" && args[0] == "-q":
return []byte(healthXML), nil
case name == "nvidia-smi" && args[0] == "nvlink":
return []byte(nvlinkOut), nil
case name == "journalctl":
if journal == "" {
return nil, errors.New("exit status 1")
}
return []byte(journal), nil
}
return nil, errors.New("unexpected " + name)
}}
got := h.Sample(t.Context())
if len(got) != 2 {
t.Fatalf("%+v", got)
}
a, b := got[0], got[1]
if a.RemappedCorrectable != 2 || a.RemappedUncorrectable != 1 || !a.RemapPending || a.RemapFailure ||
a.RetiredSBE != -1 || a.PCIeReplays != 0 || a.NVLinkCRC != 7 || a.NVLinkReplay != 1 {
t.Errorf("card 0 %+v", a)
}
if b.PCIeReplays != 4021 || b.RetiredSBE != 3 || b.RetiredDBE != 1 || !b.RetirePending || b.RemappedCorrectable != -1 {
t.Errorf("card 1 %+v", b)
}
// An XID on card 1, logged after the agent started; read on the next
// sample a minute on.
journal = "1800000030.123456 gpu07 kernel: NVRM: Xid (PCI:0000:5e:00): 79, pid=1234, name=llama-server, GPU has fallen off the bus.\n"
now = now.Add(61 * time.Second)
got = h.Sample(t.Context())
if got[1].XIDs != 1 || got[1].LastXID != 79 || !strings.Contains(got[1].LastXIDText, "fallen off the bus") ||
got[1].LastXIDAt.Unix() != 1_800_000_030 || got[0].XIDs != 0 {
t.Errorf("after an XID: %+v / %+v", got[0], got[1])
}
// Within the minute, the last reading.
journal = ""
now = now.Add(10 * time.Second)
if again := h.Sample(t.Context()); again[1].XIDs != 1 {
t.Errorf("%+v", again[1])
}
}
func TestBusIDsAreReadAsTheKernelWritesThem(t *testing.T) {
for in, want := range map[string]string{"00000000:3B:00.0": "3b:00", "0000:3b:00": "3b:00", "0000:5e:00.0": "5e:00"} {
if got := busOf(in); got != want {
t.Errorf("%s: %s, want %s", in, got, want)
}
}
}
+52 -7
View File
@@ -41,6 +41,14 @@ type GPUSensor struct {
// is running free. Idle and application clock settings are left out;
// they are not the card being held back.
Throttle []string
// How busy it is, read with the rest every sample. Discovery reads the
// same once, at start, which made a busy card look as idle as it was
// when the agent came up. -1 is not reported by the driver; zero
// MemoryTotalMB is a sample without these at all.
UtilPercent, MemUtilPercent float64
MemoryUsedMB, MemoryTotalMB float64
SMClockMHz float64
ECCUncorrected float64
}
// sensorEvery bounds how often nvidia-smi is run. Heartbeats come more often
@@ -187,13 +195,33 @@ var throttleBits = []struct {
// gpus asks nvidia-smi for each card's heat and power. The memory temperature
// is not known to every driver, and a field nvidia-smi does not know fails the
// whole query, so it is asked for first and dropped on failure.
// gpuFields are what each sample asks nvidia-smi for, in order. The busy
// figures follow the throttle reasons; the memory temperature is last, being
// the one field a driver may reject outright.
const (
gpuBase = "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active"
gpuBusy = ",utilization.gpu,utilization.memory,memory.used,memory.total,clocks.sm,ecc.errors.uncorrected.volatile.total"
gpuHBMTC = ",temperature.memory"
)
func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
withMem := "index,temperature.gpu,power.draw,enforced.power.limit,clocks_throttle_reasons.active,temperature.memory"
out, err := s.smi(ctx, withMem)
if err != nil {
if out, err = s.smi(ctx, strings.TrimSuffix(withMem, ",temperature.memory")); err != nil {
return nil
// Most fields first, fewer on a driver that refuses one: a query naming
// a field the driver does not know fails as a whole.
var out []byte
var busy bool
for _, q := range []struct {
fields string
busy bool
}{{gpuBase + gpuBusy + gpuHBMTC, true}, {gpuBase + gpuBusy, true}, {gpuBase + gpuHBMTC, false}, {gpuBase, false}} {
var err error
if out, err = s.smi(ctx, q.fields); err == nil {
busy = q.busy
break
}
out = nil
}
if out == nil {
return nil
}
var gs []GPUSensor
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
@@ -216,14 +244,31 @@ func (s *SensorSampler) gpus(ctx context.Context) []GPUSensor {
}
}
}
if len(f) > 5 {
g.MemoryTempC = num(f[5])
rest := f[5:]
if busy && len(rest) >= 6 {
g.UtilPercent, g.MemUtilPercent = numOrUnknown(rest[0]), numOrUnknown(rest[1])
g.MemoryUsedMB, g.MemoryTotalMB = num(rest[2]), num(rest[3])
g.SMClockMHz, g.ECCUncorrected = num(rest[4]), numOrUnknown(rest[5])
rest = rest[6:]
}
if len(rest) > 0 {
g.MemoryTempC = num(rest[0])
}
gs = append(gs, g)
}
return gs
}
// numOrUnknown is a figure, or -1 where nvidia-smi says it has none: a card
// without ECC has no error count, which is not the same as none.
func numOrUnknown(s string) float64 {
v, err := strconv.ParseFloat(s, 64)
if err != nil || v < 0 {
return -1
}
return v
}
// num reads a figure nvidia-smi may give as "[N/A]" or "N/A".
func num(s string) float64 {
v, err := strconv.ParseFloat(s, 64)
+41
View File
@@ -167,3 +167,44 @@ func TestADriverThatDoesNotKnowMemoryTemperatureStillReportsTheRest(t *testing.T
t.Errorf("%+v", got.GPUs)
}
}
func TestEachSampleSaysHowBusyEachGPUIs(t *testing.T) {
// What the card is doing now, every sample: discovery read it once, at
// start, and a card busy since looked as idle as it was then.
smi := func(_ context.Context, fields string) ([]byte, error) {
if !strings.Contains(fields, "utilization.gpu") {
return nil, errors.New("asked without how busy the cards are")
}
return []byte(
"0, 71, 612.45, 700.00, 0x0000000000000000, 97, 64, 71234, 81559, 1980, 0, 84\n" +
"1, 30, 70.10, 700.00, 0x0000000000000001, 0, 0, 512, 81559, 345, [N/A], [N/A]\n"), nil
}
now := time.Now()
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
if len(got) != 2 {
t.Fatalf("%+v", got)
}
g0, g1 := got[0], got[1]
if g0.UtilPercent != 97 || g0.MemUtilPercent != 64 || g0.MemoryUsedMB != 71234 || g0.MemoryTotalMB != 81559 ||
g0.SMClockMHz != 1980 || g0.ECCUncorrected != 0 || g0.MemoryTempC != 84 {
t.Errorf("card 0 %+v", g0)
}
// Idle is zero; a count the card does not keep is unknown, not zero.
if g1.UtilPercent != 0 || g1.ECCUncorrected != -1 || g1.MemoryTempC != 0 {
t.Errorf("card 1 %+v", g1)
}
}
func TestADriverThatDoesNotKnowTheBusyFieldsStillReportsHeatAndPower(t *testing.T) {
smi := func(_ context.Context, fields string) ([]byte, error) {
if strings.Contains(fields, "utilization.gpu") {
return nil, errors.New(`Field "ecc.errors.uncorrected.volatile.total" is not a valid field to query.`)
}
return []byte("0, 66, 250.0, 300.0, 0x0000000000000000, 70\n"), nil
}
now := time.Now()
got := sampler(t.TempDir(), smi, &now).Sample(t.Context()).GPUs
if len(got) != 1 || got[0].TempC != 66 || got[0].MemoryTempC != 70 || got[0].MemoryTotalMB != 0 {
t.Errorf("%+v", got)
}
}
+23
View File
@@ -585,6 +585,10 @@ func loadFrom(l *llmgwv1.NodeLoad, at time.Time) *inventory.Load {
out.GPUs = append(out.GPUs, inventory.GPUSensor{
Index: int(g.GetIndex()), TempC: g.GetTempC(), MemoryTempC: g.GetMemoryTempC(),
PowerW: g.GetPowerW(), PowerLimitW: g.GetPowerLimitW(), Throttle: g.GetThrottle(),
UtilPercent: g.GetUtilizationPercent(), MemUtilPercent: g.GetMemoryUtilizationPercent(),
MemoryUsedMB: g.GetMemoryUsedMb(), MemoryTotalMB: g.GetMemoryTotalMb(),
SMClockMHz: g.GetSmClockMhz(), ECCUncorrected: g.GetEccUncorrected(),
Wear: wearFrom(g),
})
}
return out
@@ -961,3 +965,22 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
// wearFrom is a card's wear, when the agent read it.
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
if !g.GetHasWear() {
return nil
}
w := &inventory.GPUWear{
PCIeReplays: g.GetPcieReplays(), RetiredSBE: g.GetRetiredPagesSbe(), RetiredDBE: g.GetRetiredPagesDbe(),
RetirePending: g.GetRetirementPending(), RemappedCorrectable: g.GetRemappedRowsCorrectable(),
RemappedUncorrectable: g.GetRemappedRowsUncorrectable(), RemapPending: g.GetRemapPending(),
RemapFailure: g.GetRemapFailure(), NVLinkCRC: g.GetNvlinkCrcErrors(), NVLinkReplay: g.GetNvlinkReplayErrors(),
NVLinkRecovery: g.GetNvlinkRecoveryErrors(), XIDs: g.GetXidErrors(), LastXID: int(g.GetLastXid()),
LastXIDText: g.GetLastXidText(),
}
if u := g.GetLastXidUnix(); u > 0 {
w.LastXIDAt = time.Unix(u, 0)
}
return w
}
+14
View File
@@ -417,3 +417,17 @@ func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
// EngineChoices are the engine builds chosen per node, or nil.
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
// ModelOwners maps each model to the part of the tenancy tree its capacity
// counts against; a model owned by nobody is not in it.
func (c *Controller) ModelOwners() map[string]string {
out := map[string]string{}
if cat := c.Catalog(); cat != nil {
for _, m := range cat.Models() {
if m.Owner != "" {
out[m.Name] = m.Owner
}
}
}
return out
}
+147 -14
View File
@@ -73,6 +73,18 @@ type Collector struct {
gpuMemoryTotal *prometheus.Desc
gpuMemoryUsed *prometheus.Desc
gpuUtil *prometheus.Desc
gpuAssigned *prometheus.Desc
gpuPCIeReplay *prometheus.Desc
gpuRetired *prometheus.Desc
gpuRetirePend *prometheus.Desc
gpuRemapped *prometheus.Desc
gpuRemapPend *prometheus.Desc
gpuRemapFail *prometheus.Desc
gpuNVLinkErr *prometheus.Desc
gpuXID *prometheus.Desc
gpuLastXID *prometheus.Desc
gpuMemUtil *prometheus.Desc
gpuSMClock *prometheus.Desc
gpuTemp *prometheus.Desc
gpuMemTemp *prometheus.Desc
gpuPower *prometheus.Desc
@@ -149,6 +161,33 @@ func New(src Source) *Collector {
"How busy a GPU was over the last sample period. Absent when the driver does not "+
"report it, which is not the same as idle. Memory used says a model is loaded; "+
"this says whether it is doing anything.", gpu, nil),
gpuPCIeReplay: prometheus.NewDesc("llmgw_gpu_pcie_replays",
"PCIe link retries on a GPU's link since the driver loaded; one that keeps climbing is a riser, slot or cable.", gpu, nil),
gpuRetired: prometheus.NewDesc("llmgw_gpu_retired_pages",
"Memory pages retired, by cause (sbe, dbe), on GPUs before Ampere.", append(gpu, "cause"), nil),
gpuRetirePend: prometheus.NewDesc("llmgw_gpu_retirement_pending",
"1 when retired pages wait for a GPU reset to take effect.", gpu, nil),
gpuRemapped: prometheus.NewDesc("llmgw_gpu_remapped_rows",
"Memory rows remapped, by kind (correctable, uncorrectable), on Ampere and later.", append(gpu, "kind"), nil),
gpuRemapPend: prometheus.NewDesc("llmgw_gpu_remap_pending",
"1 when remapped rows wait for a GPU reset to take effect.", gpu, nil),
gpuRemapFail: prometheus.NewDesc("llmgw_gpu_remap_failure",
"1 when a GPU has run out of spare memory rows: it needs replacing.", gpu, nil),
gpuNVLinkErr: prometheus.NewDesc("llmgw_gpu_nvlink_errors",
"NVLink errors summed over a GPU's links, by kind (crc, replay, recovery).", append(gpu, "kind"), nil),
gpuXID: prometheus.NewDesc("llmgw_gpu_xid_errors",
"The driver's XID error events for a GPU since its node's agent started.", gpu, nil),
gpuLastXID: prometheus.NewDesc("llmgw_gpu_last_xid",
"The code of a GPU's last XID error since its agent started; absent when there was none.", gpu, nil),
gpuAssigned: prometheus.NewDesc("llmgw_gpu_assigned",
"1 for each GPU a replica holds: which model, which replica, and whose quota it counts against. "+
"Joined on node and gpu with the other llmgw_gpu_ series, it says what each model's cards are doing.",
[]string{"node", "gpu", "model", "replica", "owner"}, nil),
gpuMemUtil: prometheus.NewDesc("llmgw_gpu_memory_utilization_percent",
"How much of the sample period a GPU's memory was being read or written: "+
"decode is bound by it, so a card busy here and idle above is waiting on memory.", gpu, nil),
gpuSMClock: prometheus.NewDesc("llmgw_gpu_sm_clock_mhz",
"A GPU's streaming multiprocessor clock now; one held low under load is the card being throttled.", gpu, nil),
gpuMemTemp: prometheus.NewDesc("llmgw_gpu_memory_temperature_celsius",
"A GPU's memory (HBM) temperature, where the driver reports it.", gpu, nil),
gpuPower: prometheus.NewDesc("llmgw_gpu_power_watts",
@@ -271,7 +310,9 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
c.nodeCPU, c.nodeCPUs, c.nodeLoad, c.nodeMemUsed, c.nodeMemAvail, c.shardCPU, c.shardMemory,
c.ibActive, c.ibRate, c.ibCounter, c.fabricSMs, c.fabricMaster, c.fabricProblems,
c.gpuNVLJoined, c.imexUp,
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuTemp, c.gpuHealthy,
c.gpuMemoryTotal, c.gpuMemoryUsed, c.gpuUtil, c.gpuMemUtil, c.gpuSMClock, c.gpuAssigned, c.gpuTemp, c.gpuHealthy,
c.gpuPCIeReplay, c.gpuRetired, c.gpuRetirePend, c.gpuRemapped, c.gpuRemapPend, c.gpuRemapFail,
c.gpuNVLinkErr, c.gpuXID, c.gpuLastXID,
c.gpuMemTemp, c.gpuPower, c.gpuPowerLimit, c.gpuThrottled, c.nodeCPUTemp, c.nodeCPUPower, c.nodeInletTemp,
c.gpuNVLink, c.gpuECC,
c.replicaState, c.replicaReady, c.replicaIdle,
@@ -287,6 +328,7 @@ func (c *Collector) Describe(ch chan<- *prometheus.Desc) {
func (c *Collector) Collect(ch chan<- prometheus.Metric) {
now := time.Now()
c.collectQuotas(ch)
c.collectAssignments(ch)
loads := map[string]*inventory.Load{}
for _, n := range c.src.Nodes() {
@@ -388,20 +430,28 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
byClass[class]++
labels := []string{n.ID, strconv.Itoa(g.Index), class}
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
float64(g.MemoryTotalMB)*1024*1024, labels...)
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
float64(g.MemoryUsedMB)*1024*1024, labels...)
if g.UtilizationPercent > 0 {
// Only when reported. A driver too old to give it would
// otherwise show every card at a flat zero, which reads as an
// idle fleet rather than a missing measurement.
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
float64(g.UtilizationPercent), labels...)
s, fresh := sensorFor(n.Load, g.Index, now)
if fresh && s.Busy() {
// What the card is doing now, sampled every heartbeat.
c.gpuBusy(ch, s, labels)
} else {
// An agent too old to sample it: what it read at
// registration, which is all there is.
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
float64(g.MemoryTotalMB)*1024*1024, labels...)
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
float64(g.MemoryUsedMB)*1024*1024, labels...)
if g.UtilizationPercent > 0 {
// Only when reported. A driver too old to give it would
// otherwise show every card at a flat zero, which reads as
// an idle fleet rather than a missing measurement.
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue,
float64(g.UtilizationPercent), labels...)
}
}
// The sensors' fresh reading when there is one; the temperature
// read at registration otherwise, which is all an older agent sends.
if s, ok := sensorFor(n.Load, g.Index, now); ok {
if fresh {
c.gpuSensor(ch, s, labels)
} else if g.Health.TemperatureC > 0 {
ch <- prometheus.MustNewConstMetric(c.gpuTemp, prometheus.GaugeValue,
@@ -414,8 +464,11 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
ch <- prometheus.MustNewConstMetric(c.gpuHealthy, prometheus.GaugeValue, healthy, labels...)
ch <- prometheus.MustNewConstMetric(c.gpuNVLink, prometheus.GaugeValue,
float64(len(g.NVLinkPeers)), labels...)
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue,
float64(g.Health.ECCErrors), labels...)
ecc := float64(g.Health.ECCErrors)
if fresh && s.Busy() && s.ECCUncorrected >= 0 {
ecc = s.ECCUncorrected
}
ch <- prometheus.MustNewConstMetric(c.gpuECC, prometheus.GaugeValue, ecc, labels...)
}
for class, n2 := range byClass {
ch <- prometheus.MustNewConstMetric(c.nodeGPUs, prometheus.GaugeValue,
@@ -521,6 +574,12 @@ func b2f(b bool) float64 {
return 0
}
// OwnerSource says whose quota each model counts against, for labelling
// what its replicas hold. Asked for by type assertion.
type OwnerSource interface {
ModelOwners() map[string]string
}
// QuotaSource adds the tenancy tree's capacity quotas. Asked for by type
// assertion, like HealthSource.
type QuotaSource interface {
@@ -569,6 +628,23 @@ func sensorFor(l *inventory.Load, index int, now time.Time) (inventory.GPUSensor
return inventory.GPUSensor{}, false
}
// gpuBusy is how busy a card is, from a sample that carries it.
func (c *Collector) gpuBusy(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue, s.MemoryTotalMB*1024*1024, labels...)
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue, s.MemoryUsedMB*1024*1024, labels...)
// Zero is a real reading here — an idle card — and only -1, the driver
// not saying, leaves the series out.
if s.UtilPercent >= 0 {
ch <- prometheus.MustNewConstMetric(c.gpuUtil, prometheus.GaugeValue, s.UtilPercent, labels...)
}
if s.MemUtilPercent >= 0 {
ch <- prometheus.MustNewConstMetric(c.gpuMemUtil, prometheus.GaugeValue, s.MemUtilPercent, labels...)
}
if s.SMClockMHz > 0 {
ch <- prometheus.MustNewConstMetric(c.gpuSMClock, prometheus.GaugeValue, s.SMClockMHz, labels...)
}
}
func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor, labels []string) {
for _, v := range []struct {
d *prometheus.Desc
@@ -581,4 +657,61 @@ func (c *Collector) gpuSensor(ch chan<- prometheus.Metric, s inventory.GPUSensor
for _, reason := range s.Throttle {
ch <- prometheus.MustNewConstMetric(c.gpuThrottled, prometheus.GaugeValue, 1, append(labels, reason)...)
}
if w := s.Wear; w != nil {
c.gpuWear(ch, w, labels)
}
}
// gpuWear is what wears a card out. A count the card does not keep (-1) is
// left out, not published as zero.
func (c *Collector) gpuWear(ch chan<- prometheus.Metric, w *inventory.GPUWear, labels []string) {
gauge := func(d *prometheus.Desc, v float64, extra ...string) {
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, append(append([]string{}, labels...), extra...)...)
}
flag := func(b bool) float64 {
if b {
return 1
}
return 0
}
if w.PCIeReplays >= 0 {
gauge(c.gpuPCIeReplay, float64(w.PCIeReplays))
}
if w.RetiredSBE >= 0 || w.RetiredDBE >= 0 {
gauge(c.gpuRetired, float64(max(w.RetiredSBE, 0)), "sbe")
gauge(c.gpuRetired, float64(max(w.RetiredDBE, 0)), "dbe")
gauge(c.gpuRetirePend, flag(w.RetirePending))
}
if w.RemappedCorrectable >= 0 || w.RemappedUncorrectable >= 0 {
gauge(c.gpuRemapped, float64(max(w.RemappedCorrectable, 0)), "correctable")
gauge(c.gpuRemapped, float64(max(w.RemappedUncorrectable, 0)), "uncorrectable")
gauge(c.gpuRemapPend, flag(w.RemapPending))
gauge(c.gpuRemapFail, flag(w.RemapFailure))
}
gauge(c.gpuNVLinkErr, float64(w.NVLinkCRC), "crc")
gauge(c.gpuNVLinkErr, float64(w.NVLinkReplay), "replay")
gauge(c.gpuNVLinkErr, float64(w.NVLinkRecovery), "recovery")
gauge(c.gpuXID, float64(w.XIDs))
if w.XIDs > 0 {
gauge(c.gpuLastXID, float64(w.LastXID))
}
}
// collectAssignments says which replica holds each GPU.
func (c *Collector) collectAssignments(ch chan<- prometheus.Metric) {
var owners map[string]string
if o, ok := c.src.(OwnerSource); ok {
owners = o.ModelOwners()
}
for _, r := range c.src.Replicas() {
if !r.State.Counts() {
continue
}
for _, sh := range r.Shards {
for _, g := range sh.GPUIndices {
ch <- prometheus.MustNewConstMetric(c.gpuAssigned, prometheus.GaugeValue, 1,
sh.NodeID, strconv.Itoa(g), r.Model, r.ID, owners[r.Model])
}
}
}
}
@@ -7,6 +7,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
)
func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
@@ -39,3 +40,90 @@ func TestSensorsArePublishedFreshAndOnlyWhenMeasured(t *testing.T) {
}
}
}
func TestHowBusyAGPUIsComesFromTheLatestSampleNotRegistration(t *testing.T) {
// Discovery read 87% and 70 GB used when the agent came up; the card has
// been idle since. The sample says so, zero included.
reg := discovery.GPU{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
MemoryUsedMB: 70000, UtilizationPercent: 87}
n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{reg},
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 35,
UtilPercent: 0, MemUtilPercent: 0, MemoryUsedMB: 512, MemoryTotalMB: 81559,
SMClockMHz: 345, ECCUncorrected: 2}}}}
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
for _, want := range []string{
"llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
"llmgw_gpu_memory_utilization_percent class=h100-80g gpu=0 node=g1 -> 0\n",
"llmgw_gpu_memory_used_bytes class=h100-80g gpu=0 node=g1 -> 536870912\n",
"llmgw_gpu_sm_clock_mhz class=h100-80g gpu=0 node=g1 -> 345\n",
"llmgw_gpu_ecc_errors class=h100-80g gpu=0 node=g1 -> 2\n",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %s in:\n%s", want, out)
}
}
if strings.Contains(out, "-> 87\n") {
t.Error("the utilization read at registration was published over the sample's")
}
// An agent too old to sample it: registration's figures, and no zero
// standing in for a utilization the driver did not give.
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35}}
out = gather(t, fakeSource{nodes: []inventory.Node{n}})
if !strings.Contains(out, "llmgw_gpu_utilization_percent class=h100-80g gpu=0 node=g1 -> 87") {
t.Errorf("an older agent's registration figure was dropped:\n%s", out)
}
n.Load.GPUs = []inventory.GPUSensor{{Index: 0, TempC: 35, UtilPercent: -1, MemUtilPercent: -1, MemoryTotalMB: 81559}}
if out := gather(t, fakeSource{nodes: []inventory.Node{n}}); strings.Contains(out, "llmgw_gpu_utilization_percent ") {
t.Errorf("a utilization the driver did not report was published:\n%s", out)
}
}
// ownedSource adds owners to fakeSource.
type ownedSource struct{ fakeSource }
func (ownedSource) ModelOwners() map[string]string { return map[string]string{"chat": "acme/ml"} }
func TestEachGPUAReplicaHoldsSaysWhoseItIs(t *testing.T) {
src := ownedSource{fakeSource{replicas: []reconciler.Replica{
{ID: "chat-1", Model: "chat", State: reconciler.Ready, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{0, 1}}}},
{ID: "old-1", Model: "old", State: reconciler.Stopped, Shards: []reconciler.Shard{{NodeID: "g1", GPUIndices: []int{2}}}},
}}}
out := gather(t, src)
for _, want := range []string{
"llmgw_gpu_assigned gpu=0 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
"llmgw_gpu_assigned gpu=1 model=chat node=g1 owner=acme/ml replica=chat-1 -> 1",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %s in:\n%s", want, out)
}
}
if strings.Contains(out, "replica=old-1") {
t.Error("a stopped replica's GPU was counted as held")
}
}
func TestAGPUsWearIsPublishedAndWhatItDoesNotKeepIsLeftOut(t *testing.T) {
n := inventory.Node{ID: "g1", State: inventory.Ready,
GPUs: []discovery.GPU{{Index: 0, Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559}},
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 40, MemoryTotalMB: 81559,
Wear: &inventory.GPUWear{PCIeReplays: 12, RetiredSBE: -1, RetiredDBE: -1, RemappedCorrectable: 2,
RemappedUncorrectable: 1, RemapPending: true, NVLinkCRC: 7, XIDs: 3, LastXID: 79}}}}}
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
for _, want := range []string{
"llmgw_gpu_pcie_replays class=h100-80g gpu=0 node=g1 -> 12\n",
"llmgw_gpu_remapped_rows class=h100-80g gpu=0 kind=uncorrectable node=g1 -> 1\n",
"llmgw_gpu_remap_pending class=h100-80g gpu=0 node=g1 -> 1\n",
"llmgw_gpu_nvlink_errors class=h100-80g gpu=0 kind=crc node=g1 -> 7\n",
"llmgw_gpu_xid_errors class=h100-80g gpu=0 node=g1 -> 3\n",
"llmgw_gpu_last_xid class=h100-80g gpu=0 node=g1 -> 79\n",
} {
if !strings.Contains(out, want) {
t.Errorf("missing %s", want)
}
}
// An H100 keeps no retired pages; none are published.
if strings.Contains(out, "llmgw_gpu_retired_pages") {
t.Errorf("a count the card does not keep was published:\n%s", out)
}
}
+67
View File
@@ -3,6 +3,7 @@
package inventory
import (
"fmt"
"maps"
"slices"
"strings"
@@ -319,8 +320,74 @@ type GPUSensor struct {
PowerW float64 `json:"power_w,omitempty"`
PowerLimitW float64 `json:"power_limit_w,omitempty"`
Throttle []string `json:"throttle,omitempty"`
// How busy the card is, as of this sample; -1 is not reported. Zero
// MemoryTotalMB is an agent too old to send these, whose figures are
// the ones it read at registration.
UtilPercent float64 `json:"util_percent,omitempty"`
MemUtilPercent float64 `json:"mem_util_percent,omitempty"`
MemoryUsedMB float64 `json:"memory_used_mb,omitempty"`
MemoryTotalMB float64 `json:"memory_total_mb,omitempty"`
SMClockMHz float64 `json:"sm_clock_mhz,omitempty"`
ECCUncorrected float64 `json:"ecc_uncorrected,omitempty"`
// Wear is what wears the card out, read every minute; nil from an agent
// that does not read it.
Wear *GPUWear `json:"wear,omitempty"`
}
// GPUWear is a card's wear and failures. Counts of -1 are not reported for
// it: a card before Ampere has no remapped rows, one after no retired pages.
type GPUWear struct {
PCIeReplays int64 `json:"pcie_replays"`
RetiredSBE int64 `json:"retired_pages_sbe"`
RetiredDBE int64 `json:"retired_pages_dbe"`
RetirePending bool `json:"retire_pending,omitempty"`
RemappedCorrectable int64 `json:"remapped_rows_correctable"`
RemappedUncorrectable int64 `json:"remapped_rows_uncorrectable"`
RemapPending bool `json:"remap_pending,omitempty"`
RemapFailure bool `json:"remap_failure,omitempty"`
NVLinkCRC int64 `json:"nvlink_crc_errors"`
NVLinkReplay int64 `json:"nvlink_replay_errors"`
NVLinkRecovery int64 `json:"nvlink_recovery_errors"`
XIDs int64 `json:"xid_errors"`
LastXID int `json:"last_xid,omitempty"`
LastXIDText string `json:"last_xid_text,omitempty"`
LastXIDAt time.Time `json:"last_xid_at,omitempty"`
}
// Problems says, in a few words each, what is wrong with the card; empty
// when nothing is.
func (w *GPUWear) Problems() []string {
if w == nil {
return nil
}
var out []string
if w.RemapFailure {
out = append(out, "out of spare memory rows: replace the card")
}
if w.RemapPending {
out = append(out, "memory rows remapped, waiting for a reset")
}
if w.RetirePending {
out = append(out, "memory pages waiting for a reset to be retired")
}
if w.RemappedUncorrectable > 0 {
out = append(out, fmt.Sprintf("%d rows remapped for uncorrectable errors", w.RemappedUncorrectable))
}
if w.RetiredDBE > 0 {
out = append(out, fmt.Sprintf("%d pages retired for double-bit errors", w.RetiredDBE))
}
if w.XIDs > 0 {
out = append(out, fmt.Sprintf("XID %d (%s)", w.LastXID, w.LastXIDText))
}
if w.NVLinkCRC > 0 || w.NVLinkReplay > 0 || w.NVLinkRecovery > 0 {
out = append(out, fmt.Sprintf("NVLink errors: %d CRC, %d replay, %d recovery", w.NVLinkCRC, w.NVLinkReplay, w.NVLinkRecovery))
}
return out
}
// Busy reports whether the sample carries how busy the card is.
func (s GPUSensor) Busy() bool { return s.MemoryTotalMB > 0 }
// ShardLoad is one engine's use: cores busy, and resident memory.
type ShardLoad struct {
CPUCores float64
+3
View File
@@ -203,6 +203,9 @@ func (r *Router) Handler() http.Handler {
}
pages.RegisterAdminQuotas(mux, r.store, r.acct)
// And the report of what that tree actually used.
if r.history != nil {
pages.SetHistory(r.history)
}
pages.RegisterAdminUsage(mux, r.store, r.acct)
}
+6
View File
@@ -64,6 +64,9 @@ type AccountPages struct {
// hasQuotas is set when /admin/quotas is registered.
hasQuotas bool
// capacity sets capacity quotas, from the quotas page; nil leaves them off.
// history is the metrics store, for what only it knows: the GPU time
// each owner held. Nil without one.
history History
capacity CapacityAdmin
// personalOff refuses personal keys; see SetPersonalOff.
personalOff bool
@@ -651,3 +654,6 @@ func errorRef() string {
_, _ = rand.Read(b[:])
return "err-" + hex.EncodeToString(b[:])
}
// SetHistory gives the pages the metrics store.
func (p *AccountPages) SetHistory(h History) { p.history = h }
+44
View File
@@ -3,6 +3,7 @@ package web
import (
"context"
"net/http"
"sort"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
@@ -80,6 +81,12 @@ type adminUsageData struct {
// chart along the bottom reads as "nobody spent anything".
Priced bool
// GPUHeld and GPUHours are the cards each owner's models held over the
// window, from the metrics store; empty without one, or with no owned
// model on a GPU.
GPUHeld TimeSeries
GPUHours []gpuHoursRow
Error string
Version string
Now string
@@ -135,9 +142,46 @@ func (p *AccountPages) renderUsage(w http.ResponseWriter, r *http.Request,
}
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
render(w, "admin_usage", d)
}
// gpuHoursRow is the GPU time one owner's models held over the window.
type gpuHoursRow struct {
Owner string
Hours float64
// Now is how many cards they hold at the end of the window.
Now float64
}
// gpuHours is the cards held by each owner's models over the window, drawn
// and summed into GPU-hours. From the metrics store, because what a model
// holds is placement's, not the request log's: a model nobody asks still
// holds its cards. As far back as the store keeps history, which for a
// default Prometheus is fifteen days.
func (p *AccountPages) gpuHours(ctx context.Context, win usageWindow) (TimeSeries, []gpuHoursRow) {
if p.history == nil {
return TimeSeries{}, nil
}
series, err := p.history.RangeQuery(ctx, `sum by (owner) (llmgw_gpu_assigned)`, "owner", win.Span, win.Width)
if err != nil || len(series) == 0 {
return TimeSeries{}, nil
}
var rows []gpuHoursRow
for i := range series {
if series[i].Name == "" {
series[i].Name = "no owner"
}
row := gpuHoursRow{Owner: series[i].Name, Now: series[i].Last()}
// Each point is the cards held at that instant, standing for the
// step that follows it.
row.Hours = series[i].Sum() * win.Width.Hours()
rows = append(rows, row)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Hours > rows[j].Hours })
return LineChart(series, "", 6), rows
}
// seriesOf turns each group's buckets into a line of one measure.
//
// One pass per measure rather than one query per measure: the buckets are
+30
View File
@@ -9,6 +9,7 @@ import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -162,3 +163,32 @@ func TestAPersonWithNoAccountIsNamedRatherThanBlank(t *testing.T) {
t.Errorf("unowned traffic was left unlabelled; got %s", body)
}
}
func TestTheUsagePageCountsTheGPUTimeEachOwnerHeld(t *testing.T) {
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
// Over a day in 20-minute steps: team:research held 8 cards throughout,
// nobody's model 2 for the last three steps.
res, none := promq.Series{Name: "team:research"}, promq.Series{Name: ""}
for i := range 72 {
res.Points = append(res.Points, promq.Point{Value: 8})
if i >= 69 {
none.Points = append(none.Points, promq.Point{Value: 2})
}
}
h := &fakeHistory{byExpr: map[string][]promq.Series{`sum by (owner) (llmgw_gpu_assigned)`: {res, none}}}
p.SetHistory(h)
mux := http.NewServeMux()
p.RegisterAdminUsage(mux, &fakeScopeSeries{}, nil)
body := mirrorGet(t, mux, "/admin/usage?window=24h").Body.String()
for _, want := range []string{"GPU time, by owner", "<td>team:research</td><td class=\"num\">192.0</td>",
"<td>no owner</td><td class=\"num\">2.0</td>"} {
if !strings.Contains(body, want) {
t.Errorf("the page lacks %q", want)
}
}
// Without a metrics store, nothing is said about GPU time.
if body := mirrorGet(t, usagePage(t, &fakeScopeSeries{}, nil), "/admin/usage").Body.String(); strings.Contains(body, "GPU time") {
t.Error("GPU time shown with no metrics store to read it from")
}
}
+15
View File
@@ -20,6 +20,13 @@ type DashboardData struct {
CPU cpuCapacity
// Menu is the shared top menu; Nav below is what the cluster page adds.
Menu Nav
// GPUMap is every card coloured by one figure, and IdleGPUs the replicas
// holding cards that are doing nothing; both from the last heartbeats.
GPUMap *gpuMap
IdleGPUs []idleHolding
// Efficiency is what each model gets from its cards, from the metrics
// store; nil without one.
Efficiency []modelEfficiency
// Notice and Problem are what the last change made from this page did,
// carried over the redirect back to it.
Notice, Problem string
@@ -198,9 +205,17 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
d.AnyFabrics = d.AnyFabrics || len(n.Fabrics) > 0
}
d.NodePrefix = sharedPrefix(ids)
if d.AnyGPUs {
d.GPUMap = buildGPUMap(d.Nodes, d.Replicas, r.URL.Query().Get("gpumap"), d.NodePrefix)
d.IdleGPUs = idleHoldings(d.Nodes, d.Replicas, d.NodePrefix)
}
for _, m := range d.Models {
d.AnyFallbacks = d.AnyFallbacks || len(m.Fallbacks) > 0
}
if d.AnyGPUs {
d.Fleet.LoadGPU(r.Context(), nav.History, nav.Window)
d.Efficiency = loadEfficiency(r.Context(), nav.History)
}
d.Summary = src.Summary()
if fs, ok := src.(FabricSource); ok {
r := fs.Fabrics()
+26 -1
View File
@@ -24,7 +24,10 @@ type FleetCharts struct {
Queue TimeSeries
KV TimeSeries
Ready TimeSeries
Window string
// GPUBusy and GPUMemory are the fleet's cards by class, for a fleet that
// has them.
GPUBusy, GPUMemory TimeSeries
Window string
// Err is set when the metrics store could not be read. Shown as a line of
// text where the charts would be: an empty panel reads as a healthy cluster
// with nothing happening, which is the opposite of not knowing.
@@ -90,3 +93,25 @@ func LoadFleetCharts(ctx context.Context, h History, window time.Duration,
}
return out
}
// LoadGPU adds the fleet's GPUs, by class: how busy on average, and how full
// their memory. Asked only of a fleet with GPUs. A failure leaves these two
// empty and the rest of the page as it was; they are not what the other
// panels' all-or-nothing is about.
func (f *FleetCharts) LoadGPU(ctx context.Context, h History, window time.Duration) {
if f == nil || h == nil || f.Err != "" {
return
}
if window <= 0 {
window = time.Hour
}
step := window / 60
if s, err := h.RangeQuery(ctx, `avg by (class) (llmgw_gpu_utilization_percent) / 100`, "class", window, step); err == nil {
f.GPUBusy = LineChart(s, "%", 6)
}
if s, err := h.RangeQuery(ctx,
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`,
"class", window, step); err == nil {
f.GPUMemory = LineChart(s, "%", 6)
}
}
+83
View File
@@ -0,0 +1,83 @@
package web
import (
"context"
"sort"
"time"
)
// modelEfficiency is what a model gets out of the GPUs it holds, over the
// last quarter hour.
type modelEfficiency struct {
Model string
// TokensPerSec is output tokens a second; GPUs the cards it holds now.
TokensPerSec float64
GPUs float64
// Watts is what those cards draw; Util their average utilisation.
Watts, Util float64
// PerGPU and PerWh are the two ways of saying how much work the cards
// do: output tokens a second per card, and per watt-hour drawn.
PerGPU, PerWh float64
}
// The queries behind the table. Each model's cards are found by joining the
// GPU series to llmgw_gpu_assigned on the card, which carries the model.
var efficiencyQueries = []struct {
expr string
into func(*modelEfficiency, float64)
}{
{`sum by (model) (rate(llmgw_tokens_total{direction="completion"}[15m]))`,
func(e *modelEfficiency, v float64) { e.TokensPerSec = v }},
{`count by (model) (llmgw_gpu_assigned)`,
func(e *modelEfficiency, v float64) { e.GPUs = v }},
{`sum by (model) (llmgw_gpu_power_watts * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
func(e *modelEfficiency, v float64) { e.Watts = v }},
{`avg by (model) (llmgw_gpu_utilization_percent * on (node, gpu) group_left (model) llmgw_gpu_assigned)`,
func(e *modelEfficiency, v float64) { e.Util = v }},
}
// loadEfficiency is the table, best per card first; nil when the metrics
// store cannot be read or no model holds a GPU.
func loadEfficiency(ctx context.Context, h History) []modelEfficiency {
if h == nil {
return nil
}
byModel := map[string]*modelEfficiency{}
for _, q := range efficiencyQueries {
// The last of a short range: what History offers, and the one point
// the table needs.
series, err := h.RangeQuery(ctx, q.expr, "model", 10*time.Minute, 5*time.Minute)
if err != nil {
return nil
}
for _, s := range series {
if len(s.Points) == 0 {
continue
}
e := byModel[s.Name]
if e == nil {
e = &modelEfficiency{Model: s.Name}
byModel[s.Name] = e
}
q.into(e, s.Last())
}
}
var out []modelEfficiency
for _, e := range byModel {
if e.GPUs == 0 {
continue // a model on the CPU; this table is about cards
}
e.PerGPU = e.TokensPerSec / e.GPUs
if e.Watts > 0 {
e.PerWh = e.TokensPerSec * 3600 / e.Watts
}
out = append(out, *e)
}
sort.Slice(out, func(i, j int) bool {
if out[i].PerGPU != out[j].PerGPU {
return out[i].PerGPU > out[j].PerGPU
}
return out[i].Model < out[j].Model
})
return out
}
+243
View File
@@ -0,0 +1,243 @@
package web
import (
"fmt"
"html/template"
"sort"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
// The GPU map is every card in the fleet, a square each, a row per node,
// coloured by one figure: how busy, how hot, or how full its memory is. A
// hundred cards in a table are a hundred rows to read; on the map the one
// that is not like the others is seen before anything is read.
// gpuMapModes are what the map can be coloured by.
var gpuMapModes = []struct{ Key, Label string }{
{"util", "busy"}, {"memory", "memory"}, {"temp", "temperature"},
}
type gpuMap struct {
By string
Modes []struct{ Key, Label string }
Rows []gpuMapRow
// Legend says what the colours mean in this mode.
Legend string
}
type gpuMapRow struct {
Node string
Cells []gpuCell
}
type gpuCell struct {
Index int
// Model and Replica hold the card; empty is a free card.
Model, Replica string
// Style is the cell's colour, built here from numbers alone, which is
// what makes it safe to put in a style attribute; Title says everything
// known about the card.
Style template.CSS
Title string
// Label is the figure written in the cell.
Label string
// Faulty is a card whose wear needs seeing to.
Faulty bool
}
// idleHolding is a replica holding GPUs that are doing nothing.
type idleHolding struct {
Model, Replica, Node string
GPUs []int
// MaxUtil is the busiest of its cards, now.
MaxUtil float64
// Idle is how long since a request was routed to it, or "" when not
// known.
Idle string
}
// buildGPUMap draws the map from what the nodes last reported and where
// replicas are. Nil for a fleet without GPUs.
func buildGPUMap(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, by, prefix string) *gpuMap {
held := heldBy(reps)
m := &gpuMap{By: by, Modes: gpuMapModes}
if m.By == "" {
m.By = "util"
}
switch m.By {
case "temp":
m.Legend = "Blue is cool, red is 90 °C and over."
case "memory":
m.Legend = "Pale is empty, dark is full. Engines reserve most of a card's memory when they start."
default:
m.By = "util"
m.Legend = "Pale is idle, dark green is fully busy. A dashed border is a card no replica holds."
}
for _, n := range nodes {
if n.GPUTotal == 0 {
continue
}
row := gpuMapRow{Node: strings.TrimPrefix(n.ID, prefix)}
for i := range n.GPUTotal {
c := gpuCell{Index: i}
if h, ok := held[gpuKey(n.ID, i)]; ok {
c.Model, c.Replica = h.model, h.replica
}
var title []string
title = append(title, fmt.Sprintf("%s GPU %d", n.ID, i))
if c.Model != "" {
title = append(title, c.Model+" ("+c.Replica+")")
} else {
title = append(title, "free")
}
s, sampled := sensorOf(n, i)
if sampled {
if s.MemoryTotalMB > 0 && s.UtilPercent >= 0 {
title = append(title, fmt.Sprintf("%.0f%% busy", s.UtilPercent))
}
if s.MemoryTotalMB > 0 {
title = append(title, fmt.Sprintf("%.1f of %.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024))
}
if s.TempC > 0 {
title = append(title, fmt.Sprintf("%.0f °C", s.TempC))
}
if s.PowerW > 0 {
title = append(title, fmt.Sprintf("%.0f W", s.PowerW))
}
if len(s.Throttle) > 0 {
title = append(title, "held back by "+strings.Join(s.Throttle, ", "))
}
title = append(title, s.Problems...)
}
c.Title = strings.Join(title, " · ")
style, label := cellColour(m.By, s, sampled)
if c.Model == "" {
style += ";border-style:dashed"
}
// A card wearing out is outlined whatever the map is coloured
// by: it is the one thing on the map that needs doing.
if len(s.Problems) > 0 {
style += ";border:2px solid #c0392b"
c.Faulty = true
}
c.Style, c.Label = template.CSS(style), label
row.Cells = append(row.Cells, c)
}
m.Rows = append(m.Rows, row)
}
if len(m.Rows) == 0 {
return nil
}
return m
}
// cellColour is a cell's background, and the figure written in it.
func cellColour(by string, s sensorView, sampled bool) (string, string) {
const unknown = "background:var(--panel-2,#eee)"
switch by {
case "temp":
if !sampled || s.TempC <= 0 {
return unknown, ""
}
// 30 °C blue to 90 °C red.
f := clamp01((s.TempC - 30) / 60)
return fmt.Sprintf("background:hsl(%.0f,70%%,55%%)", 220-220*f), fmt.Sprintf("%.0f", s.TempC)
case "memory":
if !sampled || s.MemoryTotalMB <= 0 {
return unknown, ""
}
f := clamp01(s.MemoryUsedMB / s.MemoryTotalMB)
return fmt.Sprintf("background:hsl(265,45%%,%.0f%%)", 90-55*f), fmt.Sprintf("%.0f", 100*f)
}
if !sampled || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
return unknown, ""
}
f := clamp01(s.UtilPercent / 100)
return fmt.Sprintf("background:hsl(140,%.0f%%,%.0f%%)", 15+50*f, 88-50*f), fmt.Sprintf("%.0f", s.UtilPercent)
}
// idleHoldings are the replicas whose GPUs all read under 5% busy now:
// capacity held and not asked for, the first place to find room.
func idleHoldings(nodes []adminapi.NodeView, reps []adminapi.ReplicaView, prefix string) []idleHolding {
byID := map[string]adminapi.NodeView{}
for _, n := range nodes {
byID[n.ID] = n
}
var out []idleHolding
for _, r := range reps {
if r.State != "ready" {
continue
}
for _, sh := range r.Shards {
if len(sh.GPUs) == 0 {
continue
}
h := idleHolding{Model: r.Model, Replica: r.ID, Node: strings.TrimPrefix(sh.Node, prefix), GPUs: sh.GPUs}
idle, known := true, true
for _, g := range sh.GPUs {
s, ok := sensorOf(byID[sh.Node], g)
if !ok || s.MemoryTotalMB <= 0 || s.UtilPercent < 0 {
known = false
break
}
h.MaxUtil = max(h.MaxUtil, s.UtilPercent)
if s.UtilPercent >= 5 {
idle = false
}
}
if !known || !idle {
continue
}
if r.IdleSec >= 0 {
h.Idle = (time.Duration(r.IdleSec) * time.Second).String()
}
out = append(out, h)
}
}
sort.SliceStable(out, func(i, j int) bool { return len(out[i].GPUs) > len(out[j].GPUs) })
return out
}
type holder struct{ model, replica string }
func heldBy(reps []adminapi.ReplicaView) map[string]holder {
out := map[string]holder{}
for _, r := range reps {
switch r.State {
case "stopped", "failed":
continue
}
for _, sh := range r.Shards {
for _, g := range sh.GPUs {
out[gpuKey(sh.Node, g)] = holder{r.Model, r.ID}
}
}
}
return out
}
func gpuKey(node string, gpu int) string { return fmt.Sprintf("%s/%d", node, gpu) }
// sensorView is a card's last sample.
type sensorView = struct {
TempC, PowerW, UtilPercent, MemoryUsedMB, MemoryTotalMB float64
Throttle []string
Problems []string
}
func sensorOf(n adminapi.NodeView, gpu int) (sensorView, bool) {
if n.Load == nil {
return sensorView{}, false
}
for _, s := range n.Load.GPUs {
if s.Index == gpu {
return sensorView{TempC: s.TempC, PowerW: s.PowerW, UtilPercent: s.UtilPercent,
MemoryUsedMB: s.MemoryUsedMB, MemoryTotalMB: s.MemoryTotalMB, Throttle: s.Throttle,
Problems: s.Wear.Problems()}, true
}
}
return sensorView{}, false
}
+85
View File
@@ -0,0 +1,85 @@
package web
import (
"context"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/promq"
)
// mapFleet is a node of four cards: chat holds two and is idle, code holds
// one and is busy, one is free.
type mapFleet struct{ fakeSource }
func (mapFleet) Nodes() []adminapi.NodeView {
busy := func(i int, util, temp float64) inventory.GPUSensor {
return inventory.GPUSensor{Index: i, UtilPercent: util, TempC: temp, MemoryUsedMB: 40000, MemoryTotalMB: 81559, PowerW: 300}
}
return []adminapi.NodeView{{ID: "gpu07", State: "ready", GPUTotal: 4, GPUFree: 1, Classes: []string{"h100-80g"},
Load: &adminapi.LoadView{GPUs: []inventory.GPUSensor{busy(0, 1, 35), busy(1, 0, 34), busy(2, 93, 81), worn(busy(3, 0, 30))}}}}
}
func worn(s inventory.GPUSensor) inventory.GPUSensor {
s.Wear = &inventory.GPUWear{RemapPending: true, XIDs: 1, LastXID: 79, LastXIDText: "GPU has fallen off the bus."}
return s
}
func (mapFleet) Replicas(string) []adminapi.ReplicaView {
return []adminapi.ReplicaView{
{ID: "chat-1", Model: "chat", State: "ready", IdleSec: 1800, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{0, 1}}}},
{ID: "code-1", Model: "code", State: "ready", IdleSec: 2, Shards: []adminapi.ShardView{{Node: "gpu07", GPUs: []int{2}}}},
}
}
func TestTheClusterPageMapsEveryCardAndNamesTheIdleOnes(t *testing.T) {
_, body := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
if n := strings.Count(body, `class="gpu-cell"`); n != 4 {
t.Errorf("%d cards on the map, want 4", n)
}
for _, want := range []string{
"gpu07 GPU 2 · code (code-1) · 93% busy",
"gpu07 GPU 3 · free",
"memory rows remapped, waiting for a reset · XID 79 (GPU has fallen off the bus.)",
"border:2px solid #c0392b",
"border-style:dashed",
"Cards held and doing nothing",
"<code>chat-1</code></td><td><code>gpu07</code>", "30m0s ago",
} {
if !strings.Contains(body, want) {
t.Errorf("the page lacks %q", want)
}
}
if strings.Contains(body, "<code>code-1</code></td><td><code>gpu07</code>") {
t.Error("a busy replica was listed as idle")
}
_, temp := get(t, Dashboard(mapFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/?gpumap=temp", nil)
if !strings.Contains(temp, "<strong>temperature</strong>") || !strings.Contains(temp, ">81</span>") {
t.Error("the map coloured by temperature does not say so, or show it")
}
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
if strings.Contains(cpu, "gpu-map") {
t.Error("a fleet without GPUs has a GPU map")
}
}
func TestEachModelSaysWhatItGetsFromItsCards(t *testing.T) {
h := &fakeHistory{byExpr: map[string][]promq.Series{}}
for i, vals := range [][2]float64{{120, 30}, {2, 2}, {1200, 400}, {80, 40}} {
h.byExpr[efficiencyQueries[i].expr] = []promq.Series{ts("chat", vals[0], vals[0]), ts("code", vals[1], vals[1])}
}
got := loadEfficiency(context.Background(), h)
if len(got) != 2 || got[0].Model != "chat" {
t.Fatalf("%+v", got)
}
c := got[0]
if c.PerGPU != 60 || c.PerWh != 120*3600/1200.0 || c.Util != 80 {
t.Errorf("chat: %+v", c)
}
if got[1].PerGPU != 15 {
t.Errorf("code: %+v", got[1])
}
}
+36 -4
View File
@@ -40,8 +40,10 @@ type nodeEngine struct {
// nodeCharts are the node's recent history, from the metrics store.
type nodeCharts struct {
CPU, Memory, Load, EngineCPU, EngineMemory, Queue TimeSeries
Window string
Err string
// The GPUs', one line per card; empty on a node without them.
GPUUtil, GPUMemory, GPUTemp, GPUPower TimeSeries
Window string
Err string
}
// nodePanels are the charts, each a query scoped to this node. %s is the
@@ -63,6 +65,23 @@ var nodePanels = []struct {
func(c *nodeCharts, t TimeSeries) { c.EngineMemory = t }},
}
// gpuPanels are the GPUs' charts, one line per card, for a node that has
// them. Read from the heartbeat's samples, not from what the node said when
// it registered.
var gpuPanels = []struct {
expr, unit string
into func(*nodeCharts, TimeSeries)
}{
{`llmgw_gpu_utilization_percent{node=%s} / 100`, "%",
func(c *nodeCharts, t TimeSeries) { c.GPUUtil = t }},
{`llmgw_gpu_memory_used_bytes{node=%s}`, "bytes",
func(c *nodeCharts, t TimeSeries) { c.GPUMemory = t }},
{`llmgw_gpu_temperature_celsius{node=%s}`, "",
func(c *nodeCharts, t TimeSeries) { c.GPUTemp = t }},
{`llmgw_gpu_power_watts{node=%s}`, "",
func(c *nodeCharts, t TimeSeries) { c.GPUPower = t }},
}
// NodePage shows one node. Its charts come from the metrics store when one
// is configured; the current figures come from the node's last heartbeat
// either way.
@@ -115,14 +134,14 @@ func NodePage(src Source, history History, window time.Duration, onAdminListener
sort.Slice(d.Engines, func(i, j int) bool { return d.Engines[i].CPUCores > d.Engines[j].CPUCores })
if history != nil {
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, window, log)
d.Charts = loadNodeCharts(r.Context(), history, id, replicas, d.Node.GPUTotal, window, log)
}
noStore(w)
render(w, "node", d)
}
}
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string,
func loadNodeCharts(ctx context.Context, h History, node string, replicas []string, gpus int,
window time.Duration, log *slog.Logger,
) *nodeCharts {
step := window / 60
@@ -139,6 +158,19 @@ func loadNodeCharts(ctx context.Context, h History, node string, replicas []stri
}
p.into(out, LineChart(series, p.unit, 4))
}
// Every card its own line, up to the eight a node commonly has.
if gpus > 0 {
for _, p := range gpuPanels {
series, err := h.RangeQuery(ctx, fmt.Sprintf(p.expr, quoted), "gpu", window, step)
if err != nil {
continue
}
for i := range series {
series[i].Name = "GPU " + series[i].Name
}
p.into(out, LineChart(series, p.unit, max(gpus, 8)))
}
}
// The engines' queues, for the replicas that are here now.
if len(replicas) > 0 {
re := make([]string, len(replicas))
+65
View File
@@ -205,3 +205,68 @@ func TestANodesSensorsAreOnItsPageAndNothingItCannotMeasure(t *testing.T) {
}
}
}
// gpuSource is a node with two GPUs, one busy and one idle.
type gpuSource struct{ fakeSource }
func (gpuSource) Nodes() []adminapi.NodeView {
n := fakeSource{}.Nodes()[0]
n.GPUTotal = 2
n.Load = &adminapi.LoadView{CPUPercent: 40, CPUs: 64, MemTotal: 1 << 40, MemUsed: 1 << 39,
GPUs: []inventory.GPUSensor{
{Index: 0, TempC: 71, PowerW: 612, PowerLimitW: 700, UtilPercent: 97, MemUtilPercent: 64,
MemoryUsedMB: 72909, MemoryTotalMB: 81559, SMClockMHz: 1980, ECCUncorrected: 0},
{Index: 1, TempC: 30, PowerW: 70, PowerLimitW: 700, UtilPercent: 0, MemUtilPercent: 0,
MemoryUsedMB: 512, MemoryTotalMB: 81559, SMClockMHz: 345, ECCUncorrected: 3},
}}
return []adminapi.NodeView{n}
}
func TestAGPUNodesPageShowsHowBusyEachCardIsAndOverTime(t *testing.T) {
q := func(expr string) string { return strings.Replace(expr, "%s", `"llmgw-node01"`, 1) }
h := &fakeHistory{byExpr: map[string][]promq.Series{
q(`llmgw_gpu_utilization_percent{node=%s} / 100`): {ts("0", 0.9, 0.97), ts("1", 0, 0)},
q(`llmgw_gpu_memory_used_bytes{node=%s}`): {ts("0", 7e10, 7.6e10)},
q(`llmgw_gpu_temperature_celsius{node=%s}`): {ts("0", 65, 71)},
q(`llmgw_gpu_power_watts{node=%s}`): {ts("0", 500, 612)},
}}
_, body := get(t, nodeMux(gpuSource{}, h), "http://x/admin/nodes/llmgw-node01", nil)
for _, want := range []string{
"97%", "memory 64%", "71.2 <span class=\"note\">of 80 GB", "1980 MHz",
"3 uncorrected ECC errors",
"GPU busy, by card", "GPU memory used, by card", "GPU temperature, °C", "GPU power, W", "GPU 0",
} {
if !strings.Contains(body, want) {
t.Errorf("the page lacks %q", want)
}
}
// A node without GPUs is not asked about them.
h2 := &fakeHistory{}
get(t, nodeMux(cpuFleet{}, h2), "http://x/admin/nodes/llmgw-node01", nil)
for _, e := range h2.asked {
if strings.Contains(e, "llmgw_gpu_") {
t.Errorf("a node with no GPU was asked for %s", e)
}
}
}
func TestTheClusterPageDrawsItsGPUsByClass(t *testing.T) {
h := &fakeHistory{byExpr: map[string][]promq.Series{
`avg by (class) (llmgw_gpu_utilization_percent) / 100`: {ts("h100-80g", 0.2, 0.6)},
`sum by (class) (llmgw_gpu_memory_used_bytes) / sum by (class) (llmgw_gpu_memory_total_bytes)`: {ts("h100-80g", 0.8, 0.9)},
}}
_, body := get(t, Dashboard(fakeSource{}, nil, "v1", time.Second, true, NavExtras{History: h, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
if !strings.Contains(body, "GPUs busy, by class") || !strings.Contains(body, "GPU memory in use, by class") {
t.Error("a fleet with GPUs has no GPU charts")
}
h2 := &fakeHistory{}
_, cpu := get(t, Dashboard(cpuFleet{}, nil, "v1", time.Second, true, NavExtras{History: h2, Window: time.Hour}, nil), "http://127.0.0.1:9090/", nil)
if strings.Contains(cpu, "GPUs busy") {
t.Error("a fleet without GPUs was drawn GPU charts")
}
for _, e := range h2.asked {
if strings.Contains(e, "llmgw_gpu_") {
t.Errorf("a fleet without GPUs was asked for %s", e)
}
}
}
+7
View File
@@ -862,3 +862,10 @@ table.rows td.warn { color: var(--warn); cursor: help; }
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
/* The GPU map: a square per card, a row per node. */
.gpu-map { display: flex; flex-direction: column; gap: 3px; margin: 6px 0; overflow-x: auto; }
.gpu-row { display: flex; align-items: center; gap: 3px; }
.gpu-node { width: 7em; flex: none; font-size: 12px; color: var(--muted, #666); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.gpu-cell { width: 26px; height: 26px; flex: none; border: 1px solid rgba(0,0,0,.18); border-radius: 4px;
font-size: 10px; line-height: 26px; text-align: center; color: rgba(0,0,0,.7); cursor: default; }
+15
View File
@@ -63,6 +63,21 @@
<p class="empty">Nothing recorded in this window.</p>
{{end}}
{{with .GPUHours}}
<h2>GPU time, by owner</h2>
{{/* What each owner's models held, whether or not anybody asked them
anything: a model kept loaded holds its cards all the same, and that is
the time a GPU quota is about. */}}
<div class="chart-row">
{{template "tschart" dict "t" $.GPUHeld "title" "GPUs held, by owner" "note" "The cards held by the replicas of each owner's models: models whose owner is a part of the quota tree."}}
</div>
<table class="rows">
<tr><th>Owner</th><th class="num">GPU-hours</th><th class="num">Held now</th></tr>
{{range .}}<tr><td>{{.Owner}}</td><td class="num">{{printf "%.1f" .Hours}}</td><td class="num">{{printf "%.0f" .Now}}</td></tr>{{end}}
</table>
<p class="note">From the metrics store, which keeps its history for as long as it is set to (fifteen days by default); a model with no owner is counted under “no owner”.</p>
{{end}}
{{/* The charts fold their tail; the table does not. Both are here because they
answer different questions, and the one that is easy to get wrong is
thinking the four coloured lines are all of it. */}}
+38
View File
@@ -49,6 +49,36 @@
</div>
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
{{with .GPUMap}}
<h2>GPUs <span class="opt">by
{{range .Modes}}{{if eq .Key $.GPUMap.By}}<strong>{{.Label}}</strong>{{else}}<a href="?gpumap={{.Key}}">{{.Label}}</a>{{end}} {{end}}</span></h2>
<div class="gpu-map">
{{range .Rows}}<div class="gpu-row"><span class="gpu-node">{{.Node}}</span>{{range .Cells}}<span class="gpu-cell" style="{{.Style}}" title="{{.Title}}">{{.Label}}</span>{{end}}</div>
{{end}}</div>
<p class="note">{{.Legend}} A red outline is a card wearing out — XID errors, memory rows remapped or waiting for a reset, NVLink errors. The figure is the one coloured by; the rest of a card is under the pointer.</p>
{{end}}
{{with .Efficiency}}
<h3>What each model gets from its cards <span class="opt">last 15 minutes</span></h3>
{{/* Per card, because a model on eight cards making twice the tokens of one
on two is the worse of the two; per watt-hour, because that is what the
room is paying. */}}
<table class="rows"><tr><th>Model</th><th class="num">GPUs</th><th class="num">Output tokens/s</th><th class="num">Per GPU</th><th class="num">Busy</th><th class="num">Draw</th><th class="num">Tokens per Wh</th></tr>
{{range .}}<tr><td>{{.Model}}</td><td class="num">{{printf "%.0f" .GPUs}}</td><td class="num">{{printf "%.1f" .TokensPerSec}}</td>
<td class="num">{{printf "%.1f" .PerGPU}}</td><td class="num">{{printf "%.0f" .Util}}%</td>
<td class="num">{{if .Watts}}{{printf "%.0f" .Watts}} W{{else}}—{{end}}</td><td class="num">{{if .PerWh}}{{printf "%.0f" .PerWh}}{{else}}—{{end}}</td></tr>
{{end}}</table>
{{end}}
{{with .IdleGPUs}}
<h3>Cards held and doing nothing</h3>
{{/* The first place to find room: replicas whose every card reads under 5%
busy now. A model kept warm on purpose is here too; whether it should
be is the operator's call. */}}
<table class="rows"><tr><th>Model</th><th>Replica</th><th>Node</th><th class="num">GPUs</th><th class="num">Busiest card</th><th>Last request</th></tr>
{{range .}}<tr><td>{{.Model}}</td><td class="clip"><code>{{.Replica}}</code></td><td><code>{{.Node}}</code></td>
<td class="num">{{len .GPUs}}</td><td class="num">{{printf "%.0f" .MaxUtil}}%</td><td>{{if .Idle}}{{.Idle}} ago{{else}}<span class="note">not known</span>{{end}}</td></tr>
{{end}}</table>
{{end}}
<h2>Nodes</h2>
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
{{if .Nodes}}
@@ -213,6 +243,14 @@ unused nodes, which is what decides whether such a replica can land.</p>
{{template "tschart" dict "t" .Ready "title" "Replicas ready, by model"
"note" "A model deliberately stopped has no line at all, which is why an on-demand catalogue looks quiet here."}}
</div>
{{if $.AnyGPUs}}
<div class="chart-row">
{{template "tschart" dict "t" .GPUBusy "title" "GPUs busy, by class"
"note" "The average card of each class. Low here with replicas placed is capacity held and not asked for; each node's page has its cards one by one."}}
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory in use, by class"
"note" "Engines reserve most of a card's memory when they start, so this follows replicas placed more than requests served."}}
</div>
{{end}}
<div class="chart-row">
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica"
"note" "Requests accepted and not yet started. The first thing to move when a replica is falling behind."}}
+17 -2
View File
@@ -35,13 +35,18 @@
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
{{with .Load}}{{if .GPUs}}
<h2>GPUs</h2>
<table class="rows"><tr><th class="num">GPU</th><th class="num">Temperature</th><th class="num">Memory</th><th>Power</th><th>Held back by</th></tr>
<table class="rows"><tr><th class="num">GPU</th><th>Busy</th><th>Memory used</th><th class="num">Temperature</th><th class="num">Memory temp.</th><th>Power</th><th class="num">Clock</th><th>Held back by, and wear</th></tr>
{{range .GPUs}}<tr>
<td class="num">{{.Index}}</td>
{{/* From the last sample; a dash where the driver or an older agent did
not say, rather than a zero nobody measured. */}}
<td class="load">{{if and .MemoryTotalMB (ge .UtilPercent 0.0)}}{{template "minibar" pctbar .UtilPercent}}<span class="load-v">{{printf "%.0f" .UtilPercent}}%{{if ge .MemUtilPercent 0.0}} <span class="note">memory {{printf "%.0f" .MemUtilPercent}}%</span>{{end}}</span>{{else}}—{{end}}</td>
<td class="load">{{if .MemoryTotalMB}}{{template "minibar" pctbar (percent .MemoryUsedMB .MemoryTotalMB)}}<span class="load-v">{{printf "%.1f" (gib .MemoryUsedMB)}} <span class="note">of {{printf "%.0f" (gib .MemoryTotalMB)}} GB</span></span>{{else}}—{{end}}</td>
<td class="num">{{if .TempC}}{{printf "%.0f" .TempC}} °C{{else}}—{{end}}</td>
<td class="num">{{if .MemoryTempC}}{{printf "%.0f" .MemoryTempC}} °C{{else}}—{{end}}</td>
<td class="load">{{if and .PowerW .PowerLimitW}}{{template "minibar" pctbar (percent .PowerW .PowerLimitW)}}<span class="load-v">{{printf "%.0f" .PowerW}} W <span class="note">of {{printf "%.0f" .PowerLimitW}}</span></span>{{else if .PowerW}}{{printf "%.0f" .PowerW}} W{{else}}—{{end}}</td>
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}</td>
<td class="num">{{if .SMClockMHz}}{{printf "%.0f" .SMClockMHz}} MHz{{else}}—{{end}}</td>
<td>{{if .Throttle}}<span class="bad">{{join .Throttle}}</span>{{else}}<span class="note">nothing</span>{{end}}{{if gt .ECCUncorrected 0.0}} <span class="bad">{{printf "%.0f" .ECCUncorrected}} uncorrected ECC errors</span>{{end}}{{with .Wear}}{{with .Problems}}<br><span class="bad">{{join .}}</span>{{end}}{{if gt .PCIeReplays 0}} <span class="note">{{.PCIeReplays}} PCIe retries</span>{{end}}{{end}}</td>
</tr>{{end}}</table>
{{end}}{{end}}
{{end}}
@@ -120,6 +125,16 @@ traffic between nodes should go: PIX and PXB are under the same PCIe switch, SYS
{{template "tschart" dict "t" .EngineCPU "title" "Cores busy, by model" "note" "Each engine's process group, the way top counts it."}}
{{template "tschart" dict "t" .EngineMemory "title" "Engine memory, by model" "note" "Resident memory, including the weights mapped from disk. The kernel can take those pages back, so this can exceed RAM in use."}}
</div>
{{if $.Node.GPUTotal}}
<div class="chart-row">
{{template "tschart" dict "t" .GPUUtil "title" "GPU busy, by card" "note" "Of each sample period, the share a kernel was running. A card held by a replica and idle here is one nobody is asking."}}
{{template "tschart" dict "t" .GPUMemory "title" "GPU memory used, by card" "note" "Framebuffer in use: the weights and the KV cache an engine reserved, and anything not llmgw's."}}
</div>
<div class="chart-row">
{{template "tschart" dict "t" .GPUTemp "title" "GPU temperature, °C" "note" "Throttling usually starts in the high 80s; see what holds a card back in the table above."}}
{{template "tschart" dict "t" .GPUPower "title" "GPU power, W" "note" "Draw against the card's enforced limit; at the limit is sw_power_cap."}}
</div>
{{end}}
<div class="chart-row">
{{template "tschart" dict "t" .Load "title" "Load average" "note" "Above the core count means work is waiting for a CPU."}}
{{template "tschart" dict "t" .Queue "title" "Engine queue depth, by replica" "note" "Requests accepted and not yet started, on the replicas here now."}}
+2
View File
@@ -110,6 +110,8 @@ var funcs = template.FuncMap{
return p
},
"ago": ago,
// gib is mebibytes as gibibytes, for a GPU's memory.
"gib": func(mib float64) float64 { return mib / 1024 },
// percent is part of whole, 0 to 100, for a bar.
"percent": func(part, whole float64) float64 {
if whole <= 0 {