Export what the cluster is made of, not only what it is serving
CI / test (push) Successful in 1m28s

The router published twenty-two metric families about requests, endpoints,
admission and scoring — everything about traffic and nothing about the
machines carrying it. There was no way to graph GPU usage, watch a node go
unhealthy, see a fabric fill up, or answer "was the cluster full at the
time", which is the first question anyone asks after an incident. The
agents exported nothing at all.

Exported by the control plane rather than by each agent. It already holds
every number: hardware arrives at registration and state on each heartbeat,
so this is a map read away. One scrape target for a fleet of hundreds, no
Prometheus configuration per node, no firewall rule per machine, and
nothing new running on the nodes. On the same /metrics as the traffic
figures, because correlating "the cluster was full" with "requests were
queueing" should not need two systems.

A collector rather than gauges kept current by a ticker: the numbers are
derived from state that already exists, so computing them when asked is
simpler than maintaining a copy that can drift, and nothing runs between
scrapes.

GPU utilisation is new, and is the number people actually watch. Memory
used says a model is loaded; this says whether it is doing anything, and a
fleet at ninety percent occupancy and two percent utilisation is a fleet
holding models nobody is asking for.

Adding it caught two things. utilization.gpu goes last in the nvidia-smi
query and is optional, because a driver too old to know it returns a
shorter row — requiring it made an llmgw upgrade silently stop discovering
GPUs on an older node, which is far worse than not knowing how busy a card
is. And I gave the new protobuf field number 14, which ecc_errors already
had; buf lint is why that was a moment rather than a corrupted wire format.

Absent is kept distinct from zero throughout, for the same reason as
everywhere else here: an unreported utilisation shown as a flat zero reads
as an idle fleet rather than a missing measurement, and a replica nothing
has reported would be the most idle thing in the cluster.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 15:09:27 +02:00
1 parent 943c23a412
commit 42fd0aa31e
12 files changed
+488 -33

No files matched your search

+19 -7
View File
@@ -640,11 +640,15 @@ type GPU struct {
NvlinkPeers []int32 `protobuf:"varint,11,rep,packed,name=nvlink_peers,json=nvlinkPeers,proto3" json:"nvlink_peers,omitempty"`
// problems keeps a GPU out of placement: ECC faults, or memory already held
// by another process.
Problems []string `protobuf:"bytes,12,rep,name=problems,proto3" json:"problems,omitempty"`
TemperatureC int32 `protobuf:"varint,13,opt,name=temperature_c,json=temperatureC,proto3" json:"temperature_c,omitempty"`
EccErrors int32 `protobuf:"varint,14,opt,name=ecc_errors,json=eccErrors,proto3" json:"ecc_errors,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
Problems []string `protobuf:"bytes,12,rep,name=problems,proto3" json:"problems,omitempty"`
TemperatureC int32 `protobuf:"varint,13,opt,name=temperature_c,json=temperatureC,proto3" json:"temperature_c,omitempty"`
EccErrors int32 `protobuf:"varint,14,opt,name=ecc_errors,json=eccErrors,proto3" json:"ecc_errors,omitempty"`
// utilization_percent is how busy the card was over the last sample period.
// Zero when the driver is too old to report it, which is not the same as
// idle: the metric is simply absent rather than wrong.
UtilizationPercent int32 `protobuf:"varint,15,opt,name=utilization_percent,json=utilizationPercent,proto3" json:"utilization_percent,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *GPU) Reset() {
@@ -775,6 +779,13 @@ func (x *GPU) GetEccErrors() int32 {
return 0
}
func (x *GPU) GetUtilizationPercent() int32 {
if x != nil {
return x.UtilizationPercent
}
return 0
}
type FabricAttachment struct {
state protoimpl.MessageState `protogen:"open.v1"`
// fabric is the operator-assigned name. It is not derived from the GID
@@ -1546,7 +1557,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\vleader_addr\x18\r \x01(\tR\n" +
"leaderAddr\x12\x1f\n" +
"\vleader_port\x18\x0e \x01(\x05R\n" +
"leaderPort\"\xc8\x03\n" +
"leaderPort\"\xf9\x03\n" +
"\x03GPU\x12\x14\n" +
"\x05index\x18\x01 \x01(\x05R\x05index\x12\x12\n" +
"\x04uuid\x18\x02 \x01(\tR\x04uuid\x12\x12\n" +
@@ -1564,7 +1575,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\bproblems\x18\f \x03(\tR\bproblems\x12#\n" +
"\rtemperature_c\x18\r \x01(\x05R\ftemperatureC\x12\x1d\n" +
"\n" +
"ecc_errors\x18\x0e \x01(\x05R\teccErrors\"[\n" +
"ecc_errors\x18\x0e \x01(\x05R\teccErrors\x12/\n" +
"\x13utilization_percent\x18\x0f \x01(\x05R\x12utilizationPercent\"[\n" +
"\x10FabricAttachment\x12\x16\n" +
"\x06fabric\x18\x01 \x01(\tR\x06fabric\x12\x12\n" +
"\x04hcas\x18\x02 \x03(\tR\x04hcas\x12\x1b\n" +
+4
View File
@@ -137,6 +137,10 @@ message GPU {
repeated string problems = 12;
int32 temperature_c = 13;
int32 ecc_errors = 14;
// utilization_percent is how busy the card was over the last sample period.
// Zero when the driver is too old to report it, which is not the same as
// idle: the metric is simply absent rather than wrong.
int32 utilization_percent = 15;
}
message FabricAttachment {