Ilker ManapandClaude Opus 5.5
82a63cfc6e
Every minute the agent reads nvidia-smi's full report, its NVLink error counters and the kernel's XID lines. They are published per card, shown on the node page and the GPU map, charted in Grafana, and five alerts watch them. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
353 lines
16 KiB
YAML
353 lines
16 KiB
YAML
# Alerts for llmgw. Every expression here is against a series this cluster
|
|
# actually publishes; nothing is written against a metric that might exist.
|
|
groups:
|
|
- name: llmgw-fleet
|
|
rules:
|
|
- alert: ControlPlaneUnreachable
|
|
expr: up{job="llmgw"} == 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "The control plane is not answering"
|
|
description: >-
|
|
Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
|
|
running keep serving — the router is in the same process, so if
|
|
this is down, so is the gateway.
|
|
|
|
- alert: NodeNotReady
|
|
expr: llmgw_node_up == 0
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} is {{ $labels.state }}"
|
|
description: >-
|
|
The node is known but not taking work. Two minutes, because a node
|
|
restarting its agent is briefly not ready and is not a problem.
|
|
|
|
- alert: AgentGoneQuiet
|
|
expr: llmgw_node_last_seen_seconds > 60
|
|
for: 1m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Heartbeats are every 5s. A minute of silence is the network or the
|
|
agent, and the control plane stages expiry at 15s unreachable and
|
|
60s lost — so this fires around the point it stops placing there.
|
|
|
|
- alert: ModelHasNothingServing
|
|
expr: llmgw_model_replicas_ready == 0
|
|
for: 10m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has no replica ready"
|
|
description: >-
|
|
The model should be running and nothing is serving it. Ten minutes,
|
|
because a large model on CPU legitimately takes minutes to load. The
|
|
series is published at 0 for every model that should be running —
|
|
before v0.10.4 it was published only while a replica was ready, so
|
|
this could not fire at all. A model deliberately stopped, or an
|
|
on-demand one nobody is asking for, publishes nothing.
|
|
|
|
- alert: ModelParked
|
|
expr: llmgw_model_parked == 1
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
|
|
description: >-
|
|
Five replicas in a row failed at start. Before this existed a model
|
|
whose weights its engine could not load was started 155 times over
|
|
eight hours. `llmgw ctl status` has the last error; once it is dealt
|
|
with, `llmgw ctl retry <model>`. Changing the model's definition
|
|
also retries it.
|
|
|
|
- alert: ModelFailingToStart
|
|
expr: llmgw_model_start_failures >= 2
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
|
|
description: >-
|
|
On its way to being parked. The engine's own words are in
|
|
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
|
|
the control plane's log for the last one.
|
|
|
|
- alert: ModelCannotBePlaced
|
|
expr: llmgw_model_unplaceable == 1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is wanted and fits on no node"
|
|
description: >-
|
|
Short of replicas, and the last attempt found nowhere with the
|
|
memory, cards or engine it needs. `llmgw ctl explain-placement
|
|
<model>` says what each node lacks. For an on-demand model, requests
|
|
for it are told this rather than asked to retry.
|
|
|
|
- alert: ModelPreempted
|
|
expr: increase(llmgw_model_preempted_total[15m]) > 0
|
|
labels: { severity: info }
|
|
annotations:
|
|
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
|
|
description: >-
|
|
Capacity ran short and a model of a higher class took this one's
|
|
room. Expected now and then; if it keeps firing for the same model,
|
|
the cluster is too small for what is asked of it at these classes.
|
|
`llmgw ctl models` shows who it gave way to.
|
|
|
|
- alert: CapacityAtLimit
|
|
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
|
|
for: 15m
|
|
labels: { severity: info }
|
|
annotations:
|
|
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
|
|
description: >-
|
|
Its models get no further replicas, under load or not, until
|
|
something of theirs stops or the limit is raised: `llmgw ctl quota
|
|
list`, `llmgw ctl quota set`.
|
|
|
|
- alert: EngineInstallFailed
|
|
expr: llmgw_node_engine_install{state="failed"} == 1
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
|
|
description: >-
|
|
The node takes no model needing it until it does. The agent retries
|
|
on its own; `llmgw ctl nodes` and the node's logs on the cluster
|
|
page say why it failed.
|
|
|
|
- alert: ControlPlaneLeadership
|
|
expr: sum(llmgw_control_plane_leader) != 1
|
|
for: 2m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $value }} control planes believe they are placing work"
|
|
description: >-
|
|
None means nothing is placed or replaced; two means both are, each
|
|
into GPUs the other believes free. The lease is in the store: check
|
|
it is reachable from every control plane.
|
|
|
|
- alert: CertificateExpiresSoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Certificates last a year and nothing renews them. When the agent
|
|
server's runs out every agent refuses to connect at once. Reissue
|
|
with `llmgw ctl cert` and restart the control plane.
|
|
|
|
- alert: CertificateExpiresVerySoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: See CertificateExpiresSoon; a week is not long.
|
|
|
|
- alert: BackupTooOld
|
|
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
|
|
description: >-
|
|
llmgw-backup.timer runs nightly. Also fires when there has never
|
|
been a backup (the series is 0 then). `journalctl -u
|
|
llmgw-backup.service` on the control plane says why the last one
|
|
failed; docs/install.md, "Backing up", says how to set it up.
|
|
|
|
- alert: PoolSaturated
|
|
expr: llmgw_pool_saturation > 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is over its healthy limit"
|
|
description: >-
|
|
Saturation is the worse of queue depth and KV occupancy; 1 is the
|
|
limit measured as healthy, not the point of failure. Sustained above
|
|
it means requests are queueing — add a replica or lower the load.
|
|
|
|
- alert: EndpointBreakerOpen
|
|
expr: llmgw_endpoint_breaker_open == 1
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} has been taken out of rotation"
|
|
description: >-
|
|
The router stopped sending to this replica after repeated failures.
|
|
Traffic is going elsewhere; if it is the only replica, the model is
|
|
down.
|
|
|
|
- alert: EndpointMetricsStale
|
|
expr: llmgw_endpoint_metrics_stale == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} is being routed to blind"
|
|
description: >-
|
|
Its queue depth and KV usage are old, so the scheduler is choosing
|
|
with stale numbers. The replica may be serving perfectly well; the
|
|
decisions about it are the problem.
|
|
|
|
- alert: NodeDiskFilling
|
|
expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
|
|
description: >-
|
|
Weights are tens of gigabytes. A node that cannot hold the next one
|
|
is skipped by placement rather than failing, so this is quiet until
|
|
somebody wonders why a model will not land there.
|
|
|
|
- name: llmgw-gpus
|
|
rules:
|
|
- alert: GPUThermalThrottling
|
|
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
|
|
description: >-
|
|
Ten minutes held back by temperature: every replica on the card is
|
|
slower than its measurements say. Usually airflow, a fan, or the
|
|
room — the node page shows the card's temperature over time.
|
|
|
|
- alert: GPUUncorrectedECCErrors
|
|
expr: llmgw_gpu_ecc_errors > 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
|
|
description: >-
|
|
Uncorrected memory errors since the driver loaded: what runs on the
|
|
card may be computing on corrupted memory. Drain the node
|
|
(llmgw ctl drain) and have the card looked at.
|
|
|
|
- alert: GPUXidError
|
|
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
|
description: >-
|
|
The driver reported an error on the card (its code is
|
|
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
|
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
|
94/95 contained and uncontained ECC errors: those need the card
|
|
looked at. Others are often an engine's own fault, and pass.
|
|
|
|
- alert: GPUOutOfSpareRows
|
|
expr: llmgw_gpu_remap_failure == 1
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
|
description: >-
|
|
The card cannot remap another failing row. Drain the node and
|
|
have the card replaced.
|
|
|
|
- alert: GPUNeedsReset
|
|
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
|
description: >-
|
|
Rows remapped or pages retired take effect only after the card is
|
|
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
|
it back.
|
|
|
|
- alert: GPUPCIeRetriesClimbing
|
|
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
|
description: >-
|
|
The card's link to the host is retrying. A riser, the slot or a
|
|
cable is the usual cause, and transfers to the card slow down.
|
|
|
|
- alert: GPUNVLinkErrorsClimbing
|
|
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
|
description: >-
|
|
Errors on the card's NVLink links: tensor-parallel replicas on it
|
|
slow down, and a link that keeps failing takes a replica with it.
|
|
|
|
- alert: GPUHot
|
|
expr: llmgw_gpu_temperature_celsius > 87
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
|
|
description: >-
|
|
Ten minutes above 87 °C, where most data-centre cards begin to
|
|
throttle.
|
|
|
|
- name: monitoring-host
|
|
rules:
|
|
- alert: MonitoringDiskFilling
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
|
|
description: >-
|
|
This filesystem holds the model mirror and the metrics database at
|
|
once. Filling it stops new weights arriving and stops Prometheus
|
|
writing — so monitoring goes quiet at the moment it is most needed,
|
|
which is why this fires at fifteen percent rather than five.
|
|
|
|
- alert: MonitoringDiskAlmostFull
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
|
|
|
|
- name: llmgw-traffic
|
|
rules:
|
|
# These need traffic through the gateway to have any series at all. A
|
|
# counter with no observations publishes nothing, so a quiet cluster
|
|
# simply has no data here rather than a rule that is always false.
|
|
- alert: HighErrorRate
|
|
expr: >-
|
|
sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
|
|
/ sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
|
|
|
|
- alert: SlowFirstToken
|
|
expr: >-
|
|
histogram_quantile(0.99,
|
|
sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
|
|
description: >-
|
|
On CPU this is expected under load and is why the threshold is
|
|
thirty seconds rather than one. Compare with what ctl bench measured
|
|
before treating it as a fault.
|
|
|
|
- alert: RequestsBeingRejected
|
|
expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "Admission is turning away {{ $labels.band }} traffic"
|
|
description: >-
|
|
Working as designed when the cluster is full: the lowest band is
|
|
held first. Worth knowing about because the callers being refused
|
|
do not know why.
|