Ilker ManapandClaude Opus 5.5
36cdbce125
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
480 lines
22 KiB
YAML
480 lines
22 KiB
YAML
# Alerts for llmgw. Every expression here is against a series this cluster
|
|
# actually publishes; nothing is written against a metric that might exist.
|
|
groups:
|
|
- name: llmgw-fleet
|
|
rules:
|
|
- alert: ControlPlaneUnreachable
|
|
expr: up{job="llmgw"} == 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "The control plane is not answering"
|
|
description: >-
|
|
Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
|
|
running keep serving — the router is in the same process, so if
|
|
this is down, so is the gateway.
|
|
|
|
- alert: NodeNotReady
|
|
expr: llmgw_node_up == 0
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} is {{ $labels.state }}"
|
|
description: >-
|
|
The node is known but not taking work. Two minutes, because a node
|
|
restarting its agent is briefly not ready and is not a problem.
|
|
|
|
- alert: AgentGoneQuiet
|
|
expr: llmgw_node_last_seen_seconds > 60
|
|
for: 1m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Heartbeats are every 5s. A minute of silence is the network or the
|
|
agent, and the control plane stages expiry at 15s unreachable and
|
|
60s lost — so this fires around the point it stops placing there.
|
|
|
|
- alert: ModelHasNothingServing
|
|
expr: llmgw_model_replicas_ready == 0
|
|
for: 10m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has no replica ready"
|
|
description: >-
|
|
The model should be running and nothing is serving it. Ten minutes,
|
|
because a large model on CPU legitimately takes minutes to load. The
|
|
series is published at 0 for every model that should be running —
|
|
before v0.10.4 it was published only while a replica was ready, so
|
|
this could not fire at all. A model deliberately stopped, or an
|
|
on-demand one nobody is asking for, publishes nothing.
|
|
|
|
- alert: ModelParked
|
|
expr: llmgw_model_parked == 1
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
|
|
description: >-
|
|
Five replicas in a row failed at start. Before this existed a model
|
|
whose weights its engine could not load was started 155 times over
|
|
eight hours. `llmgw ctl status` has the last error; once it is dealt
|
|
with, `llmgw ctl retry <model>`. Changing the model's definition
|
|
also retries it.
|
|
|
|
- alert: ModelFailingToStart
|
|
expr: llmgw_model_start_failures >= 2
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
|
|
description: >-
|
|
On its way to being parked. The engine's own words are in
|
|
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
|
|
the control plane's log for the last one.
|
|
|
|
- alert: ModelCannotBePlaced
|
|
expr: llmgw_model_unplaceable == 1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is wanted and fits on no node"
|
|
description: >-
|
|
Short of replicas, and the last attempt found nowhere with the
|
|
memory, cards or engine it needs. `llmgw ctl explain-placement
|
|
<model>` says what each node lacks. For an on-demand model, requests
|
|
for it are told this rather than asked to retry.
|
|
|
|
- alert: ModelPreempted
|
|
expr: increase(llmgw_model_preempted_total[15m]) > 0
|
|
labels: { severity: info }
|
|
annotations:
|
|
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
|
|
description: >-
|
|
Capacity ran short and a model of a higher class took this one's
|
|
room. Expected now and then; if it keeps firing for the same model,
|
|
the cluster is too small for what is asked of it at these classes.
|
|
`llmgw ctl models` shows who it gave way to.
|
|
|
|
- alert: CapacityAtLimit
|
|
expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit
|
|
for: 15m
|
|
labels: { severity: info }
|
|
annotations:
|
|
summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may"
|
|
description: >-
|
|
Its models get no further replicas, under load or not, until
|
|
something of theirs stops or the limit is raised: `llmgw ctl quota
|
|
list`, `llmgw ctl quota set`.
|
|
|
|
- alert: EngineInstallFailed
|
|
expr: llmgw_node_engine_install{state="failed"} == 1
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
|
|
description: >-
|
|
The node takes no model needing it until it does. The agent retries
|
|
on its own; `llmgw ctl nodes` and the node's logs on the cluster
|
|
page say why it failed.
|
|
|
|
- alert: ControlPlaneLeadership
|
|
expr: sum(llmgw_control_plane_leader) != 1
|
|
for: 2m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $value }} control planes believe they are placing work"
|
|
description: >-
|
|
None means nothing is placed or replaced; two means both are, each
|
|
into GPUs the other believes free. The lease is in the store: check
|
|
it is reachable from every control plane.
|
|
|
|
- alert: CertificateExpiresSoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Certificates last a year and nothing renews them. When the agent
|
|
server's runs out every agent refuses to connect at once. Reissue
|
|
with `llmgw ctl cert` and restart the control plane.
|
|
|
|
- alert: CertificateExpiresVerySoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: See CertificateExpiresSoon; a week is not long.
|
|
|
|
- alert: BackupTooOld
|
|
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
|
|
description: >-
|
|
llmgw-backup.timer runs nightly. Also fires when there has never
|
|
been a backup (the series is 0 then). `journalctl -u
|
|
llmgw-backup.service` on the control plane says why the last one
|
|
failed; docs/install.md, "Backing up", says how to set it up.
|
|
|
|
- alert: PoolSaturated
|
|
expr: llmgw_pool_saturation > 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is over its healthy limit"
|
|
description: >-
|
|
Saturation is the worse of queue depth and KV occupancy; 1 is the
|
|
limit measured as healthy, not the point of failure. Sustained above
|
|
it means requests are queueing — add a replica or lower the load.
|
|
|
|
- alert: EndpointBreakerOpen
|
|
expr: llmgw_endpoint_breaker_open == 1
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} has been taken out of rotation"
|
|
description: >-
|
|
The router stopped sending to this replica after repeated failures.
|
|
Traffic is going elsewhere; if it is the only replica, the model is
|
|
down.
|
|
|
|
- alert: EndpointMetricsStale
|
|
expr: llmgw_endpoint_metrics_stale == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} is being routed to blind"
|
|
description: >-
|
|
Its queue depth and KV usage are old, so the scheduler is choosing
|
|
with stale numbers. The replica may be serving perfectly well; the
|
|
decisions about it are the problem.
|
|
|
|
- alert: NodeDiskFilling
|
|
expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
|
|
description: >-
|
|
Weights are tens of gigabytes. A node that cannot hold the next one
|
|
is skipped by placement rather than failing, so this is quiet until
|
|
somebody wonders why a model will not land there.
|
|
|
|
- name: llmgw-gpus
|
|
rules:
|
|
- alert: GPUThermalThrottling
|
|
expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})"
|
|
description: >-
|
|
Ten minutes held back by temperature: every replica on the card is
|
|
slower than its measurements say. Usually airflow, a fan, or the
|
|
room — the node page shows the card's temperature over time.
|
|
|
|
- alert: GPUUncorrectedECCErrors
|
|
expr: llmgw_gpu_ecc_errors > 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors"
|
|
description: >-
|
|
Uncorrected memory errors since the driver loaded: what runs on the
|
|
card may be computing on corrupted memory. Drain the node
|
|
(llmgw ctl drain) and have the card looked at.
|
|
|
|
- alert: GPUXidError
|
|
expr: increase(llmgw_gpu_xid_errors[15m]) > 0
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error"
|
|
description: >-
|
|
The driver reported an error on the card (its code is
|
|
llmgw_gpu_last_xid, and the node page says it in words). 79 is a
|
|
card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink,
|
|
94/95 contained and uncontained ECC errors: those need the card
|
|
looked at. Others are often an engine's own fault, and pass.
|
|
|
|
- alert: GPUOutOfSpareRows
|
|
expr: llmgw_gpu_remap_failure == 1
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left"
|
|
description: >-
|
|
The card cannot remap another failing row. Drain the node and
|
|
have the card replaced.
|
|
|
|
- alert: GPUNeedsReset
|
|
expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset"
|
|
description: >-
|
|
Rows remapped or pages retired take effect only after the card is
|
|
reset. Drain the node (llmgw ctl drain), reset or reboot, and bring
|
|
it back.
|
|
|
|
- alert: GPUPCIeRetriesClimbing
|
|
expr: increase(llmgw_gpu_pcie_replays[1h]) > 100
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour"
|
|
description: >-
|
|
The card's link to the host is retrying. A riser, the slot or a
|
|
cable is the usual cause, and transfers to the card slow down.
|
|
|
|
- alert: GPUNVLinkErrorsClimbing
|
|
expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10
|
|
for: 0m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing"
|
|
description: >-
|
|
Errors on the card's NVLink links: tensor-parallel replicas on it
|
|
slow down, and a link that keeps failing takes a replica with it.
|
|
|
|
- alert: GPUHot
|
|
expr: llmgw_gpu_temperature_celsius > 87
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C"
|
|
description: >-
|
|
Ten minutes above 87 °C, where most data-centre cards begin to
|
|
throttle.
|
|
|
|
- name: monitoring-host
|
|
rules:
|
|
- alert: MonitoringDiskFilling
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
|
|
description: >-
|
|
This filesystem holds the model mirror and the metrics database at
|
|
once. Filling it stops new weights arriving and stops Prometheus
|
|
writing — so monitoring goes quiet at the moment it is most needed,
|
|
which is why this fires at fifteen percent rather than five.
|
|
|
|
- alert: MonitoringDiskAlmostFull
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
|
|
|
|
- name: llmgw-traffic
|
|
rules:
|
|
# These need traffic through the gateway to have any series at all. A
|
|
# counter with no observations publishes nothing, so a quiet cluster
|
|
# simply has no data here rather than a rule that is always false.
|
|
- alert: HighErrorRate
|
|
expr: >-
|
|
sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
|
|
/ sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
|
|
|
|
- alert: SlowFirstToken
|
|
expr: >-
|
|
histogram_quantile(0.99,
|
|
sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
|
|
description: >-
|
|
On CPU this is expected under load and is why the threshold is
|
|
thirty seconds rather than one. Compare with what ctl bench measured
|
|
before treating it as a fault.
|
|
|
|
- alert: RequestsBeingRejected
|
|
expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "Admission is turning away {{ $labels.band }} traffic"
|
|
description: >-
|
|
Working as designed when the cluster is full: the lowest band is
|
|
held first. Worth knowing about because the callers being refused
|
|
do not know why.
|
|
|
|
# Per tenant: the level router.tenant_label names (a team by default), and
|
|
# every organization, team, project and person given a budget.
|
|
- name: llmgw-tenants
|
|
rules:
|
|
- alert: BudgetMostlySpent
|
|
expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
|
|
description: >-
|
|
Its requests are refused once it reaches all of it, until the
|
|
period turns. Raise it on the people page, or tell them now
|
|
rather than when they are refused.
|
|
|
|
- alert: BudgetSpent
|
|
expr: llmgw_budget_spent_ratio >= 1
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
|
|
description: >-
|
|
Every request under it is refused with 429 until the period
|
|
turns, unless a cheaper model is named as its fallback.
|
|
|
|
- alert: TenantHittingLimits
|
|
expr: >-
|
|
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
|
|
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
|
|
description: >-
|
|
A rate, token or budget limit is turning their traffic away. Either
|
|
the limit is too low for what they now do, or something of theirs
|
|
is looping.
|
|
|
|
- alert: TenantErrors
|
|
expr: >-
|
|
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
|
|
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
|
|
and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
|
|
description: >-
|
|
Not counting refusals by a limit. If every tenant fails alike it
|
|
is a model (ModelFailing); one alone is usually what they send —
|
|
prompts past the context, a model they may not use.
|
|
|
|
- alert: TenantTrafficSpike
|
|
expr: >-
|
|
sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
|
|
> 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
|
|
and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
|
|
for: 15m
|
|
labels: { severity: info }
|
|
annotations:
|
|
summary: "{{ $labels.tenant }} is sending three times its weekly average"
|
|
description: >-
|
|
A new workload, a batch job, or a client retrying in a loop. Only
|
|
for a tenant sending more than one request every five seconds, so
|
|
a quiet team starting work in the morning does not page anybody.
|
|
|
|
- alert: TenantSlowFirstToken
|
|
expr: >-
|
|
histogram_quantile(0.95,
|
|
sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
|
|
and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
|
|
description: >-
|
|
What their people feel. Compare with SlowFirstToken per model: if
|
|
only this tenant is slow, it is the models it uses or its queue
|
|
band, not the cluster.
|
|
# The store, when it is a replicated cluster under Patroni (llmgw server
|
|
# init --patroni). Published only for a store of several hosts.
|
|
- name: llmgw-store
|
|
rules:
|
|
- alert: StoreHasNoLeader
|
|
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "The store's cluster has no primary"
|
|
description: >-
|
|
Every write fails — keys, usage, budgets, leadership. Patroni
|
|
promotes a standby within its 30-second TTL; a minute without one
|
|
means it cannot: etcd has lost its majority, or no standby is fit.
|
|
`patronictl -c /etc/patroni/config.yml list` on a member says which.
|
|
|
|
- alert: StorePatroniUnreachable
|
|
expr: llmgw_store_patroni_up == 0
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "No member of the store's cluster answers on its Patroni port"
|
|
description: >-
|
|
The control plane reads the cluster's state from port 8008 on each
|
|
member. The database may still be serving; nothing here can tell.
|
|
|
|
- alert: StoreReplicaNotStreaming
|
|
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
|
|
description: >-
|
|
A standby that is not streaming is not a standby: if the primary
|
|
is lost now, there is nothing to promote. "missing" is a host of
|
|
the store address the cluster no longer lists — usually a machine
|
|
that is down.
|
|
|
|
- alert: StoreReplicaBehind
|
|
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
|
|
description: >-
|
|
Patroni will not promote a standby more than 1 MB behind, so a
|
|
failover now would find none fit. -1 is a lag Patroni cannot read.
|