Ilker ManapandClaude Opus 5.5
1da4a2e1ef
There were no backups: docs/install.md gave a pg_dump command, and nothing ran it. The store holds every key, person, quota and usage record, the models added at runtime, and now the secret each engine's key is derived from; the certificate authority's key sits in /etc/llmgw/pki beside it. Losing the disk lost all of it. `llmgw server backup` writes both to one archive: pg_dump's custom format, or a VACUUM INTO snapshot for SQLite, plus /etc/llmgw. The archive holds the CA's key, so it is 0600; the directory is listable, so the control plane — not root — can see when the newest was written. pg_dump is given the connection through its environment, since a DSN carries the password and a process's arguments are readable by every user. Written beside its name and renamed, so an interrupted backup is never the newest; the newest 14 are kept, and nothing the command did not write is ever removed. llmgw-backup.timer runs it nightly and catches up after downtime. The control plane publishes the newest archive's time, and BackupTooOld fires past 26 hours — or when there has never been one. The install guide now covers restoring, and says to copy archives off the machine, since they share its disk. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
243 lines
11 KiB
YAML
243 lines
11 KiB
YAML
# Alerts for llmgw. Every expression here is against a series this cluster
|
|
# actually publishes; nothing is written against a metric that might exist.
|
|
groups:
|
|
- name: llmgw-fleet
|
|
rules:
|
|
- alert: ControlPlaneUnreachable
|
|
expr: up{job="llmgw"} == 0
|
|
for: 1m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "The control plane is not answering"
|
|
description: >-
|
|
Nothing has scraped 10.0.0.140:9091 for a minute. Engines already
|
|
running keep serving — the router is in the same process, so if
|
|
this is down, so is the gateway.
|
|
|
|
- alert: NodeNotReady
|
|
expr: llmgw_node_up == 0
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} is {{ $labels.state }}"
|
|
description: >-
|
|
The node is known but not taking work. Two minutes, because a node
|
|
restarting its agent is briefly not ready and is not a problem.
|
|
|
|
- alert: AgentGoneQuiet
|
|
expr: llmgw_node_last_seen_seconds > 60
|
|
for: 1m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Heartbeats are every 5s. A minute of silence is the network or the
|
|
agent, and the control plane stages expiry at 15s unreachable and
|
|
60s lost — so this fires around the point it stops placing there.
|
|
|
|
- alert: ModelHasNothingServing
|
|
expr: llmgw_model_replicas_ready == 0
|
|
for: 10m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has no replica ready"
|
|
description: >-
|
|
The model should be running and nothing is serving it. Ten minutes,
|
|
because a large model on CPU legitimately takes minutes to load. The
|
|
series is published at 0 for every model that should be running —
|
|
before v0.10.4 it was published only while a replica was ready, so
|
|
this could not fire at all. A model deliberately stopped, or an
|
|
on-demand one nobody is asking for, publishes nothing.
|
|
|
|
- alert: ModelParked
|
|
expr: llmgw_model_parked == 1
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
|
|
description: >-
|
|
Five replicas in a row failed at start. Before this existed a model
|
|
whose weights its engine could not load was started 155 times over
|
|
eight hours. `llmgw ctl status` has the last error; once it is dealt
|
|
with, `llmgw ctl retry <model>`. Changing the model's definition
|
|
also retries it.
|
|
|
|
- alert: ModelFailingToStart
|
|
expr: llmgw_model_start_failures >= 2
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
|
|
description: >-
|
|
On its way to being parked. The engine's own words are in
|
|
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
|
|
the control plane's log for the last one.
|
|
|
|
- alert: ModelCannotBePlaced
|
|
expr: llmgw_model_unplaceable == 1
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is wanted and fits on no node"
|
|
description: >-
|
|
Short of replicas, and the last attempt found nowhere with the
|
|
memory, cards or engine it needs. `llmgw ctl explain-placement
|
|
<model>` says what each node lacks. For an on-demand model, requests
|
|
for it are told this rather than asked to retry.
|
|
|
|
- alert: EngineInstallFailed
|
|
expr: llmgw_node_engine_install{state="failed"} == 1
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
|
|
description: >-
|
|
The node takes no model needing it until it does. The agent retries
|
|
on its own; `llmgw ctl nodes` and the node's logs on the cluster
|
|
page say why it failed.
|
|
|
|
- alert: ControlPlaneLeadership
|
|
expr: sum(llmgw_control_plane_leader) != 1
|
|
for: 2m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $value }} control planes believe they are placing work"
|
|
description: >-
|
|
None means nothing is placed or replaced; two means both are, each
|
|
into GPUs the other believes free. The lease is in the store: check
|
|
it is reachable from every control plane.
|
|
|
|
- alert: CertificateExpiresSoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: >-
|
|
Certificates last a year and nothing renews them. When the agent
|
|
server's runs out every agent refuses to connect at once. Reissue
|
|
with `llmgw ctl cert` and restart the control plane.
|
|
|
|
- alert: CertificateExpiresVerySoon
|
|
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
|
description: See CertificateExpiresSoon; a week is not long.
|
|
|
|
- alert: BackupTooOld
|
|
expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old"
|
|
description: >-
|
|
llmgw-backup.timer runs nightly. Also fires when there has never
|
|
been a backup (the series is 0 then). `journalctl -u
|
|
llmgw-backup.service` on the control plane says why the last one
|
|
failed; docs/install.md, "Backing up", says how to set it up.
|
|
|
|
- alert: PoolSaturated
|
|
expr: llmgw_pool_saturation > 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is over its healthy limit"
|
|
description: >-
|
|
Saturation is the worse of queue depth and KV occupancy; 1 is the
|
|
limit measured as healthy, not the point of failure. Sustained above
|
|
it means requests are queueing — add a replica or lower the load.
|
|
|
|
- alert: EndpointBreakerOpen
|
|
expr: llmgw_endpoint_breaker_open == 1
|
|
for: 2m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} has been taken out of rotation"
|
|
description: >-
|
|
The router stopped sending to this replica after repeated failures.
|
|
Traffic is going elsewhere; if it is the only replica, the model is
|
|
down.
|
|
|
|
- alert: EndpointMetricsStale
|
|
expr: llmgw_endpoint_metrics_stale == 1
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.replica }} is being routed to blind"
|
|
description: >-
|
|
Its queue depth and KV usage are old, so the scheduler is choosing
|
|
with stale numbers. The replica may be serving perfectly well; the
|
|
decisions about it are the problem.
|
|
|
|
- alert: NodeDiskFilling
|
|
expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10
|
|
for: 15m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free"
|
|
description: >-
|
|
Weights are tens of gigabytes. A node that cannot hold the next one
|
|
is skipped by placement rather than failing, so this is quiet until
|
|
somebody wonders why a model will not land there.
|
|
|
|
- name: monitoring-host
|
|
rules:
|
|
- alert: MonitoringDiskFilling
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free"
|
|
description: >-
|
|
This filesystem holds the model mirror and the metrics database at
|
|
once. Filling it stops new weights arriving and stops Prometheus
|
|
writing — so monitoring goes quiet at the moment it is most needed,
|
|
which is why this fires at fifteen percent rather than five.
|
|
|
|
- alert: MonitoringDiskAlmostFull
|
|
expr: >-
|
|
node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"}
|
|
/ node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left"
|
|
|
|
- name: llmgw-traffic
|
|
rules:
|
|
# These need traffic through the gateway to have any series at all. A
|
|
# counter with no observations publishes nothing, so a quiet cluster
|
|
# simply has no data here rather than a rule that is always false.
|
|
- alert: HighErrorRate
|
|
expr: >-
|
|
sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m]))
|
|
/ sum by (model) (rate(llmgw_requests_total[5m])) > 0.05
|
|
for: 5m
|
|
labels: { severity: critical }
|
|
annotations:
|
|
summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests"
|
|
|
|
- alert: SlowFirstToken
|
|
expr: >-
|
|
histogram_quantile(0.99,
|
|
sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30
|
|
for: 10m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99"
|
|
description: >-
|
|
On CPU this is expected under load and is why the threshold is
|
|
thirty seconds rather than one. Compare with what ctl bench measured
|
|
before treating it as a fault.
|
|
|
|
- alert: RequestsBeingRejected
|
|
expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0
|
|
for: 5m
|
|
labels: { severity: warning }
|
|
annotations:
|
|
summary: "Admission is turning away {{ $labels.band }} traffic"
|
|
description: >-
|
|
Working as designed when the cluster is full: the lowest band is
|
|
held first. Worth knowing about because the callers being refused
|
|
do not know why.
|