# Alerts for llmgw. Every expression here is against a series this cluster # actually publishes; nothing is written against a metric that might exist. groups: - name: llmgw-fleet rules: - alert: ControlPlaneUnreachable expr: up{job="llmgw"} == 0 for: 1m labels: { severity: critical } annotations: summary: "The control plane is not answering" description: >- Nothing has scraped 10.0.0.140:9091 for a minute. Engines already running keep serving — the router is in the same process, so if this is down, so is the gateway. - alert: NodeNotReady expr: llmgw_node_up == 0 for: 2m labels: { severity: warning } annotations: summary: "{{ $labels.node }} is {{ $labels.state }}" description: >- The node is known but not taking work. Two minutes, because a node restarting its agent is briefly not ready and is not a problem. - alert: AgentGoneQuiet expr: llmgw_node_last_seen_seconds > 60 for: 1m labels: { severity: warning } annotations: summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}" description: >- Heartbeats are every 5s. A minute of silence is the network or the agent, and the control plane stages expiry at 15s unreachable and 60s lost — so this fires around the point it stops placing there. - alert: ModelHasNothingServing expr: llmgw_model_replicas_ready == 0 for: 10m labels: { severity: critical } annotations: summary: "{{ $labels.model }} has no replica ready" description: >- The model should be running and nothing is serving it. Ten minutes, because a large model on CPU legitimately takes minutes to load. The series is published at 0 for every model that should be running — before v0.10.4 it was published only while a replica was ready, so this could not fire at all. A model deliberately stopped, or an on-demand one nobody is asking for, publishes nothing. - alert: ModelParked expr: llmgw_model_parked == 1 labels: { severity: critical } annotations: summary: "{{ $labels.model }} failed every time it started and is no longer retried" description: >- Five replicas in a row failed at start. Before this existed a model whose weights its engine could not load was started 155 times over eight hours. `llmgw ctl status` has the last error; once it is dealt with, `llmgw ctl retry `. Changing the model's definition also retries it. - alert: ModelFailingToStart expr: llmgw_model_start_failures >= 2 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row" description: >- On its way to being parked. The engine's own words are in `llmgw ctl logs --source ` for the next attempt, or in the control plane's log for the last one. - alert: ModelCannotBePlaced expr: llmgw_model_unplaceable == 1 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.model }} is wanted and fits on no node" description: >- Short of replicas, and the last attempt found nowhere with the memory, cards or engine it needs. `llmgw ctl explain-placement ` says what each node lacks. For an on-demand model, requests for it are told this rather than asked to retry. - alert: EngineInstallFailed expr: llmgw_node_engine_install{state="failed"} == 1 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}" description: >- The node takes no model needing it until it does. The agent retries on its own; `llmgw ctl nodes` and the node's logs on the cluster page say why it failed. - alert: ControlPlaneLeadership expr: sum(llmgw_control_plane_leader) != 1 for: 2m labels: { severity: critical } annotations: summary: "{{ $value }} control planes believe they are placing work" description: >- None means nothing is placed or replaced; two means both are, each into GPUs the other believes free. The lease is in the store: check it is reachable from every control plane. - alert: CertificateExpiresSoon expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400 labels: { severity: warning } annotations: summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}" description: >- Certificates last a year and nothing renews them. When the agent server's runs out every agent refuses to connect at once. Reissue with `llmgw ctl cert` and restart the control plane. - alert: CertificateExpiresVerySoon expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400 labels: { severity: critical } annotations: summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}" description: See CertificateExpiresSoon; a week is not long. - alert: BackupTooOld expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600 labels: { severity: warning } annotations: summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old" description: >- llmgw-backup.timer runs nightly. Also fires when there has never been a backup (the series is 0 then). `journalctl -u llmgw-backup.service` on the control plane says why the last one failed; docs/install.md, "Backing up", says how to set it up. - alert: PoolSaturated expr: llmgw_pool_saturation > 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.model }} is over its healthy limit" description: >- Saturation is the worse of queue depth and KV occupancy; 1 is the limit measured as healthy, not the point of failure. Sustained above it means requests are queueing — add a replica or lower the load. - alert: EndpointBreakerOpen expr: llmgw_endpoint_breaker_open == 1 for: 2m labels: { severity: warning } annotations: summary: "{{ $labels.replica }} has been taken out of rotation" description: >- The router stopped sending to this replica after repeated failures. Traffic is going elsewhere; if it is the only replica, the model is down. - alert: EndpointMetricsStale expr: llmgw_endpoint_metrics_stale == 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.replica }} is being routed to blind" description: >- Its queue depth and KV usage are old, so the scheduler is choosing with stale numbers. The replica may be serving perfectly well; the decisions about it are the problem. - alert: NodeDiskFilling expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free" description: >- Weights are tens of gigabytes. A node that cannot hold the next one is skipped by placement rather than failing, so this is quiet until somebody wonders why a model will not land there. - name: monitoring-host rules: - alert: MonitoringDiskFilling expr: >- node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free" description: >- This filesystem holds the model mirror and the metrics database at once. Filling it stops new weights arriving and stops Prometheus writing — so monitoring goes quiet at the moment it is most needed, which is why this fires at fifteen percent rather than five. - alert: MonitoringDiskAlmostFull expr: >- node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left" - name: llmgw-traffic rules: # These need traffic through the gateway to have any series at all. A # counter with no observations publishes nothing, so a quiet cluster # simply has no data here rather than a rule that is always false. - alert: HighErrorRate expr: >- sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m])) / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests" - alert: SlowFirstToken expr: >- histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99" description: >- On CPU this is expected under load and is why the threshold is thirty seconds rather than one. Compare with what ctl bench measured before treating it as a fault. - alert: RequestsBeingRejected expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0 for: 5m labels: { severity: warning } annotations: summary: "Admission is turning away {{ $labels.band }} traffic" description: >- Working as designed when the cluster is full: the lowest band is held first. Worth knowing about because the callers being refused do not know why.