# Alerts for llmgw. Every expression here is against a series this cluster # actually publishes; nothing is written against a metric that might exist. groups: - name: llmgw-fleet rules: - alert: ControlPlaneUnreachable expr: up{job="llmgw"} == 0 for: 1m labels: { severity: critical } annotations: summary: "The control plane is not answering" description: >- Nothing has scraped 10.0.0.140:9091 for a minute. Engines already running keep serving — the router is in the same process, so if this is down, so is the gateway. - alert: NodeNotReady expr: llmgw_node_up == 0 for: 2m labels: { severity: warning } annotations: summary: "{{ $labels.node }} is {{ $labels.state }}" description: >- The node is known but not taking work. Two minutes, because a node restarting its agent is briefly not ready and is not a problem. - alert: AgentGoneQuiet expr: llmgw_node_last_seen_seconds > 60 for: 1m labels: { severity: warning } annotations: summary: "{{ $labels.node }} has not reported for {{ $value | humanizeDuration }}" description: >- Heartbeats are every 5s. A minute of silence is the network or the agent, and the control plane stages expiry at 15s unreachable and 60s lost — so this fires around the point it stops placing there. - alert: ModelHasNothingServing expr: llmgw_model_replicas_ready == 0 for: 10m labels: { severity: critical } annotations: summary: "{{ $labels.model }} has no replica ready" description: >- The model should be running and nothing is serving it. Ten minutes, because a large model on CPU legitimately takes minutes to load. The series is published at 0 for every model that should be running — before v0.10.4 it was published only while a replica was ready, so this could not fire at all. A model deliberately stopped, or an on-demand one nobody is asking for, publishes nothing. - alert: ModelParked expr: llmgw_model_parked == 1 labels: { severity: critical } annotations: summary: "{{ $labels.model }} failed every time it started and is no longer retried" description: >- Five replicas in a row failed at start. Before this existed a model whose weights its engine could not load was started 155 times over eight hours. `llmgw ctl status` has the last error; once it is dealt with, `llmgw ctl retry `. Changing the model's definition also retries it. - alert: ModelFailingToStart expr: llmgw_model_start_failures >= 2 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row" description: >- On its way to being parked. The engine's own words are in `llmgw ctl logs --source ` for the next attempt, or in the control plane's log for the last one. - alert: ModelCannotBePlaced expr: llmgw_model_unplaceable == 1 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.model }} is wanted and fits on no node" description: >- Short of replicas, and the last attempt found nowhere with the memory, cards or engine it needs. `llmgw ctl explain-placement ` says what each node lacks. For an on-demand model, requests for it are told this rather than asked to retry. - alert: ModelPreempted expr: increase(llmgw_model_preempted_total[15m]) > 0 labels: { severity: info } annotations: summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class" description: >- Capacity ran short and a model of a higher class took this one's room. Expected now and then; if it keeps firing for the same model, the cluster is too small for what is asked of it at these classes. `llmgw ctl models` shows who it gave way to. - alert: CapacityAtLimit expr: llmgw_capacity_holds >= on(scope, resource) llmgw_capacity_limit for: 15m labels: { severity: info } annotations: summary: "{{ $labels.scope }} holds all the {{ $labels.resource }} it may" description: >- Its models get no further replicas, under load or not, until something of theirs stops or the limit is raised: `llmgw ctl quota list`, `llmgw ctl quota set`. - alert: EngineInstallFailed expr: llmgw_node_engine_install{state="failed"} == 1 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}" description: >- The node takes no model needing it until it does. The agent retries on its own; `llmgw ctl nodes` and the node's logs on the cluster page say why it failed. - alert: ControlPlaneLeadership expr: sum(llmgw_control_plane_leader) != 1 for: 2m labels: { severity: critical } annotations: summary: "{{ $value }} control planes believe they are placing work" description: >- None means nothing is placed or replaced; two means both are, each into GPUs the other believes free. The lease is in the store: check it is reachable from every control plane. - alert: CertificateExpiresSoon expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400 labels: { severity: warning } annotations: summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}" description: >- Certificates last a year and nothing renews them. When the agent server's runs out every agent refuses to connect at once. Reissue with `llmgw ctl cert` and restart the control plane. - alert: CertificateExpiresVerySoon expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400 labels: { severity: critical } annotations: summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}" description: See CertificateExpiresSoon; a week is not long. - alert: BackupTooOld expr: time() - llmgw_backup_last_success_timestamp_seconds > 26 * 3600 labels: { severity: warning } annotations: summary: "The newest llmgw backup is {{ $value | humanizeDuration }} old" description: >- llmgw-backup.timer runs nightly. Also fires when there has never been a backup (the series is 0 then). `journalctl -u llmgw-backup.service` on the control plane says why the last one failed; docs/install.md, "Backing up", says how to set it up. - alert: ModelUpdateAvailable expr: llmgw_model_update_available == 1 for: 1h labels: { severity: info } annotations: summary: "{{ $labels.repo }} has new weights for {{ $labels.model }}" description: >- The registry repository the model was served from has published different files under the same names. `llmgw ctl model updates` lists what changed; `llmgw ctl model update {{ $labels.model }}` moves it to them, and `llmgw ctl model rollback` goes back. - alert: PoolSaturated expr: llmgw_pool_saturation > 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.model }} is over its healthy limit" description: >- Saturation is the worse of queue depth and KV occupancy; 1 is the limit measured as healthy, not the point of failure. Sustained above it means requests are queueing — add a replica or lower the load. - alert: EndpointBreakerOpen expr: llmgw_endpoint_breaker_open == 1 for: 2m labels: { severity: warning } annotations: summary: "{{ $labels.replica }} has been taken out of rotation" description: >- The router stopped sending to this replica after repeated failures. Traffic is going elsewhere; if it is the only replica, the model is down. - alert: EndpointMetricsStale expr: llmgw_endpoint_metrics_stale == 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.replica }} is being routed to blind" description: >- Its queue depth and KV usage are old, so the scheduler is choosing with stale numbers. The replica may be serving perfectly well; the decisions about it are the problem. - alert: NodeDiskFilling expr: llmgw_node_disk_free_bytes / llmgw_node_disk_total_bytes < 0.10 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.node }} has {{ $value | humanizePercentage }} of its model disk free" description: >- Weights are tens of gigabytes. A node that cannot hold the next one is skipped by placement rather than failing, so this is quiet until somebody wonders why a model will not land there. - name: llmgw-gpus rules: - alert: GPUThermalThrottling expr: llmgw_gpu_throttled{reason=~"sw_thermal|hw_thermal|hw_slowdown"} == 1 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} is slowed by heat ({{ $labels.reason }})" description: >- Ten minutes held back by temperature: every replica on the card is slower than its measurements say. Usually airflow, a fan, or the room — the node page shows the card's temperature over time. - alert: GPUUncorrectedECCErrors expr: llmgw_gpu_ecc_errors > 0 for: 1m labels: { severity: critical } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has {{ $value }} uncorrected ECC errors" description: >- Uncorrected memory errors since the driver loaded: what runs on the card may be computing on corrupted memory. Drain the node (llmgw ctl drain) and have the card looked at. - alert: GPUXidError expr: increase(llmgw_gpu_xid_errors[15m]) > 0 for: 0m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} logged an XID error" description: >- The driver reported an error on the card (its code is llmgw_gpu_last_xid, and the node page says it in words). 79 is a card fallen off the bus, 48 and 63/64 memory errors, 74 NVLink, 94/95 contained and uncontained ECC errors: those need the card looked at. Others are often an engine's own fault, and pass. - alert: GPUOutOfSpareRows expr: llmgw_gpu_remap_failure == 1 for: 1m labels: { severity: critical } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has no spare memory rows left" description: >- The card cannot remap another failing row. Drain the node and have the card replaced. - alert: GPUNeedsReset expr: llmgw_gpu_remap_pending == 1 or llmgw_gpu_retirement_pending == 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} has memory repairs waiting for a reset" description: >- Rows remapped or pages retired take effect only after the card is reset. Drain the node (llmgw ctl drain), reset or reboot, and bring it back. - alert: GPUPCIeRetriesClimbing expr: increase(llmgw_gpu_pcie_replays[1h]) > 100 for: 0m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: {{ $value }} PCIe retries in an hour" description: >- The card's link to the host is retrying. A riser, the slot or a cable is the usual cause, and transfers to the card slow down. - alert: GPUNVLinkErrorsClimbing expr: increase(llmgw_gpu_nvlink_errors{kind="crc"}[1h]) > 10 for: 0m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }}: NVLink CRC errors climbing" description: >- Errors on the card's NVLink links: tensor-parallel replicas on it slow down, and a link that keeps failing takes a replica with it. - alert: GPUHot expr: llmgw_gpu_temperature_celsius > 87 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.node }} GPU {{ $labels.gpu }} at {{ $value }} °C" description: >- Ten minutes above 87 °C, where most data-centre cards begin to throttle. - name: monitoring-host rules: - alert: MonitoringDiskFilling expr: >- node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.15 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.mountpoint }} on the monitoring host is {{ $value | humanizePercentage }} free" description: >- This filesystem holds the model mirror and the metrics database at once. Filling it stops new weights arriving and stops Prometheus writing — so monitoring goes quiet at the moment it is most needed, which is why this fires at fifteen percent rather than five. - alert: MonitoringDiskAlmostFull expr: >- node_filesystem_avail_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="monitoring-host",fstype!~"tmpfs|overlay|squashfs"} < 0.05 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.mountpoint }} on the monitoring host has {{ $value | humanizePercentage }} left" - name: llmgw-traffic rules: # These need traffic through the gateway to have any series at all. A # counter with no observations publishes nothing, so a quiet cluster # simply has no data here rather than a rule that is always false. - alert: HighErrorRate expr: >- sum by (model) (rate(llmgw_requests_total{status=~"5.."}[5m])) / sum by (model) (rate(llmgw_requests_total[5m])) > 0.05 for: 5m labels: { severity: critical } annotations: summary: "{{ $labels.model }} is failing {{ $value | humanizePercentage }} of requests" - alert: SlowFirstToken expr: >- histogram_quantile(0.99, sum by (model, le) (rate(llmgw_ttft_seconds_bucket[10m]))) > 30 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.model }} takes {{ $value | humanizeDuration }} to first token at p99" description: >- On CPU this is expected under load and is why the threshold is thirty seconds rather than one. Compare with what ctl bench measured before treating it as a fault. - alert: RequestsBeingRejected expr: sum by (band) (rate(llmgw_admission_rejected_total[5m])) > 0 for: 5m labels: { severity: warning } annotations: summary: "Admission is turning away {{ $labels.band }} traffic" description: >- Working as designed when the cluster is full: the lowest band is held first. Worth knowing about because the callers being refused do not know why. # Per tenant: the level router.tenant_label names (a team by default), and # every organization, team, project and person given a budget. - name: llmgw-tenants rules: - alert: BudgetMostlySpent expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget" description: >- Its requests are refused once it reaches all of it, until the period turns. Raise it on the people page, or tell them now rather than when they are refused. - alert: BudgetSpent expr: llmgw_budget_spent_ratio >= 1 for: 1m labels: { severity: critical } annotations: summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget" description: >- Every request under it is refused with 429 until the period turns, unless a cheaper model is named as its fallback. - alert: TenantHittingLimits expr: >- sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m])) / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1 for: 10m labels: { severity: warning } annotations: summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit" description: >- A rate, token or budget limit is turning their traffic away. Either the limit is too low for what they now do, or something of theirs is looping. - alert: TenantErrors expr: >- sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m])) / sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05 and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05 for: 10m labels: { severity: warning } annotations: summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail" description: >- Not counting refusals by a limit. If every tenant fails alike it is a model (ModelFailing); one alone is usually what they send — prompts past the context, a model they may not use. - alert: TenantTrafficSpike expr: >- sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d])) and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2 for: 15m labels: { severity: info } annotations: summary: "{{ $labels.tenant }} is sending three times its weekly average" description: >- A new workload, a batch job, or a client retrying in a loop. Only for a tenant sending more than one request every five seconds, so a quiet team starting work in the morning does not page anybody. - alert: TenantSlowFirstToken expr: >- histogram_quantile(0.95, sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10 and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95" description: >- What their people feel. Compare with SlowFirstToken per model: if only this tenant is slow, it is the models it uses or its queue band, not the cluster. # The store, when it is a replicated cluster under Patroni (llmgw server # init --patroni). Published only for a store of several hosts. - name: llmgw-store rules: - alert: StoreHasNoLeader expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0 for: 1m labels: { severity: critical } annotations: summary: "The store's cluster has no primary" description: >- Every write fails — keys, usage, budgets, leadership. Patroni promotes a standby within its 30-second TTL; a minute without one means it cannot: etcd has lost its majority, or no standby is fit. `patronictl -c /etc/patroni/config.yml list` on a member says which. - alert: StorePatroniUnreachable expr: llmgw_store_patroni_up == 0 for: 2m labels: { severity: warning } annotations: summary: "No member of the store's cluster answers on its Patroni port" description: >- The control plane reads the cluster's state from port 8008 on each member. The database may still be serving; nothing here can tell. - alert: StoreReplicaNotStreaming expr: llmgw_store_member{role!="leader", state!="streaming"} == 1 for: 5m labels: { severity: warning } annotations: summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary" description: >- A standby that is not streaming is not a standby: if the primary is lost now, there is nothing to promote. "missing" is a host of the store address the cluster no longer lists — usually a machine that is down. - alert: StoreReplicaBehind expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1 for: 10m labels: { severity: warning } annotations: summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary" description: >- Patroni will not promote a standby more than 1 MB behind, so a failover now would find none fit. -1 is a lag Patroni cannot read. - name: llmgw-guard rules: - alert: GuardFailingOpen expr: >- sum(rate(llmgw_guard_fail_open_total[5m])) / clamp_min(sum(rate(llmgw_guard_checks_total[5m])), 1e-9) > 0.01 for: 5m labels: { severity: warning } annotations: summary: "{{ $value | humanizePercentage }} of judged traffic is served unjudged" description: >- A guard policy's judge model is not answering within its latency budget, and the policy fails open: requests go through flagged "judge unavailable" instead of being judged. llmgw_guard_judge_errors_total says whether it is timing out or not there; `llmgw ctl replicas` whether the judge has replicas. - alert: GuardJudgeSlow expr: >- histogram_quantile(0.95, sum by (le, judge_model) (rate(llmgw_guard_judge_latency_seconds_bucket[5m]))) > 0.25 for: 10m labels: { severity: warning } annotations: summary: "Judge {{ $labels.judge_model }} answers in {{ $value | humanizeDuration }} at p95" description: >- Over the default latency budget: every judged request waits this long, and those over their policy's budget are served unjudged or refused. Give the judge more replicas, or faster ones. - alert: GuardJudgeUnavailable expr: sum by (reason) (rate(llmgw_guard_judge_errors_total{reason!="timeout"}[5m])) > 0 for: 5m labels: { severity: critical } annotations: summary: "The guardrails cannot ask their judge ({{ $labels.reason }})" description: >- no_judge: a policy names a model that is not in the catalog with role: guard. unavailable: it has no replica answering. format: it answers in a form its guard.format does not read. - alert: GuardBlocksSpiking expr: >- sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[1h])) > 3 * sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[7d])) and sum by (tenant) (increase(llmgw_tenant_guard_blocks_total[1h])) > 10 for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.tenant }} is being blocked three times as often as usual" description: >- A tenant's requests or answers are blocked by the guardrails more than three times their weekly rate. Someone probing, a client sending data it should not, or a policy that changed. The /admin/guard page lists the recent ones by request. - alert: GuardSecretInAnswer expr: sum(increase(llmgw_guard_checks_total{stage="output",category="secret",action!="allowed"}[5m])) > 0 labels: { severity: critical } annotations: summary: "A model put a credential in an answer" description: >- An answer contained something shaped like a key or a private key. It was stopped or rewritten if the policy enforces; either way a model has seen it, in its prompt or its training. The usage rows with guard_category secret say which request. - name: llmgw-slo rules: - alert: ModelTTFTOverSLO expr: >- histogram_quantile(0.99, sum by (le, model) (rate(llmgw_ttft_seconds_bucket[10m]))) > on (model) group_left llmgw_model_slo_seconds{kind="ttft_p99"} for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.model }} answers its first token past its SLO at p99" description: >- The p99 time to the first token over ten minutes is past the model's slo.ttft_p99. Admission already holds the lower bands back while it is; this says it has lasted. More replicas, or less traffic in the bands above. - alert: ModelTPOTOverSLO expr: >- histogram_quantile(0.99, sum by (le, model) (rate(llmgw_tpot_seconds_bucket[10m]))) > on (model) group_left llmgw_model_slo_seconds{kind="tpot_p99"} for: 15m labels: { severity: warning } annotations: summary: "{{ $labels.model }} produces tokens slower than its SLO at p99" description: >- The p99 time per output token is past slo.tpot_p99: replicas are batching more than they can decode quickly, or a slower card is serving. llmgw_output_tokens_per_second by pool says which.