Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 15:11:50 +02:00
1 parent 43f6a585a6
commit 47e847423d
13 files changed
+300 -17

No files matched your search

+66
View File
@@ -489,3 +489,69 @@ groups:
description: >-
Patroni will not promote a standby more than 1 MB behind, so a
failover now would find none fit. -1 is a lag Patroni cannot read.
- name: llmgw-guard
rules:
- alert: GuardFailingOpen
expr: >-
sum(rate(llmgw_guard_fail_open_total[5m]))
/ clamp_min(sum(rate(llmgw_guard_checks_total[5m])), 1e-9) > 0.01
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $value | humanizePercentage }} of judged traffic is served unjudged"
description: >-
A guard policy's judge model is not answering within its latency
budget, and the policy fails open: requests go through flagged
"judge unavailable" instead of being judged.
llmgw_guard_judge_errors_total says whether it is timing out or
not there; `llmgw ctl replicas` whether the judge has replicas.
- alert: GuardJudgeSlow
expr: >-
histogram_quantile(0.95, sum by (le, judge_model) (rate(llmgw_guard_judge_latency_seconds_bucket[5m]))) > 0.25
for: 10m
labels: { severity: warning }
annotations:
summary: "Judge {{ $labels.judge_model }} answers in {{ $value | humanizeDuration }} at p95"
description: >-
Over the default latency budget: every judged request waits this
long, and those over their policy's budget are served unjudged or
refused. Give the judge more replicas, or faster ones.
- alert: GuardJudgeUnavailable
expr: sum by (reason) (rate(llmgw_guard_judge_errors_total{reason!="timeout"}[5m])) > 0
for: 5m
labels: { severity: critical }
annotations:
summary: "The guardrails cannot ask their judge ({{ $labels.reason }})"
description: >-
no_judge: a policy names a model that is not in the catalog with
role: guard. unavailable: it has no replica answering. format: it
answers in a form its guard.format does not read.
- alert: GuardBlocksSpiking
expr: >-
sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[1h]))
> 3 * sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[7d]))
and sum by (tenant) (increase(llmgw_tenant_guard_blocks_total[1h])) > 10
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.tenant }} is being blocked three times as often as usual"
description: >-
A tenant's requests or answers are blocked by the guardrails
more than three times their weekly rate. Someone probing, a
client sending data it should not, or a policy that changed.
The /admin/guard page lists the recent ones by request.
- alert: GuardSecretInAnswer
expr: sum(increase(llmgw_guard_checks_total{stage="output",category="secret",action!="allowed"}[5m])) > 0
labels: { severity: critical }
annotations:
summary: "A model put a credential in an answer"
description: >-
An answer contained something shaped like a key or a private key.
It was stopped or rewritten if the policy enforces; either way a
model has seen it, in its prompt or its training. The usage rows
with guard_category secret say which request.