Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
43f6a585a6
commit
47e847423d
13 files changed
+300
-17
No files matched your search
@@ -489,3 +489,69 @@ groups:
|
||||
description: >-
|
||||
Patroni will not promote a standby more than 1 MB behind, so a
|
||||
failover now would find none fit. -1 is a lag Patroni cannot read.
|
||||
|
||||
- name: llmgw-guard
|
||||
rules:
|
||||
- alert: GuardFailingOpen
|
||||
expr: >-
|
||||
sum(rate(llmgw_guard_fail_open_total[5m]))
|
||||
/ clamp_min(sum(rate(llmgw_guard_checks_total[5m])), 1e-9) > 0.01
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $value | humanizePercentage }} of judged traffic is served unjudged"
|
||||
description: >-
|
||||
A guard policy's judge model is not answering within its latency
|
||||
budget, and the policy fails open: requests go through flagged
|
||||
"judge unavailable" instead of being judged.
|
||||
llmgw_guard_judge_errors_total says whether it is timing out or
|
||||
not there; `llmgw ctl replicas` whether the judge has replicas.
|
||||
|
||||
- alert: GuardJudgeSlow
|
||||
expr: >-
|
||||
histogram_quantile(0.95, sum by (le, judge_model) (rate(llmgw_guard_judge_latency_seconds_bucket[5m]))) > 0.25
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "Judge {{ $labels.judge_model }} answers in {{ $value | humanizeDuration }} at p95"
|
||||
description: >-
|
||||
Over the default latency budget: every judged request waits this
|
||||
long, and those over their policy's budget are served unjudged or
|
||||
refused. Give the judge more replicas, or faster ones.
|
||||
|
||||
- alert: GuardJudgeUnavailable
|
||||
expr: sum by (reason) (rate(llmgw_guard_judge_errors_total{reason!="timeout"}[5m])) > 0
|
||||
for: 5m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "The guardrails cannot ask their judge ({{ $labels.reason }})"
|
||||
description: >-
|
||||
no_judge: a policy names a model that is not in the catalog with
|
||||
role: guard. unavailable: it has no replica answering. format: it
|
||||
answers in a form its guard.format does not read.
|
||||
|
||||
- alert: GuardBlocksSpiking
|
||||
expr: >-
|
||||
sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[1h]))
|
||||
> 3 * sum by (tenant) (rate(llmgw_tenant_guard_blocks_total[7d]))
|
||||
and sum by (tenant) (increase(llmgw_tenant_guard_blocks_total[1h])) > 10
|
||||
for: 15m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.tenant }} is being blocked three times as often as usual"
|
||||
description: >-
|
||||
A tenant's requests or answers are blocked by the guardrails
|
||||
more than three times their weekly rate. Someone probing, a
|
||||
client sending data it should not, or a policy that changed.
|
||||
The /admin/guard page lists the recent ones by request.
|
||||
|
||||
- alert: GuardSecretInAnswer
|
||||
expr: sum(increase(llmgw_guard_checks_total{stage="output",category="secret",action!="allowed"}[5m])) > 0
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "A model put a credential in an answer"
|
||||
description: >-
|
||||
An answer contained something shaped like a key or a private key.
|
||||
It was stopped or rewritten if the policy enforces; either way a
|
||||
model has seen it, in its prompt or its training. The usage rows
|
||||
with guard_category secret say which request.
|
||||
Reference in new issue
Block a user