Publish per-tenant request and budget series, and alert on them

Requests by outcome, tokens and first-token time per tenant, at the level
router.tenant_label names (team by default), and every budget's share spent
this period. Rules alert on budgets at 80% and 100%, tenants hitting limits,
failing, spiking past their weekly average, or waiting on first tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 19:55:24 +02:00
1 parent 9f479b9a70
commit ce006a6e6f
10 files changed
+376 -2

No files matched your search

+80
View File
@@ -350,3 +350,83 @@ groups:
Working as designed when the cluster is full: the lowest band is
held first. Worth knowing about because the callers being refused
do not know why.
# Per tenant: the level router.tenant_label names (a team by default), and
# every organization, team, project and person given a budget.
- name: llmgw-tenants
rules:
- alert: BudgetMostlySpent
expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
description: >-
Its requests are refused once it reaches all of it, until the
period turns. Raise it on the people page, or tell them now
rather than when they are refused.
- alert: BudgetSpent
expr: llmgw_budget_spent_ratio >= 1
for: 1m
labels: { severity: critical }
annotations:
summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
description: >-
Every request under it is refused with 429 until the period
turns, unless a cheaper model is named as its fallback.
- alert: TenantHittingLimits
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
description: >-
A rate, token or budget limit is turning their traffic away. Either
the limit is too low for what they now do, or something of theirs
is looping.
- alert: TenantErrors
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
description: >-
Not counting refusals by a limit. If every tenant fails alike it
is a model (ModelFailing); one alone is usually what they send —
prompts past the context, a model they may not use.
- alert: TenantTrafficSpike
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
> 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
for: 15m
labels: { severity: info }
annotations:
summary: "{{ $labels.tenant }} is sending three times its weekly average"
description: >-
A new workload, a batch job, or a client retrying in a loop. Only
for a tenant sending more than one request every five seconds, so
a quiet team starting work in the morning does not page anybody.
- alert: TenantSlowFirstToken
expr: >-
histogram_quantile(0.95,
sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
description: >-
What their people feel. Compare with SlowFirstToken per model: if
only this tenant is slow, it is the models it uses or its queue
band, not the cluster.