Publish per-tenant request and budget series, and alert on them
Requests by outcome, tokens and first-token time per tenant, at the level router.tenant_label names (team by default), and every budget's share spent this period. Rules alert on budgets at 80% and 100%, tenants hitting limits, failing, spiking past their weekly average, or waiting on first tokens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
9f479b9a70
commit
ce006a6e6f
10 files changed
+376
-2
No files matched your search
@@ -350,3 +350,83 @@ groups:
|
||||
Working as designed when the cluster is full: the lowest band is
|
||||
held first. Worth knowing about because the callers being refused
|
||||
do not know why.
|
||||
|
||||
# Per tenant: the level router.tenant_label names (a team by default), and
|
||||
# every organization, team, project and person given a budget.
|
||||
- name: llmgw-tenants
|
||||
rules:
|
||||
- alert: BudgetMostlySpent
|
||||
expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
|
||||
description: >-
|
||||
Its requests are refused once it reaches all of it, until the
|
||||
period turns. Raise it on the people page, or tell them now
|
||||
rather than when they are refused.
|
||||
|
||||
- alert: BudgetSpent
|
||||
expr: llmgw_budget_spent_ratio >= 1
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
|
||||
description: >-
|
||||
Every request under it is refused with 429 until the period
|
||||
turns, unless a cheaper model is named as its fallback.
|
||||
|
||||
- alert: TenantHittingLimits
|
||||
expr: >-
|
||||
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
|
||||
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
|
||||
description: >-
|
||||
A rate, token or budget limit is turning their traffic away. Either
|
||||
the limit is too low for what they now do, or something of theirs
|
||||
is looping.
|
||||
|
||||
- alert: TenantErrors
|
||||
expr: >-
|
||||
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
|
||||
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
|
||||
and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
|
||||
description: >-
|
||||
Not counting refusals by a limit. If every tenant fails alike it
|
||||
is a model (ModelFailing); one alone is usually what they send —
|
||||
prompts past the context, a model they may not use.
|
||||
|
||||
- alert: TenantTrafficSpike
|
||||
expr: >-
|
||||
sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
|
||||
> 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
|
||||
and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
|
||||
for: 15m
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.tenant }} is sending three times its weekly average"
|
||||
description: >-
|
||||
A new workload, a batch job, or a client retrying in a loop. Only
|
||||
for a tenant sending more than one request every five seconds, so
|
||||
a quiet team starting work in the morning does not page anybody.
|
||||
|
||||
- alert: TenantSlowFirstToken
|
||||
expr: >-
|
||||
histogram_quantile(0.95,
|
||||
sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
|
||||
and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
|
||||
for: 15m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
|
||||
description: >-
|
||||
What their people feel. Compare with SlowFirstToken per model: if
|
||||
only this tenant is slow, it is the models it uses or its queue
|
||||
band, not the cluster.
|
||||
Reference in new issue
Block a user