Merge usage-rollups: usage sums, per-tenant stats, alerts and dashboard

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 22:56:50 +02:00
commit 211688fe7c
34 files changed
+2443 -60

No files matched your search

+58
View File
@@ -75,6 +75,64 @@ and a figure the driver does not give is left out rather than shown as zero.
each card and whose quota it counts against, for joining the GPU series to
models and tenants.
### Usage is read from sums kept as it is written
Every chart and total — the usage tab, quotas, the dashboard, a person's own
page, and budgets restored at start — summed the request table again on each
view: a row per request, scanned whole. Usage is now also summed as it is
written, in the same transaction, into five-minute slices, hours and days per
key, person, project and model (`usage_5m`, `usage_1h`, `usage_1d`; migration
0015 fills them from what is already recorded).
- **Charts** read the coarsest of the three their slices are made of: a day in
20-minute slices from the five-minute table, a week or a month from hours.
- **Totals and budgets** read the rows up to the next five-minute slice, then
slices, hours and days — exact, not rounded to a slice. On three million
requests over thirty days, a month's total per person went from 705 ms to
18 ms on PostgreSQL.
- **Latency and time to first token** are kept as histograms in the sums, so a
percentile per team or model can be read after the rows are gone.
- **`auth.usage_retention`** (e.g. `2160h`) deletes request records older than
that, hourly. Totals, charts and budgets are unchanged by it — they read the
sums — but `ctl request` finds nothing older. Unset keeps every record, as
before.
### Alerts per tenant and per budget
- **`llmgw_tenant_requests_total{tenant,outcome}`**, `llmgw_tenant_tokens_total`
and `llmgw_tenant_ttft_seconds`: requests by outcome (ok, error, limited),
tokens and time to first token per tenant. The tenant is the level
`router.tenant_label` names — `team` by default, or `org`, `project`, or
`none` — named by its path, "acme/platform". One level rather than every
project, so a cluster of thousands of projects does not become a Prometheus
of millions of series.
- **`llmgw_budget_spent_ratio{scope,id,name,period}`**: this period's spend
over the budget for every organization, team, project and person given one,
refreshed every minute.
- **Alerts** in `deploy/monitoring/llmgw-rules.yml`: `BudgetMostlySpent` (80%),
`BudgetSpent`, `TenantHittingLimits` (a tenth of requests refused),
`TenantErrors`, `TenantTrafficSpike` (three times the weekly average) and
`TenantSlowFirstToken`.
- **A Grafana dashboard per tenant**, `deploy/monitoring/grafana/llmgw-tenants.json`
(uid `llmgw-tenants`): requests and tokens by tenant, the share failing and
refused by a limit, first-token p95, every budget's share spent, and who used
most in the last day — with a tenant chooser at the top.
### How each team's requests fared, not only what they cost
- **Percentiles from the sums.** First token and latency, p50, p95 and p99,
per organization, team, project, person or model, read from the histograms
kept with the usage sums: on the usage tab's table, and in a branch's or
person's panel on the people page, beside the error rate and requests
refused by a limit.
- **Refused requests are counted against their owner.** A request a rate,
token or budget limit refused was counted only in Prometheus, by model; a
team running into its limit looked like a team that had gone quiet. It is
now recorded as a 429 against its key, person and project — no tokens, no
cost — and shown as "Limited".
- **Usage by model** is a grouping of the usage tab, by the model that
answered.
### People, limits and usage are one page
Managing a team meant its members on the people page, its quota on the quotas
+5
View File
@@ -92,6 +92,11 @@ agent_listen: ":7443"
# flush_interval: 200ms
# batch_size: 500
# buffer_size: 8192
#
# # How long the record of each request is kept (what `ctl request` reads).
# # Charts, totals and budgets come from five-minute sums that are kept, so
# # this only bounds the table that grows by a row a request. 0 keeps all.
# usage_retention: 2160h # 90 days
router:
scrape_interval: 2s
+2
View File
@@ -66,6 +66,8 @@ install -m 0644 grafana/datasource.yaml /etc/grafana/provisioning/datasources/
install -m 0644 grafana/dashboards.yaml /etc/grafana/provisioning/dashboards/llmgw.yaml
install -m 0644 -o grafana -g grafana grafana/llmgw-overview.json \
/var/lib/grafana/dashboards/llmgw.json
install -m 0644 -o grafana -g grafana grafana/llmgw-tenants.json \
/var/lib/grafana/dashboards/llmgw-tenants.json
systemctl restart grafana-server
```
@@ -0,0 +1,650 @@
{
"uid": "llmgw-tenants",
"title": "llmgw \u2014 tenants",
"schemaVersion": 39,
"version": 1,
"editable": true,
"time": {
"from": "now-24h",
"to": "now"
},
"refresh": "1m",
"tags": [
"llmgw"
],
"description": "Requests, tokens, failures, refusals and first-token time per tenant \u2014 the level router.tenant_label names \u2014 and every budget's share spent.",
"templating": {
"list": [
{
"name": "tenant",
"label": "Tenant",
"type": "query",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"query": {
"query": "label_values(llmgw_tenant_requests_total, tenant)",
"refId": "tenant"
},
"definition": "label_values(llmgw_tenant_requests_total, tenant)",
"multi": true,
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"refresh": 2,
"sort": 1
}
]
},
"panels": [
{
"id": 1,
"title": "Requests a second",
"type": "stat",
"description": "Across the tenants chosen above.",
"gridPos": {
"x": 0,
"y": 0,
"w": 6,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 2,
"title": "Failing",
"type": "stat",
"description": "Requests that failed, not counting refusals by a limit.",
"gridPos": {
"x": 6,
"y": 0,
"w": 6,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.02
},
{
"color": "red",
"value": 0.05
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 3,
"title": "Refused by a limit",
"type": "stat",
"description": "Rate, token and budget limits; 429 to the caller.",
"gridPos": {
"x": 12,
"y": 0,
"w": 6,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum(rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.02
},
{
"color": "red",
"value": 0.1
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 4,
"title": "First token, p95",
"type": "stat",
"description": "What people wait for before anything appears.",
"gridPos": {
"x": 18,
"y": 0,
"w": 6,
"h": 4
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[5m])))",
"legendFormat": "",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 3
},
{
"color": "red",
"value": 10
}
]
}
},
"overrides": []
},
"options": {
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"textMode": "auto",
"colorMode": "value"
}
},
{
"id": 5,
"title": "Requests by tenant",
"type": "timeseries",
"description": "Stacked, so the top edge is the whole of it.",
"gridPos": {
"x": 0,
"y": 4,
"w": 12,
"h": 9
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "{{tenant}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"custom": {
"stacking": {
"mode": "normal"
},
"fillOpacity": 30
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 6,
"title": "Tokens by tenant",
"type": "timeseries",
"description": "Prompt and completion together: what was consumed.",
"gridPos": {
"x": 12,
"y": 4,
"w": 12,
"h": 9
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (tenant) (rate(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "{{tenant}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"stacking": {
"mode": "normal"
},
"fillOpacity": 30
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 7,
"title": "Refused by a limit, by tenant",
"type": "timeseries",
"description": "The share of each tenant's requests a limit turned away.",
"gridPos": {
"x": 0,
"y": 13,
"w": 12,
"h": 8
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "{{tenant}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 8,
"title": "Failing, by tenant",
"type": "timeseries",
"description": "One tenant alone failing is usually what it sends; all of them together is a model.",
"gridPos": {
"x": 12,
"y": 13,
"w": 12,
"h": 8
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"error\"}[5m])) / sum by (tenant) (rate(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[5m]))",
"legendFormat": "{{tenant}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 9,
"title": "First token p95, by tenant",
"type": "timeseries",
"description": "",
"gridPos": {
"x": 0,
"y": 21,
"w": 12,
"h": 8
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95, sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket{tenant=~\"$tenant\"}[10m])))",
"legendFormat": "{{tenant}}",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": [
"mean",
"max"
]
},
"tooltip": {
"mode": "multi",
"sort": "desc"
}
}
},
{
"id": 10,
"title": "Budgets spent this period",
"type": "bargauge",
"description": "Every organization, team, project and person with a budget. Requests are refused at 100%.",
"gridPos": {
"x": 12,
"y": 21,
"w": 12,
"h": 8
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sort_desc(llmgw_budget_spent_ratio)",
"legendFormat": "{{scope}} {{name}} ({{period}})",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
}
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "orange",
"value": 0.8
},
{
"color": "red",
"value": 1
}
]
}
},
"overrides": []
},
"options": {
"displayMode": "gradient",
"orientation": "horizontal",
"reduceOptions": {
"calcs": [
"lastNotNull"
]
},
"showUnfilled": true
}
},
{
"id": 11,
"title": "Who used most, last 24 hours",
"type": "table",
"description": "Spend in money is on the people page's usage tab, which reads it from the store.",
"gridPos": {
"x": 0,
"y": 29,
"w": 24,
"h": 8
},
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"targets": [
{
"refId": "A",
"expr": "sort_desc(sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\"}[24h])))",
"legendFormat": "requests",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"format": "table",
"instant": true
},
{
"refId": "B",
"expr": "sum by (tenant) (increase(llmgw_tenant_tokens_total{tenant=~\"$tenant\"}[24h]))",
"legendFormat": "tokens",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"format": "table",
"instant": true
},
{
"refId": "C",
"expr": "sum by (tenant) (increase(llmgw_tenant_requests_total{tenant=~\"$tenant\",outcome=\"limited\"}[24h]))",
"legendFormat": "limited",
"datasource": {
"type": "prometheus",
"uid": "llmgw-prom"
},
"format": "table",
"instant": true
}
],
"fieldConfig": {
"defaults": {},
"overrides": []
},
"options": {
"showHeader": true
},
"transformations": [
{
"id": "merge",
"options": {}
},
{
"id": "organize",
"options": {
"excludeByName": {
"Time": true
},
"renameByName": {
"Value #A": "Requests",
"Value #B": "Tokens",
"Value #C": "Refused by a limit"
}
}
}
]
}
]
}
+80
View File
@@ -350,3 +350,83 @@ groups:
Working as designed when the cluster is full: the lowest band is
held first. Worth knowing about because the callers being refused
do not know why.
# Per tenant: the level router.tenant_label names (a team by default), and
# every organization, team, project and person given a budget.
- name: llmgw-tenants
rules:
- alert: BudgetMostlySpent
expr: llmgw_budget_spent_ratio >= 0.8 and llmgw_budget_spent_ratio < 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.scope }} {{ $labels.name }} has spent {{ $value | humanizePercentage }} of its {{ $labels.period }}ly budget"
description: >-
Its requests are refused once it reaches all of it, until the
period turns. Raise it on the people page, or tell them now
rather than when they are refused.
- alert: BudgetSpent
expr: llmgw_budget_spent_ratio >= 1
for: 1m
labels: { severity: critical }
annotations:
summary: "{{ $labels.scope }} {{ $labels.name }} has spent its {{ $labels.period }}ly budget"
description: >-
Every request under it is refused with 429 until the period
turns, unless a cheaper model is named as its fallback.
- alert: TenantHittingLimits
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="limited"}[10m]))
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests are refused by a limit"
description: >-
A rate, token or budget limit is turning their traffic away. Either
the limit is too low for what they now do, or something of theirs
is looping.
- alert: TenantErrors
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total{outcome="error"}[10m]))
/ sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
and sum by (tenant) (rate(llmgw_tenant_requests_total[10m])) > 0.05
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $value | humanizePercentage }} of {{ $labels.tenant }}'s requests fail"
description: >-
Not counting refusals by a limit. If every tenant fails alike it
is a model (ModelFailing); one alone is usually what they send —
prompts past the context, a model they may not use.
- alert: TenantTrafficSpike
expr: >-
sum by (tenant) (rate(llmgw_tenant_requests_total[30m]))
> 3 * sum by (tenant) (rate(llmgw_tenant_requests_total[7d]))
and sum by (tenant) (rate(llmgw_tenant_requests_total[30m])) > 0.2
for: 15m
labels: { severity: info }
annotations:
summary: "{{ $labels.tenant }} is sending three times its weekly average"
description: >-
A new workload, a batch job, or a client retrying in a loop. Only
for a tenant sending more than one request every five seconds, so
a quiet team starting work in the morning does not page anybody.
- alert: TenantSlowFirstToken
expr: >-
histogram_quantile(0.95,
sum by (tenant, le) (rate(llmgw_tenant_ttft_seconds_bucket[15m]))) > 10
and sum by (tenant) (rate(llmgw_tenant_ttft_seconds_count[15m])) > 0.02
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.tenant }} waits {{ $value | humanizeDuration }} for a first token at p95"
description: >-
What their people feel. Compare with SlowFirstToken per model: if
only this tenant is slow, it is the models it uses or its queue
band, not the cluster.
+23
View File
@@ -39,6 +39,7 @@ rather than repeated.
| `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `name` | `string` | [Band](#band) |
| `priority` | `int` | [Band](#band) |
| `saturation_ceiling` | `float64` | [Band](#band) |
@@ -161,6 +162,7 @@ rather than repeated.
| `prefix` | `Prefix` | [Router](#router) |
| `session_header` | `string` | [Router](#router) |
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
@@ -367,6 +369,17 @@ BufferSize bounds the queue. Beyond it records are dropped rather than
blocking a request: losing accounting is bad, turning a stalled database
into an outage is worse.
### `usage_retention`
`time.Duration`
UsageRetention is how long the record of each request is kept: who sent
it, which replica answered, how long it queued — what `ctl request`
reads. Zero keeps it for ever. Totals, charts and budgets are read from
five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Band
### `name`
@@ -1242,6 +1255,16 @@ SessionHeader carries the session id for affinity (plan §7.5 tier 0).
InjectUsage asks engines for usage even when the client did not, so token
accounting works regardless of client behaviour.
### `tenant_label`
`string`
TenantLabel is the level of the tree the per-tenant request series are
labelled by — llmgw_tenant_requests_total and its siblings: "org",
"team" or "project", or "none" for no such series. A label per project
on a cluster of thousands is more series than Prometheus wants, and a
team is what an alert is usually about; "team" is the default.
### `max_body_bytes`
`int64`
+1
View File
@@ -54,6 +54,7 @@ require (
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/knadh/koanf/maps v0.1.2 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
+11
View File
@@ -316,6 +316,17 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
}
}
// RecordRefused queues the row of a request a limit refused: nothing is
// charged — it used nothing — but it is counted, so a team running into its
// limits shows as one rather than as a team that went quiet.
func (a *Accountant) RecordRefused(u store.Usage) {
if u.KeyID == "" || a.recorder == nil {
return
}
u.PromptTokens, u.CompletionTokens, u.CostMicros = 0, 0, 0
a.recorder.Record(u)
}
// Status returns what a key has used, for headers and `ctl`.
func (a *Accountant) Status(p auth.Principal) (spent int64, rpm, tpm float64) {
if p.KeyID == "" {
+24
View File
@@ -257,6 +257,14 @@ type Auth struct {
// blocking a request: losing accounting is bad, turning a stalled database
// into an outage is worse.
BufferSize int `koanf:"buffer_size"`
// UsageRetention is how long the record of each request is kept: who sent
// it, which replica answered, how long it queued — what `ctl request`
// reads. Zero keeps it for ever. Totals, charts and budgets are read from
// five-minute sums kept beside the records, which are not pruned, so this
// bounds the table that grows by a row a request without changing any
// figure a page shows. At least a day when set.
UsageRetention time.Duration `koanf:"usage_retention"`
}
// Personal configures use that belongs to a person rather than a project.
@@ -363,6 +371,13 @@ type Router struct {
// accounting works regardless of client behaviour.
InjectUsage bool `koanf:"inject_usage"`
// TenantLabel is the level of the tree the per-tenant request series are
// labelled by — llmgw_tenant_requests_total and its siblings: "org",
// "team" or "project", or "none" for no such series. A label per project
// on a cluster of thousands is more series than Prometheus wants, and a
// team is what an alert is usually about; "team" is the default.
TenantLabel string `koanf:"tenant_label"`
// MaxBodyBytes bounds the request body the router will read.
MaxBodyBytes int64 `koanf:"max_body_bytes"`
@@ -483,6 +498,7 @@ func Default() Server {
Cooldown: 15 * time.Second,
FailureThreshold: 3,
DefaultProfile: "chat",
TenantLabel: "team",
SessionHeader: "x-session-id",
InjectUsage: true,
MaxBodyBytes: 32 << 20,
@@ -573,6 +589,14 @@ func (s Server) Validate() error {
if s.Auth.Enabled && s.Auth.Store == "" {
return fmt.Errorf("auth.store is required when auth is enabled")
}
switch s.Router.TenantLabel {
case "", "none", "org", "team", "project":
default:
return fmt.Errorf("router.tenant_label is %q: org, team, project or none", s.Router.TenantLabel)
}
if s.Auth.UsageRetention != 0 && s.Auth.UsageRetention < 24*time.Hour {
return fmt.Errorf("auth.usage_retention is %s: at least 24h, or 0 to keep every request", s.Auth.UsageRetention)
}
if s.Threads.Store != "" && s.Threads.RetainHours <= 0 {
return fmt.Errorf("threads.retain_hours must be positive when threads.store is set")
}
+14
View File
@@ -206,3 +206,17 @@ func TestEnginesDefaultToTheCheckedVersions(t *testing.T) {
t.Error("an emptied engine is still wanted")
}
}
func TestAUsageRetentionUnderADayIsRefused(t *testing.T) {
// Shorter than a day is almost certainly a unit slip — 90m for 90 days —
// and would delete every request record within the hour.
s := Default()
s.Auth.UsageRetention = 90 * time.Minute
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "usage_retention") {
t.Errorf("a 90-minute retention was accepted: %v", err)
}
s.Auth.UsageRetention = 90 * 24 * time.Hour
if err := s.Validate(); err != nil {
t.Errorf("a 90-day retention was refused: %v", err)
}
}
+9
View File
@@ -261,6 +261,7 @@ type parsedRequest struct {
KeyID string
UserID string
ProjectID string
TeamID, OrgID string
Tenant string
ProfileOverride string
PromptPrefix string
@@ -519,6 +520,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
// the truth than charging nothing.
u.PromptTokens = int64(req.PromptTokens)
}
r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
r.acct.Record(req.reservation, u)
}
@@ -537,6 +539,12 @@ func (r *Router) rejectLimit(w http.ResponseWriter, req *parsedRequest, err erro
// which is absent when waiting does not help.
writeError(w, http.StatusTooManyRequests, "rate_limit_exceeded", le.Error())
r.m.observeRequest(r.metricModel(req.Model), "", http.StatusTooManyRequests, time.Since(start), nil, 0)
r.m.tenant.observe(req, http.StatusTooManyRequests, 0, 0, 0)
r.acct.RecordRefused(store.Usage{
At: time.Now(), KeyID: req.KeyID, UserID: req.UserID, ProjectID: req.ProjectID,
Model: req.Model, ServedModel: req.Model, Status: http.StatusTooManyRequests,
LatencyMs: time.Since(start).Milliseconds(), RequestID: req.RequestID,
})
r.accessLog(req, dispatchResult{err: err}, http.StatusTooManyRequests, time.Since(start))
}
@@ -579,6 +587,7 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
p.KeyID = principal.KeyID
p.UserID = principal.UserID
p.ProjectID = principal.ProjectID
p.TeamID, p.OrgID = principal.TeamID, principal.OrgID
// Tenant fairness within a band is per key rather than per session, so
// one key cannot take a whole band by opening many sessions.
p.Tenant = principal.KeyID
+5 -2
View File
@@ -29,6 +29,8 @@ type metrics struct {
admissionRejected *prometheus.CounterVec
prefixDecisions *prometheus.CounterVec
tenant *tenantMetrics
}
func newMetrics() *metrics {
@@ -86,16 +88,17 @@ func newMetrics() *metrics {
Name: "llmgw_admission_rejected_total",
Help: "Requests refused by flow control, by reason.",
}, []string{"band", "reason"}),
tenant: newTenantMetrics(),
}
}
// collectors returns everything to register, so registration lives in one place.
func (m *metrics) collectors() []prometheus.Collector {
return []prometheus.Collector{
return append([]prometheus.Collector{
m.requests, m.duration, m.ttft, m.tokens,
m.scorerValue, m.filtered, m.considered,
m.admissionWait, m.admissionRejected, m.prefixDecisions,
}
}, m.tenant.collectors()...)
}
// observePrefix records which branch a prefix decision took.
+74
View File
@@ -7,6 +7,8 @@ import (
"testing"
"time"
"github.com/prometheus/client_golang/prometheus/testutil"
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
@@ -188,3 +190,75 @@ func TestAnIDThatIsNotOneIsReplaced(t *testing.T) {
t.Error("a plain id was not kept")
}
}
func TestARequestALimitRefusesIsCountedAgainstItsOwner(t *testing.T) {
// Refused requests were counted only in Prometheus, by model: a team
// running into its limit looked, on every usage page, like a team that
// had gone quiet.
e := newFakeEngine(t, "qwen2.5-14b-instruct")
rt, secret := pricedRouter(t, e.srv.URL)
rpm := 1
if err := rt.store.PutLimits(t.Context(), store.Limits{Scope: store.ScopeKey, ScopeID: "key-1", RPM: &rpm}); err != nil {
t.Fatal(err)
}
body := map[string]any{"model": "q14", "messages": []any{map[string]any{"role": "user", "content": "hi"}}}
auth := map[string]string{"Authorization": "Bearer " + secret}
codes := []int{post(t, rt.Handler(), body, auth).Code, post(t, rt.Handler(), body, auth).Code}
if codes[1] != 429 {
t.Fatalf("the second request in the minute answered %v, want 429", codes)
}
deadline := time.Now().Add(2 * time.Second)
for time.Now().Before(deadline) {
stats, err := rt.store.StatsBy(t.Context(), store.GroupByProject, time.Now().Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
for _, s := range stats {
if s.Limited == 1 && s.Requests == 2 {
spend, _ := rt.store.SpendSince(t.Context(), time.Now().Add(-time.Hour))
if spend["key-1"] == 0 {
t.Error("the request that was served cost nothing")
}
return
}
}
time.Sleep(20 * time.Millisecond)
}
t.Fatal("the refused request was never counted against its project")
}
func TestTenantsAreCountedAndTheirBudgetsGauged(t *testing.T) {
e := newFakeEngine(t, "qwen2.5-14b-instruct")
rt, secret := pricedRouter(t, e.srv.URL)
ctx := t.Context()
orgs, _ := rt.store.AllOrgs(ctx)
teams, _ := rt.store.AllTeams(ctx)
budget := int64(1_000_000)
if err := rt.store.PutLimits(ctx, store.Limits{Scope: store.ScopeTeam, ScopeID: teams[0].ID,
Budget: &budget, BudgetPeriod: store.PeriodMonth}); err != nil {
t.Fatal(err)
}
if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
t.Fatal(err)
}
rec := post(t, rt.Handler(), map[string]any{"model": "q14", "messages": []any{
map[string]any{"role": "user", "content": "hello"}}},
map[string]string{"Authorization": "Bearer " + secret})
if rec.Code != 200 {
t.Fatalf("status %d", rec.Code)
}
tenant := orgs[0].Name + "/" + teams[0].Name
if got := testutil.ToFloat64(rt.m.tenant.requests.WithLabelValues(tenant, "ok")); got != 1 {
t.Errorf("%s has %v ok requests, want 1", tenant, got)
}
u := waitForUsage(t, rt)
if err := rt.m.tenant.refresh(ctx, rt.store, time.Now()); err != nil {
t.Fatal(err)
}
want := float64(u.CostMicros) / float64(budget)
got := testutil.ToFloat64(rt.m.tenant.budget.WithLabelValues("team", teams[0].ID, tenant, "month"))
if want == 0 || got != want {
t.Errorf("the team's budget gauge reads %v, want %v", got, want)
}
}
+59 -1
View File
@@ -83,7 +83,9 @@ type Router struct {
auth *auth.Authenticator
accounts *account.Service
store *storepkg.DB
acct *accounting.Accountant
// usageRetention is how long request records are kept; zero is for ever.
usageRetention time.Duration
acct *accounting.Accountant
// threads holds conversations and memory notes. It is store when auth is on,
// its own file when only the threads block is configured, and nil when
@@ -189,6 +191,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
// the decision metric is the whole difference between tuning this and
// guessing at it.
metrics := newMetrics()
metrics.tenant.level = cfg.Router.TenantLabel
var prefixScorer *scheduler.PrefixScorer
if cfg.Router.Prefix.Enabled {
prefixScorer = scheduler.NewPrefixScorer(
@@ -248,6 +251,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
return nil, fmt.Errorf("auth store: %w", err)
}
r.store = st
r.usageRetention = cfg.Auth.UsageRetention
// Applied where no scope set one, so a project created without limits
// is still metered. Zero leaves it unlimited, which was the old
// behaviour and is now a choice rather than an omission.
@@ -412,6 +416,60 @@ func (r *Router) Start(ctx context.Context) {
if r.threads != nil {
go r.pruneThreads(ctx)
}
if r.store != nil && r.usageRetention > 0 {
go r.pruneUsage(ctx, r.usageRetention)
}
if r.store != nil {
go r.refreshTenants(ctx)
}
}
// refreshTenants keeps the tenant names the request series are labelled with,
// and the budget gauges, a minute fresh.
func (r *Router) refreshTenants(ctx context.Context) {
refresh := func() {
if err := r.m.tenant.refresh(ctx, r.store, time.Now()); err != nil && ctx.Err() == nil {
r.log.Warn("could not refresh tenant names and budget gauges", "err", err)
}
}
refresh()
t := time.NewTicker(time.Minute)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
refresh()
}
}
}
// pruneUsage deletes request records older than the retention, hourly and
// once at startup, as pruneThreads does. The five-minute sums stay, so no
// total or chart moves; what goes is the trail of each old request.
func (r *Router) pruneUsage(ctx context.Context, keep time.Duration) {
prune := func() {
n, err := r.store.PruneUsage(ctx, time.Now().Add(-keep))
if err != nil {
r.log.Warn("could not prune old request records", "err", err)
return
}
if n > 0 {
r.log.Info("removed request records past their retention", "rows", n, "retention", keep)
}
}
prune()
t := time.NewTicker(time.Hour)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
prune()
}
}
}
// pruneThreads enforces each project's retention window.
+182
View File
@@ -0,0 +1,182 @@
package router
import (
"context"
"strings"
"sync"
"time"
"github.com/prometheus/client_golang/prometheus"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// tenantMetrics are the router's series per tenant: requests by how they
// ended, tokens, time to first token, and how much of each budget is spent.
//
// The request series carry one level of the tree, chosen by
// router.tenant_label, rather than every key or project: a series per project
// on a cluster of thousands is a Prometheus that falls over, and alerting on a
// team is what an operator does. Budgets are a series per limit that has one,
// which is as many as somebody wrote.
type tenantMetrics struct {
level string
requests *prometheus.CounterVec
tokens *prometheus.CounterVec
ttft *prometheus.HistogramVec
budget *prometheus.GaugeVec
mu sync.RWMutex
names map[string]string // "team:<id>" -> "acme/platform"
}
func newTenantMetrics() *tenantMetrics {
return &tenantMetrics{
level: "team",
requests: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "llmgw_tenant_requests_total",
Help: "Requests by tenant (the level router.tenant_label names) and outcome: ok, error, or limited — refused by a rate, token or budget limit.",
}, []string{"tenant", "outcome"}),
tokens: prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "llmgw_tenant_tokens_total",
Help: "Tokens by tenant and direction.",
}, []string{"tenant", "direction"}),
ttft: prometheus.NewHistogramVec(prometheus.HistogramOpts{
Name: "llmgw_tenant_ttft_seconds",
Help: "Time to first token by tenant, for streamed requests.",
Buckets: []float64{0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60},
}, []string{"tenant"}),
budget: prometheus.NewGaugeVec(prometheus.GaugeOpts{
Name: "llmgw_budget_spent_ratio",
Help: "Spend this period over the budget, for every organization, team, project and person given one. 1 is spent.",
}, []string{"scope", "id", "name", "period"}),
names: map[string]string{},
}
}
func (t *tenantMetrics) collectors() []prometheus.Collector {
return []prometheus.Collector{t.requests, t.tokens, t.ttft, t.budget}
}
// tenantOf is the label a request is counted under: the name of its
// organization, team or project at the configured level, "personal" for use
// outside any project, or "" when tenants are not labelled at all.
func (t *tenantMetrics) tenantOf(req *parsedRequest) string {
var scope, id string
switch t.level {
case "org":
scope, id = "org", req.OrgID
case "team":
scope, id = "team", req.TeamID
case "project":
scope, id = "project", req.ProjectID
default:
return ""
}
if id == "" {
return "personal"
}
t.mu.RLock()
name, ok := t.names[scope+":"+id]
t.mu.RUnlock()
if !ok {
// Not yet in the cache — made since the last refresh. The id is
// stable, which is what a label has to be; the name follows.
return id
}
return name
}
// observe counts one finished or refused request.
func (t *tenantMetrics) observe(req *parsedRequest, status int, prompt, completion int64, ttft time.Duration) {
tenant := t.tenantOf(req)
if tenant == "" {
return
}
outcome := "ok"
switch {
case status == 429:
outcome = "limited"
case status >= 400:
outcome = "error"
}
t.requests.WithLabelValues(tenant, outcome).Inc()
if prompt > 0 {
t.tokens.WithLabelValues(tenant, "prompt").Add(float64(prompt))
}
if completion > 0 {
t.tokens.WithLabelValues(tenant, "completion").Add(float64(completion))
}
if ttft > 0 {
t.ttft.WithLabelValues(tenant).Observe(ttft.Seconds())
}
}
// tenantStore is what the refresh reads.
type tenantStore interface {
Tree(ctx context.Context) ([]storepkg.TreeOrg, error)
AllLimits(ctx context.Context) (map[string]storepkg.Limits, error)
SpendOfPool(ctx context.Context, l storepkg.Limits, since time.Time) (int64, error)
Users(ctx context.Context, state storepkg.UserState) ([]storepkg.User, error)
}
// refresh reads the tree's names and every budget's spend. A minute old at
// most, which is plenty for an alert that waits five before firing.
func (t *tenantMetrics) refresh(ctx context.Context, s tenantStore, now time.Time) error {
tree, err := s.Tree(ctx)
if err != nil {
return err
}
names := map[string]string{}
for _, o := range tree {
names["org:"+o.ID] = o.Name
for _, tm := range o.Teams {
names["team:"+tm.ID] = o.Name + "/" + tm.Name
for _, p := range tm.Projects {
names["project:"+p.ID] = o.Name + "/" + tm.Name + "/" + p.Name
}
}
}
t.mu.Lock()
t.names = names
t.mu.Unlock()
limits, err := s.AllLimits(ctx)
if err != nil {
return err
}
emails := map[string]string{}
if users, err := s.Users(ctx, ""); err == nil {
for _, u := range users {
emails[u.ID] = u.Email
}
}
// Rebuilt whole, so a budget removed since the last refresh stops being
// published rather than staying at its last value for ever.
t.budget.Reset()
for _, l := range limits {
if l.Budget == nil || *l.Budget <= 0 {
continue
}
name := ""
switch l.Scope {
case storepkg.ScopeOrg, storepkg.ScopeTeam, storepkg.ScopeProject:
name = names[string(l.Scope)+":"+l.ScopeID]
case storepkg.ScopeUser:
name = emails[l.ScopeID]
default:
// Per key and per member are too many series and too little
// to act on; their pools are bounded by the levels above.
continue
}
spent, err := s.SpendOfPool(ctx, l, accounting.PeriodStart(l.BudgetPeriod, now))
if err != nil {
return err
}
t.budget.WithLabelValues(string(l.Scope), l.ScopeID, strings.TrimSpace(name), string(l.BudgetPeriod)).
Set(float64(spent) / float64(*l.Budget))
}
return nil
}
+5
View File
@@ -33,6 +33,11 @@ var copyOrder = []string{
// quota on a scope that is later removed is inert, not an error.
"capacity_quotas",
"usage",
// Copied rather than rebuilt from the rows: once request rows have a
// retention, the sums are the only record of what is older.
"usage_5m",
"usage_1h",
"usage_1d",
"models",
// No foreign key to models, deliberately: a measurement outlives the
// catalog entry it was taken for, and losing it when somebody removes a
+132
View File
@@ -0,0 +1,132 @@
package store
import (
"context"
"fmt"
"strings"
"time"
)
// Quantiles are a duration's percentiles, in milliseconds, read from the
// histogram kept with the usage sums. Within a bucket the value is
// interpolated; above the last bucket's edge it is that edge, and Over says
// so. N is how many durations they are read from — zero is no reading.
type Quantiles struct {
P50, P95, P99 int64
N int64
Over bool
}
// GroupStats is one group's requests, how they ended, and how long they took.
type GroupStats struct {
ID string
Requests int64
Errors int64
// Limited is requests refused by a rate, token or budget limit.
Limited int64
Latency Quantiles
// TTFT is time to first token, for requests that streamed one.
TTFT Quantiles
}
// ErrorRate is the share of requests that failed, limits included, in percent.
func (g GroupStats) ErrorRate() float64 {
if g.Requests == 0 {
return 0
}
return 100 * float64(g.Errors) / float64(g.Requests)
}
// StatsBy is each group's request outcomes and latency percentiles since a
// moment, keyed by the group's id.
//
// Read from the hourly sums for a window of two days or more and from the
// five-minute ones otherwise, starting at the slice the moment falls in:
// percentiles are a shape, and a window starting up to one slice early does
// not change one. Budgets, which must be exact, are not read this way.
func (s *DB) StatsBy(ctx context.Context, g Grouping, since time.Time) (map[string]GroupStats, error) {
c, ok := groupColumns[g]
if !ok {
return nil, fmt.Errorf("usage cannot be grouped by %q", g)
}
t := tier5m
if time.Since(since) >= 48*time.Hour {
t = tier1h
}
n := len(LatencyBounds) + 1
var sums []string
for i := range n {
sums = append(sums, fmt.Sprintf("SUM(u.lat_%d)", i))
}
for i := range n {
sums = append(sums, fmt.Sprintf("SUM(u.ttft_%d)", i))
}
rows, err := s.db.QueryContext(ctx, `
SELECT `+c.id+`, SUM(u.requests), SUM(u.errors), SUM(u.limited), `+strings.Join(sums, ", ")+`
FROM `+t.table+` u`+c.joins+`
WHERE u.bucket >= ?
GROUP BY 1`, t.floor(since))
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]GroupStats{}
for rows.Next() {
var gs GroupStats
lat, ttft := make([]int64, n), make([]int64, n)
dst := []any{&gs.ID, &gs.Requests, &gs.Errors, &gs.Limited}
for i := range lat {
dst = append(dst, &lat[i])
}
for i := range ttft {
dst = append(dst, &ttft[i])
}
if err := rows.Scan(dst...); err != nil {
return nil, err
}
gs.Latency, gs.TTFT = quantiles(lat), quantiles(ttft)
out[gs.ID] = gs
}
return out, rows.Err()
}
// quantiles reads p50, p95 and p99 from counts per LatencyBounds bucket.
func quantiles(counts []int64) Quantiles {
var q Quantiles
for _, c := range counts {
q.N += c
}
if q.N == 0 {
return q
}
at := func(p float64) (int64, bool) {
rank := p * float64(q.N)
var seen float64
for i, c := range counts {
if c == 0 {
continue
}
if seen+float64(c) >= rank {
if i >= len(LatencyBounds) {
return LatencyBounds[len(LatencyBounds)-1], true
}
lo := int64(0)
if i > 0 {
lo = LatencyBounds[i-1]
}
hi := LatencyBounds[i]
frac := (rank - seen) / float64(c)
return lo + int64(frac*float64(hi-lo)), false
}
seen += float64(c)
}
return LatencyBounds[len(LatencyBounds)-1], true
}
var over bool
q.P50, _ = at(0.50)
q.P95, _ = at(0.95)
q.P99, over = at(0.99)
q.Over = over
return q
}
+61
View File
@@ -0,0 +1,61 @@
package store
import (
"context"
"testing"
"time"
)
func TestQuantilesAreReadFromTheHistogram(t *testing.T) {
// 100 durations: 90 at <=100 ms, 8 in (1000, 2500], 2 above two minutes.
counts := make([]int64, len(LatencyBounds)+1)
counts[0], counts[4], counts[len(counts)-1] = 90, 8, 2
q := quantiles(counts)
if q.N != 100 || q.P50 > 100 || q.P95 <= 1000 || q.P95 > 2500 || !q.Over {
t.Errorf("%+v: want p50 within 100 ms, p95 in (1000, 2500], p99 past the last edge", q)
}
if (quantiles(make([]int64, len(LatencyBounds)+1)) != Quantiles{}) {
t.Error("no durations read as some")
}
}
func TestStatsSayHowEachGroupFaredAndHowLongItTook(t *testing.T) {
onBoth(t, func(t *testing.T, s *DB) {
ctx := context.Background()
at := slice()
var batch []Usage
for i := range 20 {
batch = append(batch, Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big",
ServedModel: "big", Status: 200, LatencyMs: 4000, TTFTMs: int64(200 + i)})
}
batch = append(batch,
Usage{At: at, KeyID: "k", UserID: "ayse", ProjectID: "p", Model: "big", ServedModel: "big", Status: 502, LatencyMs: 50},
Usage{At: at, KeyID: "k2", UserID: "bora", ProjectID: "p", Model: "small", ServedModel: "small", Status: 429},
)
if err := s.WriteUsage(ctx, batch); err != nil {
t.Fatal(err)
}
byUser, err := s.StatsBy(ctx, GroupByUser, at.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
a, b := byUser["ayse"], byUser["bora"]
if a.Requests != 21 || a.Errors != 1 || a.Limited != 0 || b.Limited != 1 || b.Errors != 1 {
t.Errorf("outcomes: ayse %+v, bora %+v", a, b)
}
if a.TTFT.N != 20 || a.TTFT.P95 <= 100 || a.TTFT.P95 > 250 || a.Latency.P95 <= 2500 || a.Latency.P95 > 5000 {
t.Errorf("ayse's first token %+v and latency %+v", a.TTFT, a.Latency)
}
byModel, err := s.StatsBy(ctx, GroupByModel, at.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if byModel["big"].Requests != 21 || byModel["small"].Requests != 1 {
t.Errorf("by model: %+v", byModel)
}
series, err := s.UsageBucketsBy(ctx, GroupByModel, at.Add(-time.Hour), time.Hour)
if err != nil || len(series) != 2 {
t.Fatalf("usage over time by model: %v, %+v", err, series)
}
})
}
@@ -0,0 +1,208 @@
-- Ported from the SQLite migration of the same number.
-- Usage summed as it is written, per key, person, project and model: into
-- five-minute slices (usage_5m), hours (usage_1h) and days (usage_1d).
--
-- Every chart and total read the raw table, one row per request, and summed it
-- again on every page view: fine for thousands of requests a day, and a full
-- scan of tens of millions on a cluster in real use. The sums are kept in the
-- transaction that writes the rows, so they never disagree with them; the raw
-- table stays for following one request, and can be given a retention of its
-- own (auth.usage_retention).
--
-- Three widths because how well a slice compresses depends on traffic: many
-- keys each sending a little leave a five-minute slice with nearly a row per
-- request, which an hour or a day folds much further. A chart reads the
-- coarsest width its slices are made of; a total since a moment reads the
-- rows up to the next five-minute slice, those slices up to the hour, hours
-- up to the day and days from there — exact, and a month in about thirty
-- rows per key and model.
--
-- Latency and time to first token are kept as histograms, a count per fixed
-- bucket (ms: <=100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000,
-- and above), which is what a percentile per team or model is read from after
-- the rows are gone; a mean cannot do that. ttft_* counts only requests whose
-- first token was measured. limited counts requests refused with 429 — a rate,
-- token or budget limit — which are recorded with no tokens and no cost.
CREATE TABLE usage_5m (
bucket BIGINT NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests BIGINT NOT NULL DEFAULT 0,
errors BIGINT NOT NULL DEFAULT 0,
limited BIGINT NOT NULL DEFAULT 0,
prompt_tokens BIGINT NOT NULL DEFAULT 0,
completion_tokens BIGINT NOT NULL DEFAULT 0,
cost_micros BIGINT NOT NULL DEFAULT 0,
latency_ms_sum BIGINT NOT NULL DEFAULT 0,
queue_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_n BIGINT NOT NULL DEFAULT 0,
last_ts BIGINT NOT NULL DEFAULT 0,
lat_0 BIGINT NOT NULL DEFAULT 0,
lat_1 BIGINT NOT NULL DEFAULT 0,
lat_2 BIGINT NOT NULL DEFAULT 0,
lat_3 BIGINT NOT NULL DEFAULT 0,
lat_4 BIGINT NOT NULL DEFAULT 0,
lat_5 BIGINT NOT NULL DEFAULT 0,
lat_6 BIGINT NOT NULL DEFAULT 0,
lat_7 BIGINT NOT NULL DEFAULT 0,
lat_8 BIGINT NOT NULL DEFAULT 0,
lat_9 BIGINT NOT NULL DEFAULT 0,
lat_10 BIGINT NOT NULL DEFAULT 0,
ttft_0 BIGINT NOT NULL DEFAULT 0,
ttft_1 BIGINT NOT NULL DEFAULT 0,
ttft_2 BIGINT NOT NULL DEFAULT 0,
ttft_3 BIGINT NOT NULL DEFAULT 0,
ttft_4 BIGINT NOT NULL DEFAULT 0,
ttft_5 BIGINT NOT NULL DEFAULT 0,
ttft_6 BIGINT NOT NULL DEFAULT 0,
ttft_7 BIGINT NOT NULL DEFAULT 0,
ttft_8 BIGINT NOT NULL DEFAULT 0,
ttft_9 BIGINT NOT NULL DEFAULT 0,
ttft_10 BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_5m_by_bucket ON usage_5m(bucket);
CREATE INDEX usage_5m_by_project ON usage_5m(project_id, bucket);
CREATE INDEX usage_5m_by_user ON usage_5m(user_id, bucket);
CREATE INDEX usage_5m_by_key ON usage_5m(key_id, bucket);
INSERT INTO usage_5m (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 300000) * 300000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
CREATE TABLE usage_1h (
bucket BIGINT NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests BIGINT NOT NULL DEFAULT 0,
errors BIGINT NOT NULL DEFAULT 0,
limited BIGINT NOT NULL DEFAULT 0,
prompt_tokens BIGINT NOT NULL DEFAULT 0,
completion_tokens BIGINT NOT NULL DEFAULT 0,
cost_micros BIGINT NOT NULL DEFAULT 0,
latency_ms_sum BIGINT NOT NULL DEFAULT 0,
queue_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_n BIGINT NOT NULL DEFAULT 0,
last_ts BIGINT NOT NULL DEFAULT 0,
lat_0 BIGINT NOT NULL DEFAULT 0,
lat_1 BIGINT NOT NULL DEFAULT 0,
lat_2 BIGINT NOT NULL DEFAULT 0,
lat_3 BIGINT NOT NULL DEFAULT 0,
lat_4 BIGINT NOT NULL DEFAULT 0,
lat_5 BIGINT NOT NULL DEFAULT 0,
lat_6 BIGINT NOT NULL DEFAULT 0,
lat_7 BIGINT NOT NULL DEFAULT 0,
lat_8 BIGINT NOT NULL DEFAULT 0,
lat_9 BIGINT NOT NULL DEFAULT 0,
lat_10 BIGINT NOT NULL DEFAULT 0,
ttft_0 BIGINT NOT NULL DEFAULT 0,
ttft_1 BIGINT NOT NULL DEFAULT 0,
ttft_2 BIGINT NOT NULL DEFAULT 0,
ttft_3 BIGINT NOT NULL DEFAULT 0,
ttft_4 BIGINT NOT NULL DEFAULT 0,
ttft_5 BIGINT NOT NULL DEFAULT 0,
ttft_6 BIGINT NOT NULL DEFAULT 0,
ttft_7 BIGINT NOT NULL DEFAULT 0,
ttft_8 BIGINT NOT NULL DEFAULT 0,
ttft_9 BIGINT NOT NULL DEFAULT 0,
ttft_10 BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_1h_by_bucket ON usage_1h(bucket);
CREATE INDEX usage_1h_by_project ON usage_1h(project_id, bucket);
CREATE INDEX usage_1h_by_user ON usage_1h(user_id, bucket);
CREATE INDEX usage_1h_by_key ON usage_1h(key_id, bucket);
INSERT INTO usage_1h (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 3600000) * 3600000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
CREATE TABLE usage_1d (
bucket BIGINT NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests BIGINT NOT NULL DEFAULT 0,
errors BIGINT NOT NULL DEFAULT 0,
limited BIGINT NOT NULL DEFAULT 0,
prompt_tokens BIGINT NOT NULL DEFAULT 0,
completion_tokens BIGINT NOT NULL DEFAULT 0,
cost_micros BIGINT NOT NULL DEFAULT 0,
latency_ms_sum BIGINT NOT NULL DEFAULT 0,
queue_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_ms_sum BIGINT NOT NULL DEFAULT 0,
ttft_n BIGINT NOT NULL DEFAULT 0,
last_ts BIGINT NOT NULL DEFAULT 0,
lat_0 BIGINT NOT NULL DEFAULT 0,
lat_1 BIGINT NOT NULL DEFAULT 0,
lat_2 BIGINT NOT NULL DEFAULT 0,
lat_3 BIGINT NOT NULL DEFAULT 0,
lat_4 BIGINT NOT NULL DEFAULT 0,
lat_5 BIGINT NOT NULL DEFAULT 0,
lat_6 BIGINT NOT NULL DEFAULT 0,
lat_7 BIGINT NOT NULL DEFAULT 0,
lat_8 BIGINT NOT NULL DEFAULT 0,
lat_9 BIGINT NOT NULL DEFAULT 0,
lat_10 BIGINT NOT NULL DEFAULT 0,
ttft_0 BIGINT NOT NULL DEFAULT 0,
ttft_1 BIGINT NOT NULL DEFAULT 0,
ttft_2 BIGINT NOT NULL DEFAULT 0,
ttft_3 BIGINT NOT NULL DEFAULT 0,
ttft_4 BIGINT NOT NULL DEFAULT 0,
ttft_5 BIGINT NOT NULL DEFAULT 0,
ttft_6 BIGINT NOT NULL DEFAULT 0,
ttft_7 BIGINT NOT NULL DEFAULT 0,
ttft_8 BIGINT NOT NULL DEFAULT 0,
ttft_9 BIGINT NOT NULL DEFAULT 0,
ttft_10 BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_1d_by_bucket ON usage_1d(bucket);
CREATE INDEX usage_1d_by_project ON usage_1d(project_id, bucket);
CREATE INDEX usage_1d_by_user ON usage_1d(user_id, bucket);
CREATE INDEX usage_1d_by_key ON usage_1d(key_id, bucket);
INSERT INTO usage_1d (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 86400000) * 86400000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
@@ -0,0 +1,207 @@
-- Usage summed as it is written, per key, person, project and model: into
-- five-minute slices (usage_5m), hours (usage_1h) and days (usage_1d).
--
-- Every chart and total read the raw table, one row per request, and summed it
-- again on every page view: fine for thousands of requests a day, and a full
-- scan of tens of millions on a cluster in real use. The sums are kept in the
-- transaction that writes the rows, so they never disagree with them; the raw
-- table stays for following one request, and can be given a retention of its
-- own (auth.usage_retention).
--
-- Three widths because how well a slice compresses depends on traffic: many
-- keys each sending a little leave a five-minute slice with nearly a row per
-- request, which an hour or a day folds much further. A chart reads the
-- coarsest width its slices are made of; a total since a moment reads the
-- rows up to the next five-minute slice, those slices up to the hour, hours
-- up to the day and days from there — exact, and a month in about thirty
-- rows per key and model.
--
-- Latency and time to first token are kept as histograms, a count per fixed
-- bucket (ms: <=100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000,
-- and above), which is what a percentile per team or model is read from after
-- the rows are gone; a mean cannot do that. ttft_* counts only requests whose
-- first token was measured. limited counts requests refused with 429 — a rate,
-- token or budget limit — which are recorded with no tokens and no cost.
CREATE TABLE usage_5m (
bucket INTEGER NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests INTEGER NOT NULL DEFAULT 0,
errors INTEGER NOT NULL DEFAULT 0,
limited INTEGER NOT NULL DEFAULT 0,
prompt_tokens INTEGER NOT NULL DEFAULT 0,
completion_tokens INTEGER NOT NULL DEFAULT 0,
cost_micros INTEGER NOT NULL DEFAULT 0,
latency_ms_sum INTEGER NOT NULL DEFAULT 0,
queue_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_n INTEGER NOT NULL DEFAULT 0,
last_ts INTEGER NOT NULL DEFAULT 0,
lat_0 INTEGER NOT NULL DEFAULT 0,
lat_1 INTEGER NOT NULL DEFAULT 0,
lat_2 INTEGER NOT NULL DEFAULT 0,
lat_3 INTEGER NOT NULL DEFAULT 0,
lat_4 INTEGER NOT NULL DEFAULT 0,
lat_5 INTEGER NOT NULL DEFAULT 0,
lat_6 INTEGER NOT NULL DEFAULT 0,
lat_7 INTEGER NOT NULL DEFAULT 0,
lat_8 INTEGER NOT NULL DEFAULT 0,
lat_9 INTEGER NOT NULL DEFAULT 0,
lat_10 INTEGER NOT NULL DEFAULT 0,
ttft_0 INTEGER NOT NULL DEFAULT 0,
ttft_1 INTEGER NOT NULL DEFAULT 0,
ttft_2 INTEGER NOT NULL DEFAULT 0,
ttft_3 INTEGER NOT NULL DEFAULT 0,
ttft_4 INTEGER NOT NULL DEFAULT 0,
ttft_5 INTEGER NOT NULL DEFAULT 0,
ttft_6 INTEGER NOT NULL DEFAULT 0,
ttft_7 INTEGER NOT NULL DEFAULT 0,
ttft_8 INTEGER NOT NULL DEFAULT 0,
ttft_9 INTEGER NOT NULL DEFAULT 0,
ttft_10 INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_5m_by_bucket ON usage_5m(bucket);
CREATE INDEX usage_5m_by_project ON usage_5m(project_id, bucket);
CREATE INDEX usage_5m_by_user ON usage_5m(user_id, bucket);
CREATE INDEX usage_5m_by_key ON usage_5m(key_id, bucket);
INSERT INTO usage_5m (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 300000) * 300000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
CREATE TABLE usage_1h (
bucket INTEGER NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests INTEGER NOT NULL DEFAULT 0,
errors INTEGER NOT NULL DEFAULT 0,
limited INTEGER NOT NULL DEFAULT 0,
prompt_tokens INTEGER NOT NULL DEFAULT 0,
completion_tokens INTEGER NOT NULL DEFAULT 0,
cost_micros INTEGER NOT NULL DEFAULT 0,
latency_ms_sum INTEGER NOT NULL DEFAULT 0,
queue_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_n INTEGER NOT NULL DEFAULT 0,
last_ts INTEGER NOT NULL DEFAULT 0,
lat_0 INTEGER NOT NULL DEFAULT 0,
lat_1 INTEGER NOT NULL DEFAULT 0,
lat_2 INTEGER NOT NULL DEFAULT 0,
lat_3 INTEGER NOT NULL DEFAULT 0,
lat_4 INTEGER NOT NULL DEFAULT 0,
lat_5 INTEGER NOT NULL DEFAULT 0,
lat_6 INTEGER NOT NULL DEFAULT 0,
lat_7 INTEGER NOT NULL DEFAULT 0,
lat_8 INTEGER NOT NULL DEFAULT 0,
lat_9 INTEGER NOT NULL DEFAULT 0,
lat_10 INTEGER NOT NULL DEFAULT 0,
ttft_0 INTEGER NOT NULL DEFAULT 0,
ttft_1 INTEGER NOT NULL DEFAULT 0,
ttft_2 INTEGER NOT NULL DEFAULT 0,
ttft_3 INTEGER NOT NULL DEFAULT 0,
ttft_4 INTEGER NOT NULL DEFAULT 0,
ttft_5 INTEGER NOT NULL DEFAULT 0,
ttft_6 INTEGER NOT NULL DEFAULT 0,
ttft_7 INTEGER NOT NULL DEFAULT 0,
ttft_8 INTEGER NOT NULL DEFAULT 0,
ttft_9 INTEGER NOT NULL DEFAULT 0,
ttft_10 INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_1h_by_bucket ON usage_1h(bucket);
CREATE INDEX usage_1h_by_project ON usage_1h(project_id, bucket);
CREATE INDEX usage_1h_by_user ON usage_1h(user_id, bucket);
CREATE INDEX usage_1h_by_key ON usage_1h(key_id, bucket);
INSERT INTO usage_1h (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 3600000) * 3600000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
CREATE TABLE usage_1d (
bucket INTEGER NOT NULL,
key_id TEXT NOT NULL,
user_id TEXT NOT NULL,
project_id TEXT NOT NULL,
model TEXT NOT NULL,
served_model TEXT NOT NULL,
requests INTEGER NOT NULL DEFAULT 0,
errors INTEGER NOT NULL DEFAULT 0,
limited INTEGER NOT NULL DEFAULT 0,
prompt_tokens INTEGER NOT NULL DEFAULT 0,
completion_tokens INTEGER NOT NULL DEFAULT 0,
cost_micros INTEGER NOT NULL DEFAULT 0,
latency_ms_sum INTEGER NOT NULL DEFAULT 0,
queue_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_ms_sum INTEGER NOT NULL DEFAULT 0,
ttft_n INTEGER NOT NULL DEFAULT 0,
last_ts INTEGER NOT NULL DEFAULT 0,
lat_0 INTEGER NOT NULL DEFAULT 0,
lat_1 INTEGER NOT NULL DEFAULT 0,
lat_2 INTEGER NOT NULL DEFAULT 0,
lat_3 INTEGER NOT NULL DEFAULT 0,
lat_4 INTEGER NOT NULL DEFAULT 0,
lat_5 INTEGER NOT NULL DEFAULT 0,
lat_6 INTEGER NOT NULL DEFAULT 0,
lat_7 INTEGER NOT NULL DEFAULT 0,
lat_8 INTEGER NOT NULL DEFAULT 0,
lat_9 INTEGER NOT NULL DEFAULT 0,
lat_10 INTEGER NOT NULL DEFAULT 0,
ttft_0 INTEGER NOT NULL DEFAULT 0,
ttft_1 INTEGER NOT NULL DEFAULT 0,
ttft_2 INTEGER NOT NULL DEFAULT 0,
ttft_3 INTEGER NOT NULL DEFAULT 0,
ttft_4 INTEGER NOT NULL DEFAULT 0,
ttft_5 INTEGER NOT NULL DEFAULT 0,
ttft_6 INTEGER NOT NULL DEFAULT 0,
ttft_7 INTEGER NOT NULL DEFAULT 0,
ttft_8 INTEGER NOT NULL DEFAULT 0,
ttft_9 INTEGER NOT NULL DEFAULT 0,
ttft_10 INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (bucket, key_id, user_id, project_id, model, served_model)
);
CREATE INDEX usage_1d_by_bucket ON usage_1d(bucket);
CREATE INDEX usage_1d_by_project ON usage_1d(project_id, bucket);
CREATE INDEX usage_1d_by_user ON usage_1d(user_id, bucket);
CREATE INDEX usage_1d_by_key ON usage_1d(key_id, bucket);
INSERT INTO usage_1d (bucket, key_id, user_id, project_id, model, served_model,
requests, errors, limited, prompt_tokens, completion_tokens, cost_micros,
latency_ms_sum, queue_ms_sum, ttft_ms_sum, ttft_n, last_ts,
lat_0, lat_1, lat_2, lat_3, lat_4, lat_5, lat_6, lat_7, lat_8, lat_9, lat_10,
ttft_0, ttft_1, ttft_2, ttft_3, ttft_4, ttft_5, ttft_6, ttft_7, ttft_8, ttft_9, ttft_10)
SELECT (ts / 86400000) * 86400000, key_id, COALESCE(user_id, ''), project_id, model, served_model,
COUNT(*), SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END), SUM(CASE WHEN status = 429 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
SUM(latency_ms), SUM(queue_ms), SUM(ttft_ms), SUM(CASE WHEN ttft_ms > 0 THEN 1 ELSE 0 END), MAX(ts),
SUM(CASE WHEN latency_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 100 AND latency_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 250 AND latency_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 500 AND latency_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 1000 AND latency_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 2500 AND latency_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 5000 AND latency_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 10000 AND latency_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 30000 AND latency_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 60000 AND latency_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN latency_ms > 120000 THEN 1 ELSE 0 END),
SUM(CASE WHEN ttft_ms > 0 AND ttft_ms <= 100 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 100 AND ttft_ms <= 250 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 250 AND ttft_ms <= 500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 500 AND ttft_ms <= 1000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 1000 AND ttft_ms <= 2500 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 2500 AND ttft_ms <= 5000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 5000 AND ttft_ms <= 10000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 10000 AND ttft_ms <= 30000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 30000 AND ttft_ms <= 60000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 60000 AND ttft_ms <= 120000 THEN 1 ELSE 0 END), SUM(CASE WHEN ttft_ms > 0 AND ttft_ms > 120000 THEN 1 ELSE 0 END)
FROM usage
GROUP BY 1, 2, 3, 4, 5, 6;
+1 -1
View File
@@ -41,7 +41,7 @@ func openPG(t *testing.T) *DB {
// inherits the previous one's lease is refused for reasons that have
// nothing to do with what it is testing. It passed on SQLite, where
// every subtest gets a fresh file, and failed only here.
`TRUNCATE thread_messages, threads, memory_notes, usage, api_keys, memberships,
`TRUNCATE thread_messages, threads, memory_notes, usage, usage_5m, usage_1h, usage_1d, api_keys, memberships,
key_claims, limits, models, sessions, users, projects, teams,
organizations, leadership RESTART IDENTITY CASCADE`)
if err != nil {
+211
View File
@@ -0,0 +1,211 @@
package store
import (
"context"
"fmt"
"strings"
"time"
)
// tier is one table usage is summed into as it is written: slices of a fixed
// width, per key, person, project and model.
type tier struct {
table string
ms int64
}
var (
tier5m = tier{"usage_5m", int64(5 * time.Minute / time.Millisecond)}
tier1h = tier{"usage_1h", int64(time.Hour / time.Millisecond)}
tier1d = tier{"usage_1d", int64(24 * time.Hour / time.Millisecond)}
// tiers, finest first.
tiers = []tier{tier5m, tier1h, tier1d}
)
// floor is the start of the slice a moment falls in. A chart read from a
// tier starts its first slice there, as a chart read from the rows always did.
func (t tier) floor(at time.Time) int64 { return (millis(at) / t.ms) * t.ms }
// ceil is the start of the first slice wholly after a moment.
func (t tier) ceil(at time.Time) int64 {
return ((millis(at) + t.ms - 1) / t.ms) * t.ms
}
// tierForSlices is the coarsest tier whose slices make up a chart slice of
// width exactly, and false when none does — a slice finer than five minutes,
// or not a whole number of them, is read from the rows.
func tierForSlices(width time.Duration) (tier, bool) {
ms := width.Milliseconds()
for i := len(tiers) - 1; i >= 0; i-- {
if ms >= tiers[i].ms && ms%tiers[i].ms == 0 {
return tiers[i], true
}
}
return tier{}, false
}
// LatencyBounds are the upper edges, in milliseconds, of the histogram buckets
// latency and time to first token are counted in; one more bucket holds
// everything above the last. Fixed, because buckets are only summable across
// slices, keys and teams when every row has the same ones.
var LatencyBounds = []int64{100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 120000}
// latencyBucket is the histogram bucket a duration is counted in.
func latencyBucket(ms int64) int {
for i, b := range LatencyBounds {
if ms <= b {
return i
}
}
return len(LatencyBounds)
}
// rollupKey is one row of a tier: a slice, and who spent what on what.
type rollupKey struct {
bucket int64
keyID, userID, projectID, model, servedModel string
}
// rollupRow is what a batch adds to one row.
type rollupRow struct {
requests, errors, limited, prompt int64
completion, cost int64
latency, queue, ttft, ttftN, last int64
lat, ttftH []int64
}
// rollupBatch sums a batch of records into the rows of one tier they add to.
func rollupBatch(t tier, records []Usage) (map[rollupKey]*rollupRow, []rollupKey) {
rows := map[rollupKey]*rollupRow{}
var order []rollupKey
n := len(LatencyBounds) + 1
for _, r := range records {
ts := millis(r.At)
k := rollupKey{(ts / t.ms) * t.ms, r.KeyID, r.UserID, r.ProjectID, r.Model, r.ServedModel}
row, ok := rows[k]
if !ok {
row = &rollupRow{lat: make([]int64, n), ttftH: make([]int64, n)}
rows[k] = row
order = append(order, k)
}
row.requests++
if r.Status >= 400 {
row.errors++
}
if r.Status == 429 {
row.limited++
}
row.prompt += r.PromptTokens
row.completion += r.CompletionTokens
row.cost += r.CostMicros
row.latency += r.LatencyMs
row.queue += r.QueueMs
row.lat[latencyBucket(r.LatencyMs)]++
if r.TTFTMs > 0 {
row.ttft += r.TTFTMs
row.ttftN++
row.ttftH[latencyBucket(r.TTFTMs)]++
}
row.last = max(row.last, ts)
}
return rows, order
}
// rollupColumns are a tier's columns, in the order rollupUpsert takes them.
var rollupColumns = func() []string {
n := len(LatencyBounds) + 1
cols := []string{"bucket", "key_id", "user_id", "project_id", "model", "served_model",
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros",
"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts"}
for i := range n {
cols = append(cols, fmt.Sprintf("lat_%d", i))
}
for i := range n {
cols = append(cols, fmt.Sprintf("ttft_%d", i))
}
return cols
}()
// rollupUpsert is the statement that adds a batch's sums to a tier.
func rollupUpsert(table string) string {
var sets []string
for _, c := range rollupColumns[6:] {
if c == "last_ts" {
sets = append(sets, "last_ts = CASE WHEN excluded.last_ts > "+table+".last_ts "+
"THEN excluded.last_ts ELSE "+table+".last_ts END")
continue
}
sets = append(sets, c+" = "+table+"."+c+" + excluded."+c)
}
return `INSERT INTO ` + table + ` (` + strings.Join(rollupColumns, ", ") + `)
VALUES (` + strings.TrimSuffix(strings.Repeat("?, ", len(rollupColumns)), ", ") + `)
ON CONFLICT (bucket, key_id, user_id, project_id, model, served_model)
DO UPDATE SET ` + strings.Join(sets, ", ")
}
// writeRollup adds a batch to every tier, inside the transaction that writes
// its rows, so the sums never disagree with them.
func writeRollup(ctx context.Context, tx conn, records []Usage) error {
for _, t := range tiers {
rows, order := rollupBatch(t, records)
stmt, err := tx.PrepareContext(ctx, rollupUpsert(t.table))
if err != nil {
return err
}
for _, k := range order {
r := rows[k]
args := []any{k.bucket, k.keyID, k.userID, k.projectID, k.model, k.servedModel,
r.requests, r.errors, r.limited, r.prompt, r.completion, r.cost,
r.latency, r.queue, r.ttft, r.ttftN, r.last}
for _, v := range r.lat {
args = append(args, v)
}
for _, v := range r.ttftH {
args = append(args, v)
}
if _, err := stmt.ExecContext(ctx, args...); err != nil {
_ = stmt.Close()
return fmt.Errorf("adding to %s: %w", t.table, err)
}
}
if err := stmt.Close(); err != nil {
return err
}
}
return nil
}
// usageSince is every request since a moment, exactly, as a table with the
// columns a tier has (but the histograms): the rows up to the next
// five-minute slice, those slices up to the next hour, hours up to the next
// day, and days from there. Totals are read from this — a budget against it
// must be exact, and so is everything else, at the cost of a few rows read
// from each of four tables rather than a window rounded to a slice.
//
// Its placeholders come first in any query it is put into.
func usageSince(since time.Time) (string, []any) {
from := millis(since)
to5m, to1h, to1d := tier5m.ceil(since), tier1h.ceil(since), tier1d.ceil(since)
const cols = `key_id, user_id, project_id, model, served_model, requests, errors, limited,
prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts`
return `(SELECT key_id, user_id, project_id, model, served_model, 1 AS requests,
CASE WHEN status >= 400 THEN 1 ELSE 0 END AS errors,
CASE WHEN status = 429 THEN 1 ELSE 0 END AS limited,
prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts
FROM usage WHERE ts >= ? AND ts < ?
UNION ALL SELECT ` + cols + ` FROM usage_5m WHERE bucket >= ? AND bucket < ?
UNION ALL SELECT ` + cols + ` FROM usage_1h WHERE bucket >= ? AND bucket < ?
UNION ALL SELECT ` + cols + ` FROM usage_1d WHERE bucket >= ?)`,
[]any{from, to5m, to5m, to1h, to1h, to1d, to1d}
}
// PruneUsage deletes request rows older than a moment. The tiers keep their
// sums, so charts, totals and budgets are unchanged; what goes is the record
// of each request — `ctl request` on one that old finds nothing.
func (s *DB) PruneUsage(ctx context.Context, before time.Time) (int64, error) {
res, err := s.db.ExecContext(ctx, `DELETE FROM usage WHERE ts < ?`, millis(before))
if err != nil {
return 0, err
}
return res.RowsAffected()
}
+219
View File
@@ -0,0 +1,219 @@
package store
import (
"context"
"database/sql"
"path/filepath"
"testing"
"time"
)
// onBoth runs a test on SQLite, and on PostgreSQL when LLMGW_TEST_POSTGRES
// names one: the rollup is written with an upsert, and the two dialects are
// where an upsert goes wrong.
func onBoth(t *testing.T, f func(t *testing.T, s *DB)) {
t.Run("sqlite", func(t *testing.T) { f(t, openDB(t)) })
t.Run("postgres", func(t *testing.T) { f(t, openPG(t)) })
}
// slice is a moment a minute into an hour, a few hours back: inside a
// five-minute slice and an hour alike.
func slice() time.Time {
return time.UnixMilli(tier1h.floor(time.Now().Add(-3 * time.Hour))).Add(time.Minute)
}
func TestTheRollupAddsUpToTheRows(t *testing.T) {
onBoth(t, func(t *testing.T, s *DB) {
ctx := context.Background()
at := slice()
// Two batches into the same slice, so the second has to add to the
// row the first made rather than replace it.
for _, batch := range [][]Usage{
{
{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
PromptTokens: 10, CompletionTokens: 5, CostMicros: 100, Status: 200, LatencyMs: 80, TTFTMs: 40},
{At: at.Add(time.Second), KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
PromptTokens: 1, CompletionTokens: 1, CostMicros: 7, Status: 500, LatencyMs: 3000},
},
{
{At: at.Add(2 * time.Second), KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
PromptTokens: 2, CompletionTokens: 3, CostMicros: 50, Status: 200, LatencyMs: 400, TTFTMs: 200},
},
} {
if err := s.WriteUsage(ctx, batch); err != nil {
t.Fatal(err)
}
}
for _, tr := range tiers {
var req, errs, prompt, cost, lat, ttftN, last, lat0, lat2, lat5, ttft0, ttft1 int64
err := s.db.QueryRowContext(ctx, `SELECT requests, errors, prompt_tokens, cost_micros,
latency_ms_sum, ttft_n, last_ts, lat_0, lat_2, lat_5, ttft_0, ttft_1 FROM `+tr.table).
Scan(&req, &errs, &prompt, &cost, &lat, &ttftN, &last, &lat0, &lat2, &lat5, &ttft0, &ttft1)
if err != nil {
t.Fatalf("%s: %v", tr.table, err)
}
if req != 3 || errs != 1 || prompt != 13 || cost != 157 || lat != 3480 {
t.Errorf("sums: %d requests, %d errors, %d prompt, %d cost, %d ms; want 3, 1, 13, 157, 3480",
req, errs, prompt, cost, lat)
}
// 80 ms in <=100, 400 in <=500, 3000 in <=5000; first tokens 40 and
// 200, and the failed request had none to count.
if lat0 != 1 || lat2 != 1 || lat5 != 1 || ttftN != 2 || ttft0 != 1 || ttft1 != 1 {
t.Errorf("histograms: lat %d/%d/%d, ttft n=%d %d/%d", lat0, lat2, lat5, ttftN, ttft0, ttft1)
}
if last != millis(at.Add(2*time.Second)) {
t.Errorf("last seen %d, want the latest request", last)
}
}
b, err := s.UsageBuckets(ctx, at.Add(-time.Hour), time.Hour)
if err != nil {
t.Fatal(err)
}
var total Bucket
for _, x := range b {
total.Requests += x.Requests
total.CostMicros += x.CostMicros
}
if total.Requests != 3 || total.CostMicros != 157 {
t.Errorf("the chart counts %d requests and %d spent, want 3 and 157", total.Requests, total.CostMicros)
}
})
}
func TestABudgetIsCountedFromTheMomentNotTheSlice(t *testing.T) {
// Charts start at the slice a window falls in; a budget must not. A
// request a minute before the period began is not the period's.
onBoth(t, func(t *testing.T, s *DB) {
ctx := context.Background()
at := slice()
// Before the moment; after it in its own slice; in a later slice of
// the same hour; and in the next hour — one from each part a budget
// is read from.
if err := s.WriteUsage(ctx, []Usage{
{At: at, KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 100, Status: 200},
{At: at.Add(2 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 10, Status: 200},
{At: at.Add(10 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 1, Status: 200},
{At: at.Add(70 * time.Minute), KeyID: "k", ProjectID: "p", Model: "m", CostMicros: 1000, Status: 200},
}); err != nil {
t.Fatal(err)
}
spend, err := s.SpendSince(ctx, at.Add(time.Minute))
if err != nil {
t.Fatal(err)
}
if spend["k"] != 1011 {
t.Errorf("spent %d since mid-slice, want 1011", spend["k"])
}
})
}
func TestPruningRequestsKeepsEveryTotal(t *testing.T) {
onBoth(t, func(t *testing.T, s *DB) {
ctx := context.Background()
at := slice()
if err := s.WriteUsage(ctx, []Usage{
{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m", ServedModel: "m",
CostMicros: 100, Status: 200, RequestID: "req-1"},
}); err != nil {
t.Fatal(err)
}
n, err := s.PruneUsage(ctx, time.Now())
if err != nil || n != 1 {
t.Fatalf("pruned %d, %v", n, err)
}
if _, ok, _ := s.UsageByRequest(ctx, "req-1"); ok {
t.Error("a pruned request can still be looked up")
}
spend, _ := s.SpendSince(ctx, time.UnixMilli(tier5m.floor(at)))
users, _ := s.UsageByUser(ctx, at.Add(-time.Hour))
if spend["k"] != 100 || len(users) != 1 || users[0].Requests != 1 {
t.Errorf("after pruning: spend %v, people %+v; the totals should not move", spend, users)
}
})
}
func TestTheMigrationRollsUpWhatIsAlreadyRecorded(t *testing.T) {
ctx := context.Background()
db, err := sql.Open("sqlite", filepath.Join(t.TempDir(), "old.db")+"?_pragma=foreign_keys(1)")
if err != nil {
t.Fatal(err)
}
defer db.Close()
db.SetMaxOpenConns(1)
if _, err := db.ExecContext(ctx,
`CREATE TABLE schema_migrations (version TEXT PRIMARY KEY, applied_at INTEGER NOT NULL)`); err != nil {
t.Fatal(err)
}
names, _ := migrationNames("sqlite")
for _, name := range names {
if name >= "0015" {
break
}
body, _ := migrationFS.ReadFile("migrations/sqlite/" + name)
if err := applyOne(ctx, db, "sqlite", name, string(body)); err != nil {
t.Fatalf("%s: %v", name, err)
}
}
for _, ts := range []int64{600_001, 600_002, 900_001} {
if _, err := db.ExecContext(ctx, `INSERT INTO usage (ts, key_id, project_id, model, served_model,
prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id, ttft_ms)
VALUES (?, 'k', 'p', 'm', 'm', 1, 1, 5, 200, 700, 'u', 0)`, ts); err != nil {
t.Fatal(err)
}
}
if err := migrate(ctx, db, "sqlite"); err != nil {
t.Fatal(err)
}
// 600,001 and 600,002 share a five-minute slice and 900,001 starts the
// next; all three are in the first hour, and the first day.
for table, want := range map[string]int64{"usage_5m": 2, "usage_1h": 1, "usage_1d": 1} {
var slices, reqs, cost, lat3, ttftN int64
if err := db.QueryRowContext(ctx, `SELECT COUNT(*), SUM(requests), SUM(cost_micros), SUM(lat_3), SUM(ttft_n)
FROM `+table).Scan(&slices, &reqs, &cost, &lat3, &ttftN); err != nil {
t.Fatal(err)
}
if slices != want || reqs != 3 || cost != 15 || lat3 != 3 || ttftN != 0 {
t.Errorf("%s: %d slices, %d requests, %d spent, %d in <=1000ms, %d first tokens; want %d, 3, 15, 3, 0",
table, slices, reqs, cost, lat3, ttftN, want)
}
}
}
func TestATotalReadsEachTierForItsOwnStretchOnly(t *testing.T) {
// A total since a moment two days back, a few minutes into a slice, is
// read from the rows, then five-minute slices, then hours, then days.
// One request lands in each stretch, and one just before the moment;
// counting any of them twice, or the early one at all, shows here.
onBoth(t, func(t *testing.T, s *DB) {
ctx := context.Background()
day := time.UnixMilli(tier1d.floor(time.Now().Add(-48 * time.Hour)))
since := day.Add(22*time.Hour + 52*time.Minute + 30*time.Second)
var batch []Usage
for i, at := range []time.Time{
since.Add(-time.Second), // before: not counted
since.Add(time.Minute), // the rows
since.Add(5 * time.Minute), // five-minute slices
since.Add(20 * time.Minute), // hours
since.Add(2 * time.Hour), // days
} {
batch = append(batch, Usage{At: at, KeyID: "k", UserID: "u", ProjectID: "p", Model: "m",
ServedModel: "m", CostMicros: int64(1) << i, Status: 200})
}
if err := s.WriteUsage(ctx, batch); err != nil {
t.Fatal(err)
}
users, err := s.UsageByUser(ctx, since)
if err != nil {
t.Fatal(err)
}
spend, err := s.SpendSince(ctx, since)
if err != nil {
t.Fatal(err)
}
if len(users) != 1 || users[0].Requests != 4 || users[0].CostMicros != 2+4+8+16 || spend["k"] != 30 {
t.Errorf("people %+v, spend %d; want 4 requests and 30 spent", users, spend["k"])
}
})
}
+18 -8
View File
@@ -28,17 +28,18 @@ func (s *DB) UsageOfUser(
if userID == "" {
return nil, nil, fmt.Errorf("whose usage?")
}
src, args := usageSince(since)
places, err := queryAll(ctx, s.db, `
SELECT COALESCE(NULLIF(u.project_id, ''), ''),
COALESCE(o.name || ' / ' || t.name || ' / ' || p.name, ''),
COUNT(*), SUM(u.prompt_tokens + u.completion_tokens), SUM(u.cost_micros)
FROM usage u
SUM(u.requests), SUM(u.prompt_tokens + u.completion_tokens), SUM(u.cost_micros)
FROM `+src+` u
LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
LEFT JOIN teams t ON t.id = p.team_id
LEFT JOIN organizations o ON o.id = t.org_id
WHERE u.user_id = ? AND u.ts >= ?
WHERE u.user_id = ?
GROUP BY 1, 2
ORDER BY 4 DESC`, []any{userID, millis(since)},
ORDER BY 4 DESC`, append(args, userID),
func(rows *sql.Rows) (PlaceUsage, error) {
var pu PlaceUsage
return pu, rows.Scan(&pu.ProjectID, &pu.Path, &pu.Requests, &pu.Tokens, &pu.CostMicros)
@@ -51,12 +52,20 @@ func (s *DB) UsageOfUser(
width = 24 * time.Hour
}
ms := width.Milliseconds()
got, err := queryAll(ctx, s.db, `
q, from := `
SELECT (ts / ?) * ?, COUNT(*),
SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
FROM usage WHERE user_id = ? AND ts >= ?
GROUP BY 1 ORDER BY 1`, []any{ms, ms, userID, millis(since)},
GROUP BY 1 ORDER BY 1`, millis(since)
if t, ok := tierForSlices(width); ok {
q, from = `
SELECT (bucket / ?) * ?, SUM(requests), SUM(errors),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
FROM `+t.table+` WHERE user_id = ? AND bucket >= ?
GROUP BY 1 ORDER BY 1`, t.floor(since)
}
got, err := queryAll(ctx, s.db, q, []any{ms, ms, userID, from},
func(rows *sql.Rows) (Bucket, error) {
var b Bucket
var start int64
@@ -131,8 +140,9 @@ func (s *DB) SpendOfPool(ctx context.Context, l Limits, since time.Time) (int64,
return 0, nil
}
var total sql.NullInt64
src, srcArgs := usageSince(since)
err := s.db.QueryRowContext(ctx,
`SELECT SUM(u.cost_micros) FROM usage u WHERE `+where+` AND u.ts >= ?`,
append(args, millis(since))...).Scan(&total)
`SELECT SUM(u.cost_micros) FROM `+src+` u WHERE `+where,
append(srcArgs, args...)...).Scan(&total)
return total.Int64, err
}
+38 -30
View File
@@ -348,14 +348,14 @@ func (s *DB) applyDefaults(e *Effective) {
// the tree. Personal keys and the chat page's requests have no project and are
// kept: dropping them is what would reset a person's budget on every restart.
func (s *DB) SpendSinceByScope(ctx context.Context, since time.Time) ([]ScopeSpend, error) {
src, args := usageSince(since)
rows, err := s.db.QueryContext(ctx,
`SELECT u.key_id, u.user_id, COALESCE(p.id, ''), COALESCE(t.id, ''), COALESCE(t.org_id, ''),
SUM(u.cost_micros)
FROM usage u
FROM `+src+` u
LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
LEFT JOIN teams t ON t.id = p.team_id
WHERE u.ts >= ?
GROUP BY u.key_id, u.user_id, p.id, t.id, t.org_id`, millis(since))
GROUP BY u.key_id, u.user_id, p.id, t.id, t.org_id`, args...)
if err != nil {
return nil, err
}
@@ -439,14 +439,15 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
return err
}
}
return nil
return writeRollup(ctx, tx, records)
})
}
// SpendSince totals cost per key since a moment.
func (s *DB) SpendSince(ctx context.Context, since time.Time) (map[string]int64, error) {
src, args := usageSince(since)
rows, err := s.db.QueryContext(ctx,
`SELECT key_id, SUM(cost_micros) FROM usage WHERE ts >= ? GROUP BY key_id`, millis(since))
`SELECT u.key_id, SUM(u.cost_micros) FROM `+src+` u GROUP BY u.key_id`, args...)
if err != nil {
return nil, err
}
@@ -620,16 +621,15 @@ var _ Store = (*DB)(nil)
// to show: a row identified only by an opaque id tells an operator nothing.
// Revoked keys still appear, because their usage still happened.
func (s *DB) UsageSince(ctx context.Context, since time.Time) ([]KeyUsage, error) {
src, args := usageSince(since)
return queryAll(ctx, s.db, `
SELECT u.key_id,
COALESCE(k.name, ''), COALESCE(k.key_prefix, ''), u.project_id,
COALESCE(us.email, ''),
COALESCE(o.name || ' / ' || t.name || ' / ' || p.name, ''),
COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros),
SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
MAX(u.ts)
FROM usage u
SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts)
FROM `+src+` u
LEFT JOIN api_keys k ON k.id = u.key_id
-- Who, as recorded with the spend; the owner of the key for rows
-- written before the person was.
@@ -637,12 +637,11 @@ func (s *DB) UsageSince(ctx context.Context, since time.Time) ([]KeyUsage, error
LEFT JOIN projects p ON p.id = NULLIF(u.project_id, '')
LEFT JOIN teams t ON t.id = p.team_id
LEFT JOIN organizations o ON o.id = t.org_id
WHERE u.ts >= ?
-- Everything selected beside the sums is grouped, which PostgreSQL
-- wants; each is determined by the key and project already there.
GROUP BY u.key_id, u.project_id, k.name, k.key_prefix, us.email, o.name, t.name, p.name
ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`,
[]any{millis(since)}, func(rows *sql.Rows) (KeyUsage, error) {
ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`,
args, func(rows *sql.Rows) (KeyUsage, error) {
var k KeyUsage
var last int64
err := rows.Scan(&k.KeyID, &k.KeyName, &k.KeyPrefix, &k.ProjectID,
@@ -664,7 +663,9 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
}
ms := width.Milliseconds()
rows, err := s.db.QueryContext(ctx, `
// From the rollup when the slices are whole rollup slices, which every
// page asks for; from the rows otherwise.
q, from := `
SELECT (ts / ?) * ?, COUNT(*),
SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
@@ -674,7 +675,16 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
-- selected one syntactically: "ts / ?" and "(ts / ?) * ?" are not the
-- same text, so it saw a bare ts and refused. The ordinal is the same
-- expression by construction, and both dialects accept it.
GROUP BY 1 ORDER BY 1`, ms, ms, millis(since))
GROUP BY 1 ORDER BY 1`, millis(since)
if t, ok := tierForSlices(width); ok {
q, from = `
SELECT (bucket / ?) * ?, SUM(requests), SUM(errors),
SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
1.0 * SUM(latency_ms_sum) / SUM(requests)
FROM `+t.table+` WHERE bucket >= ?
GROUP BY 1 ORDER BY 1`, t.floor(since)
}
rows, err := s.db.QueryContext(ctx, q, ms, ms, from)
if err != nil {
return nil, err
}
@@ -684,13 +694,13 @@ func (s *DB) UsageBuckets(ctx context.Context, since time.Time, width time.Durat
for rows.Next() {
var start int64
var b Bucket
var latency float64
var latency sql.NullFloat64
if err := rows.Scan(&start, &b.Requests, &b.Errors,
&b.Prompt, &b.Completion, &b.CostMicros, &latency); err != nil {
return nil, err
}
b.Start = fromMillis(start)
b.LatencyMs = int64(latency)
b.LatencyMs = int64(latency.Float64)
found[start] = b
}
if err := rows.Err(); err != nil {
@@ -736,23 +746,21 @@ func (u UserUsage) Tokens() int64 { return u.Prompt + u.Completion }
// gets skipped. Keys with no owner — issued by an operator before accounts
// existed — are grouped under an empty user, which is honest rather than tidy.
func (s *DB) UsageByUser(ctx context.Context, since time.Time) ([]UserUsage, error) {
src, args := usageSince(since)
rows, err := s.db.QueryContext(ctx, `
SELECT u.user_id, COALESCE(us.email, ''), COALESCE(us.name, ''),
COUNT(DISTINCT u.key_id),
COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros),
SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
MAX(u.ts)
FROM usage u
SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts)
FROM `+src+` u
LEFT JOIN users us ON us.id = NULLIF(u.user_id, '')
WHERE u.ts >= ?
-- Every selected column that is not an aggregate has to be grouped.
-- SQLite picks an arbitrary row for the rest and carries on; PostgreSQL
-- refuses, so this query worked in tests and returned nothing but an
-- error on the cluster. The email and name are functionally dependent
-- on the user id, so grouping by them changes no result.
GROUP BY u.user_id, COALESCE(us.email, ''), COALESCE(us.name, '')
ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`, millis(since))
ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`, args...)
if err != nil {
return nil, err
}
@@ -793,14 +801,14 @@ func (m ModelUsage) Tokens() int64 { return m.Prompt + m.Completion }
// cluster. The requested name is kept in the same table for anyone who needs to
// ask the other question.
func (s *DB) UsageByModel(ctx context.Context, since time.Time) ([]ModelUsage, error) {
src, args := usageSince(since)
rows, err := s.db.QueryContext(ctx, `
SELECT CASE WHEN served_model != '' THEN served_model ELSE model END,
COUNT(*), SUM(prompt_tokens), SUM(completion_tokens),
SUM(cost_micros),
SUM(CASE WHEN status >= 400 THEN 1 ELSE 0 END),
AVG(latency_ms)
FROM usage WHERE ts >= ?
GROUP BY 1 ORDER BY COUNT(*) DESC`, millis(since))
SUM(requests), SUM(prompt_tokens), SUM(completion_tokens),
SUM(cost_micros), SUM(errors),
1.0 * SUM(latency_ms_sum) / SUM(requests)
FROM `+src+` u
GROUP BY 1 ORDER BY SUM(requests) DESC`, args...)
if err != nil {
return nil, err
}
+6 -8
View File
@@ -80,21 +80,19 @@ func (s *DB) UsageByScope(ctx context.Context, scope Scope, since time.Time) ([]
// same expression it is grouped on. PostgreSQL requires both; SQLite
// silently picks a row for whatever is missing, which is how three of these
// queries shipped working in tests and failing on the cluster.
src, args := usageSince(since)
q := `
SELECT ` + c.id + `, ` + c.name + `,
COUNT(DISTINCT u.key_id),
COUNT(*), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros),
SUM(CASE WHEN u.status >= 400 THEN 1 ELSE 0 END),
MAX(u.ts),
SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros), SUM(u.errors), MAX(u.last_ts),
MAX(l.budget_micros), MAX(COALESCE(l.budget_period, ''))
FROM usage u` + c.joins + `
FROM ` + src + ` u` + c.joins + `
LEFT JOIN limits l ON l.scope_type = ? AND l.scope_id = ` + c.id + `
WHERE u.ts >= ?
GROUP BY ` + c.id + `, ` + c.name + `
ORDER BY SUM(u.cost_micros) DESC, COUNT(*) DESC`
ORDER BY SUM(u.cost_micros) DESC, SUM(u.requests) DESC`
rows, err := s.db.QueryContext(ctx, q, string(scope), millis(since))
rows, err := s.db.QueryContext(ctx, q, append(args, string(scope))...)
if err != nil {
return nil, err
}
+24 -2
View File
@@ -21,6 +21,9 @@ const (
GroupByTeam Grouping = "team"
GroupByProject Grouping = "project"
GroupByUser Grouping = "user"
// GroupByModel is by the model that answered, which differs from the one
// asked for when a fallback served it.
GroupByModel Grouping = "model"
)
// Valid reports whether this is a grouping usage can be read by.
@@ -32,7 +35,7 @@ func (g Grouping) Valid() bool {
// Groupings lists them in the order a page should offer them: widest first,
// then the one that is not a level at all.
func Groupings() []Grouping {
return []Grouping{GroupByOrg, GroupByTeam, GroupByProject, GroupByUser}
return []Grouping{GroupByOrg, GroupByTeam, GroupByProject, GroupByUser, GroupByModel}
}
// UsageSeries is one group's usage over time.
@@ -87,8 +90,13 @@ var groupColumns = map[Grouping]struct{ id, name, joins string }{
// rather than hides.
GroupByUser: {"u.user_id", "COALESCE(us.email, '')", `
LEFT JOIN users us ON us.id = NULLIF(u.user_id, '')`},
GroupByModel: {servedModel, servedModel, ""},
}
// servedModel is what answered: the served model, or the one asked for on a
// row from before the two were told apart.
const servedModel = "CASE WHEN u.served_model != '' THEN u.served_model ELSE u.model END"
// UsageBucketsBy returns each group's usage over time.
//
// One query for every group rather than one per group: the page shows them
@@ -145,8 +153,22 @@ func (s *DB) usageBuckets(
WHERE u.ts >= ?` + inOrg + `
GROUP BY 1, 2, 3
ORDER BY 1, 3`
from := millis(since)
// From the rollup for slices that are whole rollup slices — every
// window the pages offer — and from the rows for anything finer.
if t, ok := tierForSlices(width); ok {
q = `
SELECT ` + c.id + `, ` + c.name + `, (u.bucket / ?) * ?,
SUM(u.requests), SUM(u.prompt_tokens), SUM(u.completion_tokens),
SUM(u.cost_micros), SUM(u.errors)
FROM ` + t.table + ` u` + c.joins + `
WHERE u.bucket >= ?` + inOrg + `
GROUP BY 1, 2, 3
ORDER BY 1, 3`
from = t.floor(since)
}
args := []any{ms, ms, millis(since)}
args := []any{ms, ms, from}
if orgID != "" {
args = append(args, orgID)
}
+13 -4
View File
@@ -66,10 +66,14 @@ type peopleDetail struct {
BudgetsBind bool
// Its use over the last week, and the totals beside it.
HasUsage bool
Requests, Tokens, Spend TimeSeries
Totals usageRow
Priced bool
HasUsage bool
Requests, Tokens, Spend TimeSeries
Totals usageRow
Priced bool
// Stats is how its requests ended and how long they took, over the
// same week; HasStats is false where the store does not say.
Stats store.GroupStats
HasStats bool
UsageWindow, QuotaWindow string
CSRF string
@@ -248,6 +252,11 @@ func (p *AccountPages) usageDetail(ctx context.Context, d *peopleDetail) {
}
d.HasUsage = true
d.Priced = p.usagePriced != nil && p.usagePriced.AnythingPriced()
if sr, ok := p.usageStore.(StatsReader); ok {
if stats, err := sr.StatsBy(ctx, g, time.Now().Add(-win.Span)); err == nil {
d.Stats, d.HasStats = stats[d.ID], true
}
}
for _, s := range rows {
if s.ID == d.ID {
var table []usageRow
+24
View File
@@ -85,3 +85,27 @@ func TestSomethingNotInTheTreeOpensNoPanel(t *testing.T) {
t.Error("a panel opened for an organization that does not exist")
}
}
func TestHowRequestsFaredIsShownBesideWhatTheyCost(t *testing.T) {
stats := map[string]store.GroupStats{"org-1": {ID: "org-1", Requests: 200, Errors: 30, Limited: 25,
TTFT: store.Quantiles{P50: 180, P95: 900, P99: 2400, N: 150},
Latency: store.Quantiles{P50: 3000, P95: 8000, P99: 120000, N: 200, Over: true}}}
u := &fakeScopeSeries{stats: stats, byGroup: map[store.Grouping][]store.UsageSeries{
store.GroupByOrg: {{Grouping: store.GroupByOrg, ID: "org-1", Name: "Scania", Requests: 200,
Buckets: []store.Bucket{{Start: time.Now().Add(-time.Hour), Requests: 200}}}},
}}
mux := peoplePage(t, oneTeam(), u)
table := strings.Join(strings.Fields(mirrorGet(t, mux, "/admin/usage?by=org").Body.String()), " ")
for _, want := range []string{"15.0%", ">25<", "180 ms / 900 ms", "8.0 s"} {
if !strings.Contains(table, want) {
t.Errorf("the usage table lacks %q", want)
}
}
panel := strings.Join(strings.Fields(mirrorGet(t, mux, "/admin/users?sel=org:org-1").Body.String()), " ")
for _, want := range []string{"<dt>Refused by a limit</dt>", "p95 900 ms", "p99 &gt; 120 s"} {
if !strings.Contains(panel, want) {
t.Errorf("the panel lacks %q", want)
}
}
}
+50
View File
@@ -2,6 +2,7 @@ package web
import (
"context"
"fmt"
"net/http"
"sort"
"time"
@@ -16,6 +17,35 @@ type UsageAdmin interface {
since time.Time, width time.Duration) ([]store.UsageSeries, error)
}
// StatsReader is how each group fared and how long its requests took; the
// store implements it. Optional: without it the usage table has no such
// columns.
type StatsReader interface {
StatsBy(ctx context.Context, g store.Grouping, since time.Time) (map[string]store.GroupStats, error)
}
// pctile is a percentile from the usage histograms as a person reads it:
// "180 ms", "4.2 s", "> 120 s" past the last bucket, "—" with nothing to read.
func pctile(q store.Quantiles, which int) string {
if q.N == 0 {
return "—"
}
v := q.P50
switch which {
case 95:
v = q.P95
case 99:
v = q.P99
if q.Over {
return fmt.Sprintf("> %d s", v/1000)
}
}
if v < 1000 {
return fmt.Sprintf("%d ms", v)
}
return fmt.Sprintf("%.1f s", float64(v)/1000)
}
// usageWindow is one length of history the page offers, and the slice it is cut
// into.
//
@@ -54,11 +84,16 @@ type usageGroup struct {
// usageRow is one group's totals, shown under the charts.
type usageRow struct {
ID string
Name string
Requests int64
Tokens int64
Spend string
Errors int64
// Stats is how its requests ended and how long they took; HasStats is
// false where the store does not say.
Stats store.GroupStats
HasStats bool
}
type adminUsageData struct {
@@ -86,6 +121,8 @@ type adminUsageData struct {
// model on a GPU.
GPUHeld TimeSeries
GPUHours []gpuHoursRow
// HasStats adds outcomes and latency percentiles to the table.
HasStats bool
Error string
Version string
@@ -141,6 +178,16 @@ func (p *AccountPages) buildUsage(r *http.Request, s UsageAdmin, priced Priced)
}
d.Requests, d.Tokens, d.Spend, d.Rows = usageCharts(rows)
if sr, ok := s.(StatsReader); ok {
stats, err := sr.StatsBy(r.Context(), by, time.Now().Add(-win.Span))
if err != nil {
p.log.Warn("could not read latency and outcomes for the usage tab", "by", by, "err", err)
}
for i := range d.Rows {
d.Rows[i].Stats, d.Rows[i].HasStats = stats[d.Rows[i].ID], stats != nil
}
d.HasStats = stats != nil
}
d.GPUHeld, d.GPUHours = p.gpuHours(r.Context(), win)
return d
}
@@ -226,6 +273,8 @@ func groupLabel(g store.Grouping) string {
return "project"
case store.GroupByUser:
return "person"
case store.GroupByModel:
return "model"
default:
return string(g)
}
@@ -246,6 +295,7 @@ func usageCharts(rows []store.UsageSeries) (req, tok, spend TimeSeries, table []
}), "", 4)
for _, u := range rows {
table = append(table, usageRow{
ID: u.ID,
Name: displayName(u),
Requests: u.Requests,
Tokens: u.Tokens(),
+9 -3
View File
@@ -15,9 +15,11 @@ import (
type fakeScopeSeries struct {
byGroup map[store.Grouping][]store.UsageSeries
asked []store.Grouping
width time.Duration
since time.Time
// stats, when set, is what StatsBy answers for every grouping.
stats map[string]store.GroupStats
asked []store.Grouping
width time.Duration
since time.Time
}
func (f *fakeScopeSeries) UsageBucketsBy(_ context.Context, g store.Grouping,
@@ -28,6 +30,10 @@ func (f *fakeScopeSeries) UsageBucketsBy(_ context.Context, g store.Grouping,
return f.byGroup[g], nil
}
func (f *fakeScopeSeries) StatsBy(context.Context, store.Grouping, time.Time) (map[string]store.GroupStats, error) {
return f.stats, nil
}
func usagePage(t *testing.T, s *fakeScopeSeries, priced Priced) *http.ServeMux {
t.Helper()
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
+7
View File
@@ -140,6 +140,13 @@
<h3>Use, {{.UsageWindow}}</h3>
{{if .Totals.Name}}
<p class="pd-sub">{{compact .Totals.Requests}} requests · {{compact .Totals.Tokens}} tokens{{if .Priced}} · {{.Totals.Spend}} spent{{end}}{{if .Totals.Errors}} · {{.Totals.Errors}} errors{{end}}</p>
{{if and .HasStats .Stats.Requests}}{{with .Stats}}
<dl class="pd-limits">
<dt>Error rate</dt><dd{{if ge .ErrorRate 5.0}} class="q-warn"{{end}}>{{pct1 .ErrorRate}}%</dd>
<dt>Refused by a limit</dt><dd{{if .Limited}} class="q-warn"{{end}}>{{if .Limited}}{{compact .Limited}}{{else}}none{{end}}</dd>
<dt>First token</dt><dd>p50 {{pctile .TTFT 50}} · p95 {{pctile .TTFT 95}} · p99 {{pctile .TTFT 99}}</dd>
<dt>Latency</dt><dd>p50 {{pctile .Latency 50}} · p95 {{pctile .Latency 95}} · p99 {{pctile .Latency 99}}</dd>
</dl>{{end}}{{end}}
{{template "tschart" dict "t" .Requests "title" "Requests" "note" ""}}
{{template "tschart" dict "t" .Tokens "title" "Tokens" "note" ""}}
{{if .Priced}}{{template "tschart" dict "t" .Spend "title" "Spend" "note" ""}}{{end}}
+10 -1
View File
@@ -40,7 +40,11 @@
{{if .Rows}}
<table class="rows">
<tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th></tr>
<tr><th>Group</th><th class="num">Requests</th><th class="num">Tokens</th><th class="num">Spend</th><th class="num">Errors</th>
{{if .HasStats}}<th class="num" title="Failed requests, refusals by a limit included">Error rate</th>
<th class="num" title="Refused by a rate, token or budget limit">Limited</th>
<th class="num" title="Time to first token, for streamed requests">First token p50 / p95</th>
<th class="num">Latency p95</th>{{end}}</tr>
{{range .Rows}}
<tr>
<td>{{.Name}}</td>
@@ -48,6 +52,11 @@
<td class="num">{{compact .Tokens}}</td>
<td class="num">{{.Spend}}</td>
<td class="num">{{if .Errors}}{{.Errors}}{{else}}—{{end}}</td>
{{if $.HasStats}}{{with .Stats}}
<td class="num{{if ge .ErrorRate 5.0}} q-warn{{end}}">{{if .Requests}}{{pct1 .ErrorRate}}%{{else}}—{{end}}</td>
<td class="num{{if .Limited}} q-warn{{end}}">{{if .Limited}}{{compact .Limited}}{{else}}—{{end}}</td>
<td class="num">{{pctile .TTFT 50}} / {{pctile .TTFT 95}}</td>
<td class="num">{{pctile .Latency 95}}</td>{{end}}{{end}}
</tr>
{{end}}
</table>
+3
View File
@@ -22,6 +22,7 @@ import (
"io/fs"
"net/http"
"slices"
"strconv"
"strings"
"sync"
"time"
@@ -90,6 +91,8 @@ var funcs = template.FuncMap{
return strings.Join(v, ", ")
},
"latency": humanLatency,
"pctile": pctile,
"pct1": func(v float64) string { return strconv.FormatFloat(v, 'f', 1, 64) },
"duration": func(sec int64) string {
return time.Duration(sec * int64(time.Second)).Round(time.Second).String()
},