Files
llmgw/internal/store/store.go
T
Ilker ManapandClaude Opus 5 f999ee2fb2
CI / test (push) Successful in 1m40s
Make an organization's budget bound the sum of its teams
Resolution took the most restrictive value along the chain and enforced only
that one. For a single caller's ceiling that is right: a team cannot raise what
its organization set. For a pool it is not, because two levels are two pools.

An organization given 1000 with four teams given 500 each spent 2000. Each team
filled its own counter, the organization's value lost to the tighter one and
was never looked at again. The number was written, displayed, reported against
— and enforced by nothing.

Every level that sets a limit is now checked against its own counter, and
settled onto all of them, so an organization's pool fills from its teams'
spending rather than only from requests that somehow named it. A team still
stops before its organization when its own pool runs out first, which is what a
sub-allocation is for.

Effective carries the chain as well as the collapsed values. The collapsed ones
are still what a header or a profile page should show — they are the ceiling a
single caller faces — they are just not enough to enforce with.

Rate limits charge as they check, deliberately, so that two concurrent requests
cannot both see room. With several levels that means a refusal at one has
already taken from the levels before it, so those are handed back. Without the
refund a team sitting at its organization's ceiling would quietly eat its own
rate limit with requests that were never served.

Budgets are checked before any of that, and only read, so a refusal on a budget
charges nothing at all.

Status reports the fullest pool rather than the smallest number. A team inside
an organization with the same budget is bounded by whichever is nearer its
ceiling, and that is the organization as soon as a second team spends anything;
a header showing the team's would disagree with the refusal after it.

A cluster default joins the chain as a key-level entry. Left out, it would be
reported as the ceiling and then never checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 15:08:30 +02:00

295 lines
9.8 KiB
Go

// Package store persists what the cluster cannot reconstruct: who may call the
// gateway, what they are allowed, and what they have used (plan §7.9).
//
// Everything else llmgw knows is derived from the fleet and is deliberately not
// stored — a database that describes the cluster can be wrong, and the cluster
// cannot (§6.5). This package holds only the facts that have no other source.
//
// The interface exists so the engine can change. SQLite is right for a single
// instance: it is embedded, so the store shares the gateway's fate instead of
// adding a second thing that can be down, and an inference gateway that stops
// serving because its accounting database is unreachable is worse than one that
// buffers some accounting. That argument stops holding the moment there are two
// gateway instances, because budgets and rate limits then have to be counted
// somewhere shared; Postgres arrives with that, not before (plan §7.9.6).
package store
import (
"context"
"errors"
"time"
)
// ErrNotFound is returned when a lookup finds nothing.
var ErrNotFound = errors.New("not found")
// Scope is a level of the tenancy tree. Limits hang off any of them.
type Scope string
// The tenancy levels, from widest to narrowest.
const (
ScopeOrg Scope = "org"
ScopeTeam Scope = "team"
ScopeProject Scope = "project"
ScopeKey Scope = "key"
)
// Org, Team and Project are the tenancy tree.
type Org struct {
ID, Name string
}
// Team belongs to an organization.
type Team struct {
ID, OrgID, Name string
}
// Project belongs to a team and owns keys.
type Project struct {
ID, TeamID, Name string
}
// Key is an API key as stored: never the key itself.
type Key struct {
ID string
// Hash is sha256 of the key, and the only copy of anything derived from it.
Hash []byte
// Prefix is shown in listings so a key can be identified without revealing
// it. It is never used for authentication.
Prefix string
Name string
ProjectID string
// Band and RoutingProfile are empty when the key inherits them.
Band string
RoutingProfile string
CreatedAt time.Time
// ExpiresAt and RevokedAt are zero when unset. Revocation is a soft delete:
// usage rows reference the key, and an audit trail that loses its subject
// is not an audit trail.
ExpiresAt time.Time
RevokedAt time.Time
LastUsedAt time.Time
}
// Usable reports whether a key may authenticate at this instant.
func (k Key) Usable(now time.Time) bool {
if !k.RevokedAt.IsZero() {
return false
}
return k.ExpiresAt.IsZero() || now.Before(k.ExpiresAt)
}
// Limits are the caps at one scope. A nil field means "no limit here", which is
// different from zero: zero forbids everything.
type Limits struct {
Scope Scope
ScopeID string
RPM *int
TPM *int
// Budget is spend per period, in micro-units — integers rather than floats
// because a budget is accumulated by repeated addition, and repeated
// addition of floats drifts. What a unit means is the operator's choice;
// self-hosted models have no per-token price (plan §7.9.4).
Budget *int64
BudgetPeriod Period
// AllowedModels nil means every model.
AllowedModels []string
}
// Period is how often a budget resets.
type Period string
// The supported budget periods.
const (
PeriodNone Period = ""
PeriodDay Period = "day"
PeriodMonth Period = "month"
)
// Effective is a key's resolved limits, one value per field.
//
// Resolution takes the most restrictive value along key → project → team → org.
// A team cannot raise a limit its organization set, which is the only
// composition rule under which a budget means anything.
type Effective struct {
KeyID string
ProjectID string
TeamID string
OrgID string
Band string
RoutingProfile string
RPM *int
TPM *int
Budget *int64
BudgetPeriod Period
// AllowedModels is the intersection along the chain. nil means every model.
AllowedModels []string
// RPMOwner, TPMOwner and BudgetOwner say which scope set the value that
// won, so consumption can be counted against that scope rather than
// against each key under it.
//
// Without this the tree hands down values but never pools spending: a
// project given a budget of 1000 and ten keys permits ten thousand, because
// each key gets its own counter holding the project's number. That is "each
// member may spend 1000", which is a useful thing to be able to say and is
// not what somebody writing a budget on a project means.
//
// Each limit is owned separately because they resolve separately: a budget
// can come from the team while the rate limit comes from the key.
RPMOwner LimitOwner
TPMOwner LimitOwner
BudgetOwner LimitOwner
// Chain is every level that set a limit, widest first.
//
// The collapsed values above are the ceiling a single caller faces, and
// they are what a header or a profile page should show. They are not enough
// to enforce with: once a quota is a pool, two levels are two pools and both
// have to hold. An organization given 1000 and four teams given 500 each
// would otherwise spend 2000 — the team's value wins as the more
// restrictive, and the organization's is never looked at again.
Chain []Limits
}
// LimitOwner is the scope a resolved limit came from.
type LimitOwner struct {
Scope Scope
ID string
}
// CounterKey is what a usage counter for this limit should be keyed by.
//
// Empty when no scope set the limit, which means nothing is being counted and
// the caller should not be asking. Scope and id together, because ids are only
// unique within a level and a project could otherwise share a counter with a
// team that happened to be named the same.
func (o LimitOwner) CounterKey() string {
if o.ID == "" {
return ""
}
return string(o.Scope) + ":" + o.ID
}
// Allows reports whether a model may be served to this key.
func (e Effective) Allows(model string) bool {
if e.AllowedModels == nil {
return true
}
for _, m := range e.AllowedModels {
if m == model {
return true
}
}
return false
}
// Usage is one request's accounting record.
type Usage struct {
At time.Time
KeyID string
// ProjectID is denormalized so reporting does not join on every row, and so
// a row stays meaningful if the key is later revoked.
ProjectID string
Model string
// ServedModel differs from Model when a fallback served the request.
ServedModel string
PromptTokens int64
CompletionTokens int64
// CostMicros is what this request spent against the budget.
CostMicros int64
Status int
LatencyMs int64
}
// Store is the persistent half of tenancy and accounting.
//
// Nothing here is called on the request path. Authentication reads a cache that
// LookupKey fills on a miss, and usage is written in batches (plan §7.9.3).
type Store interface {
// Tenancy. Put is an upsert so applying a configuration twice is a no-op.
PutOrg(ctx context.Context, o Org) error
PutTeam(ctx context.Context, t Team) error
PutProject(ctx context.Context, p Project) error
Orgs(ctx context.Context) ([]Org, error)
Teams(ctx context.Context, orgID string) ([]Team, error)
Projects(ctx context.Context, teamID string) ([]Project, error)
// Keys.
CreateKey(ctx context.Context, k Key) error
KeyByHash(ctx context.Context, hash []byte) (Key, error)
Keys(ctx context.Context, projectID string) ([]Key, error)
RevokeKey(ctx context.Context, id string, at time.Time) error
TouchKeys(ctx context.Context, seen map[string]time.Time) error
// Limits.
PutLimits(ctx context.Context, l Limits) error
LimitsFor(ctx context.Context, scope Scope, id string) (Limits, error)
// Resolve walks a key's chain and returns the most restrictive of each.
Resolve(ctx context.Context, keyID string) (Effective, error)
// Accounting.
WriteUsage(ctx context.Context, records []Usage) error
// SpendSince totals cost per key since a moment, so an in-memory budget
// counter can be rebuilt after a restart. A restart must not reset a
// budget, which it would if the counter lived only in memory.
SpendSince(ctx context.Context, since time.Time) (map[string]int64, error)
// SpendSinceByScope is the same totals with each key's place in the tree,
// so a budget owned by a team can be rebuilt by summing every key under it.
// Per key alone cannot do that: the counter is one number for the whole
// team, and the rows that made it up are spread across its projects.
SpendSinceByScope(ctx context.Context, since time.Time) ([]ScopeSpend, error)
// UsageSince summarizes activity per key. A reporting query: it scans and
// groups, so it belongs behind a dashboard refresh rather than a request.
UsageSince(ctx context.Context, since time.Time) ([]KeyUsage, error)
Close() error
}
// KeyUsage summarizes one key's activity in a period, for the dashboard and
// `ctl`. It is a reporting query and never runs on the request path.
type KeyUsage struct {
KeyID string
KeyName string
KeyPrefix string
ProjectID string
Requests int64
Prompt int64
Completion int64
CostMicros int64
Errors int64
LastAt time.Time
}
// Bucket is one time slice of usage, for the dashboard's charts.
type Bucket struct {
Start time.Time
Requests int64
Errors int64
Prompt int64
Completion int64
CostMicros int64
// LatencyMs is the mean over the bucket. A mean rather than a percentile
// because the raw rows are already gone by rollup time; this is a shape, not
// an SLO measurement.
LatencyMs int64
}
// ScopeSpend is one key's spend in a period, with where that key sits in the
// tenancy tree.
//
// Returned rather than pre-aggregated per scope because the roll-up depends on
// which scopes own budgets, which the store does not know and the accountant
// does. One query, and the folding happens where it can be tested.
type ScopeSpend struct {
KeyID string
ProjectID string
TeamID string
OrgID string
Micros int64
}