Files
llmgw/internal/accounting/cost.go
T
Ilker ManapandClaude Opus 5 0b93fb2883 Enforce budgets and rate limits, and write usage off the request path
The second half of phase 5. A key now has a cost, a quota and a bill.

Rate limits use a sliding window rather than a fixed one. A fixed window lets
a caller send its whole minute's allowance in the last second of one window
and again in the first second of the next, so the observed rate is twice the
configured limit exactly when the system is least able to take it. Checking
and charging are one operation, or two concurrent requests both see room and
both take it.

Token limits are checked against an estimate, because the real count only
exists once the request has run; the difference is charged afterwards, so a
caller who consistently underestimates is still bounded, one window late. A
request with no max_tokens gets a modest assumption rather than zero —
assuming zero would let an unbounded request through every token limit.

Limits are checked before admission, not after: a request over its quota
should be refused rather than queued, where it would occupy a band's capacity
to no purpose. All refusals are 429, but a spent budget carries no Retry-After
because waiting does not help until the period rolls over, and saying
otherwise turns one client into a retry loop.

Usage goes to a buffered channel and is written in batches. Record never
blocks: if the buffer fills — a database stalled long enough to lose eight
thousand records — records are dropped and counted, because losing accounting
is bad and turning a stalled database into an outage is worse. A clean
shutdown drains what is queued; those are requests that were already served.
Failures are logged at error, since silently losing billing data is the kind
of thing discovered at the end of a month.

Requests are accounted whatever their outcome. A request that used a GPU and
then failed still used it, and a key that could spend nothing by failing would
be a way around a budget.

Budgets are counted in memory and restored from the store at startup, so a
restart does not hand every key a fresh allowance. The budget period travels
with the reservation rather than being re-derived, because a monthly budget
credited to a daily counter is a budget that never binds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:52:36 +02:00

71 lines
2.2 KiB
Go

// Package accounting turns finished requests into spend, enforces what a key
// is allowed, and gets both into the store without touching the request path
// (plan §7.9.3, §7.9.4).
package accounting
import (
"strings"
"sync"
)
// Price is what one model costs, in micro-units per thousand tokens.
//
// Micro-units rather than a float because spend is accumulated by repeated
// addition, and repeated addition of floats drifts. What a unit means is the
// operator's: self-hosted models have no per-token price, so it may be
// GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into
// a token quota.
type Price struct {
PromptPer1k int64 `koanf:"prompt_per_1k"`
CompletionPer1k int64 `koanf:"completion_per_1k"`
}
// Zero reports whether this model spends nothing.
//
// It matters beyond arithmetic: a zero-cost model stays available to a key
// whose budget is spent, which is what makes the plan's budget fallback work
// (§7.7).
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
// Cost computes what a request spent, rounding up.
//
// Up rather than to nearest: rounding down would make a stream of tiny requests
// free, which is exactly the shape of traffic a quota exists to bound.
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
}
func divUp(n, d int64) int64 {
if n <= 0 {
return 0
}
return (n + d - 1) / d
}
// Pricing resolves a model name to its price.
type Pricing struct {
mu sync.RWMutex
byModel map[string]Price
fallback Price
}
// NewPricing builds a price table. Names are matched case-insensitively,
// because a model name reaches us from whatever the client typed.
func NewPricing(byModel map[string]Price, fallback Price) *Pricing {
table := make(map[string]Price, len(byModel))
for name, p := range byModel {
table[strings.ToLower(name)] = p
}
return &Pricing{byModel: table, fallback: fallback}
}
// For returns a model's price, or the fallback.
func (p *Pricing) For(model string) Price {
p.mu.RLock()
defer p.mu.RUnlock()
if v, ok := p.byModel[strings.ToLower(model)]; ok {
return v
}
return p.fallback
}