CI / test (push) Successful in 1m7s
The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
71 lines
2.2 KiB
Go
71 lines
2.2 KiB
Go
// Package accounting turns finished requests into spend, enforces what a key
|
|
// is allowed, and gets both into the store without touching the request path
|
|
// (plan §7.9.3, §7.9.4).
|
|
package accounting
|
|
|
|
import (
|
|
"strings"
|
|
"sync"
|
|
)
|
|
|
|
// Price is what one model costs, in micro-units per thousand tokens.
|
|
//
|
|
// Micro-units rather than a float because spend is accumulated by repeated
|
|
// addition, and repeated addition of floats drifts. What a unit means is the
|
|
// operator's: self-hosted models have no per-token price, so it may be
|
|
// GPU-seconds, a notional rate, or 1000 everywhere — which turns a budget into
|
|
// a token quota.
|
|
type Price struct {
|
|
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
|
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
|
}
|
|
|
|
// Zero reports whether this model spends nothing.
|
|
//
|
|
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
|
// whose budget is spent, which is what makes the plan's budget fallback work
|
|
// (§7.7).
|
|
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
|
|
|
|
// Cost computes what a request spent, rounding up.
|
|
//
|
|
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
|
// free, which is exactly the shape of traffic a quota exists to bound.
|
|
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
|
|
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
|
|
}
|
|
|
|
func divUp(n, d int64) int64 {
|
|
if n <= 0 {
|
|
return 0
|
|
}
|
|
return (n + d - 1) / d
|
|
}
|
|
|
|
// Pricing resolves a model name to its price.
|
|
type Pricing struct {
|
|
mu sync.RWMutex
|
|
byModel map[string]Price
|
|
fallback Price
|
|
}
|
|
|
|
// NewPricing builds a price table. Names are matched case-insensitively,
|
|
// because a model name reaches us from whatever the client typed.
|
|
func NewPricing(byModel map[string]Price, fallback Price) *Pricing {
|
|
table := make(map[string]Price, len(byModel))
|
|
for name, p := range byModel {
|
|
table[strings.ToLower(name)] = p
|
|
}
|
|
return &Pricing{byModel: table, fallback: fallback}
|
|
}
|
|
|
|
// For returns a model's price, or the fallback.
|
|
func (p *Pricing) For(model string) Price {
|
|
p.mu.RLock()
|
|
defer p.mu.RUnlock()
|
|
if v, ok := p.byModel[strings.ToLower(model)]; ok {
|
|
return v
|
|
}
|
|
return p.fallback
|
|
}
|