Files
llmgw/internal/router/budget_fallback.go
T
Ilker ManapandClaude Opus 5 28967edab3 Serve a spent budget something it can afford
The accounting layer has always exempted a model that costs nothing, with a
comment saying it is "what lets a key with no budget left still be served a
cheap fallback". That described a caller nobody had written: the budget was
checked once, against the requested model, and a refusal was the end of it,
so the exemption could never fire.

A caller that has run out is now offered the first model in its fallbacks it
can still afford. The allowlist still applies — running out of money is not
a way to reach a model a key was never allowed — and a model with no
fallback is refused exactly as before, or a budget would stop meaning
anything. Only a spent budget falls back: a rate limit is answered by
waiting a second, and moving the request elsewhere would spend someone
else's capacity to avoid a delay nobody asked to avoid.

Which uncovered a header that has never worked. x-llmgw-served-model was set
in finish, and finish runs after the proxy has written the response — so the
one mechanism for telling a client that something else answered has been
announcing it to nobody since it was written. Nothing failed, because the
only thing asserting it was a comment. It is now set before dispatch, for
the capacity fallback as well as this one.

Finding that took bisecting a test I had written wrong: it polled for a 429
that never came, because the budget had run out and the fallback had served
the request — the feature working, reported as the feature failing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 18:26:03 +02:00

56 lines
2.0 KiB
Go

package router
import (
"errors"
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
)
// budgetFallback finds a model this caller can still afford.
//
// A spent budget was a dead end: the check refused the request and that was
// the end of it. The accounting layer has always exempted a model that costs
// nothing, with a comment saying it is "what lets a key with no budget left
// still be served a cheap fallback" — describing a caller that was never
// written, so the exemption could not fire.
//
// It matters most for the audience this is built for. A team trying models
// will exhaust a quota, and being dropped to a smaller model is a far better
// answer than a refusal: it makes a budget a guardrail rather than a wall.
//
// Silent, with the served model in a header, which is what capacity fallback
// already does. Two rules for the same situation would be worse than one
// imperfect rule, and a client that cares can read the header.
func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accounting.Reservation, bool) {
_, entry := r.resolve(req.Model)
for _, name := range entry.Fallbacks {
// The allowlist still applies. A key that may not use a model does not
// gain access to it by running out of money, which would make a budget
// a way around an allowlist rather than a limit on top of one.
if !r.permits(p, name) {
continue
}
res, err := r.acct.Check(p, name, req.estimatedTokens())
if err != nil {
continue
}
req.Model = name
return res, true
}
return accounting.Reservation{}, false
}
// isBudgetSpent reports whether a limit error is the one a fallback can answer.
//
// Only a spent budget. A rate limit is answered by waiting a second, and moving
// the request to another model would spend someone else's capacity to avoid a
// delay they did not ask to avoid.
func isBudgetSpent(err error) bool {
var le *accounting.LimitError
if !errors.As(err, &le) {
return false
}
return errors.Is(le.Err, accounting.ErrBudgetSpent)
}