The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
56 lines
2.0 KiB
Go
56 lines
2.0 KiB
Go
package router
|
|
|
|
import (
|
|
"errors"
|
|
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
|
)
|
|
|
|
// budgetFallback finds a model this caller can still afford.
|
|
//
|
|
// A spent budget was a dead end: the check refused the request and that was
|
|
// the end of it. The accounting layer has always exempted a model that costs
|
|
// nothing, with a comment saying it is "what lets a key with no budget left
|
|
// still be served a cheap fallback" — describing a caller that was never
|
|
// written, so the exemption could not fire.
|
|
//
|
|
// It matters most for the audience this is built for. A team trying models
|
|
// will exhaust a quota, and being dropped to a smaller model is a far better
|
|
// answer than a refusal: it makes a budget a guardrail rather than a wall.
|
|
//
|
|
// Silent, with the served model in a header, which is what capacity fallback
|
|
// already does. Two rules for the same situation would be worse than one
|
|
// imperfect rule, and a client that cares can read the header.
|
|
func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accounting.Reservation, bool) {
|
|
_, entry := r.resolve(req.Model)
|
|
for _, name := range entry.Fallbacks {
|
|
// The allowlist still applies. A key that may not use a model does not
|
|
// gain access to it by running out of money, which would make a budget
|
|
// a way around an allowlist rather than a limit on top of one.
|
|
if !r.permits(p, name) {
|
|
continue
|
|
}
|
|
res, err := r.acct.Check(p, name, req.estimatedTokens())
|
|
if err != nil {
|
|
continue
|
|
}
|
|
req.Model = name
|
|
return res, true
|
|
}
|
|
return accounting.Reservation{}, false
|
|
}
|
|
|
|
// isBudgetSpent reports whether a limit error is the one a fallback can answer.
|
|
//
|
|
// Only a spent budget. A rate limit is answered by waiting a second, and moving
|
|
// the request to another model would spend someone else's capacity to avoid a
|
|
// delay they did not ask to avoid.
|
|
func isBudgetSpent(err error) bool {
|
|
var le *accounting.LimitError
|
|
if !errors.As(err, &le) {
|
|
return false
|
|
}
|
|
return errors.Is(le.Err, accounting.ErrBudgetSpent)
|
|
}
|