Ilker ManapandClaude Opus 5
c9c841387a
CI / test (push) Successful in 1m40s
The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
2.8 KiB
Go
70 lines
2.8 KiB
Go
package router
|
|
|
|
import (
|
|
"errors"
|
|
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
|
)
|
|
|
|
// budgetFallback finds a model this caller can still afford.
|
|
//
|
|
// A spent budget was a dead end: the check refused the request and that was
|
|
// the end of it. The accounting layer has always exempted a model that costs
|
|
// nothing, with a comment saying it is "what lets a key with no budget left
|
|
// still be served a cheap fallback" — describing a caller that was never
|
|
// written, so the exemption could not fire.
|
|
//
|
|
// It matters most for the audience this is built for. A team trying models
|
|
// will exhaust a quota, and being dropped to a smaller model is a far better
|
|
// answer than a refusal: it makes a budget a guardrail rather than a wall.
|
|
//
|
|
// Silent, with the served model in a header, which is what capacity fallback
|
|
// already does. Two rules for the same situation would be worse than one
|
|
// imperfect rule, and a client that cares can read the header.
|
|
func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accounting.Reservation, bool) {
|
|
_, entry := r.resolve(req.Model)
|
|
for _, name := range entry.Fallbacks {
|
|
// The allowlist still applies. A key that may not use a model does not
|
|
// gain access to it by running out of money, which would make a budget
|
|
// a way around an allowlist rather than a limit on top of one.
|
|
if !r.permits(p, name) {
|
|
continue
|
|
}
|
|
// And it has to be able to hold the prompt. The fit was checked against
|
|
// the model the caller asked for, before anyone knew a different one
|
|
// would serve — so without this a 20k prompt sent to a 32k model lands
|
|
// on a 2k one, and the engine truncates it or refuses. Either way the
|
|
// caller sees a failure naming a model they never chose.
|
|
//
|
|
// The history is not trimmed to make it fit. Dropping a caller's
|
|
// messages to stay under a limit they did not ask about is a worse
|
|
// answer than saying the prompt is too large, and it would be done
|
|
// silently, to a request that was already being quietly moved.
|
|
_, cand := r.resolve(name)
|
|
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
|
|
continue
|
|
}
|
|
res, err := r.acct.Check(p, name, req.estimatedTokens())
|
|
if err != nil {
|
|
continue
|
|
}
|
|
req.Model = name
|
|
return res, true
|
|
}
|
|
return accounting.Reservation{}, false
|
|
}
|
|
|
|
// isBudgetSpent reports whether a limit error is the one a fallback can answer.
|
|
//
|
|
// Only a spent budget. A rate limit is answered by waiting a second, and moving
|
|
// the request to another model would spend someone else's capacity to avoid a
|
|
// delay they did not ask to avoid.
|
|
func isBudgetSpent(err error) bool {
|
|
var le *accounting.LimitError
|
|
if !errors.As(err, &le) {
|
|
return false
|
|
}
|
|
return errors.Is(le.Err, accounting.ErrBudgetSpent)
|
|
}
|