The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
1b7e11b91e
commit
c9c841387a
2 files changed
+221
No files matched your search
@@ -31,6 +31,20 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
|
||||
if !r.permits(p, name) {
|
||||
continue
|
||||
}
|
||||
// And it has to be able to hold the prompt. The fit was checked against
|
||||
// the model the caller asked for, before anyone knew a different one
|
||||
// would serve — so without this a 20k prompt sent to a 32k model lands
|
||||
// on a 2k one, and the engine truncates it or refuses. Either way the
|
||||
// caller sees a failure naming a model they never chose.
|
||||
//
|
||||
// The history is not trimmed to make it fit. Dropping a caller's
|
||||
// messages to stay under a limit they did not ask about is a worse
|
||||
// answer than saying the prompt is too large, and it would be done
|
||||
// silently, to a request that was already being quietly moved.
|
||||
_, cand := r.resolve(name)
|
||||
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
|
||||
continue
|
||||
}
|
||||
res, err := r.acct.Check(p, name, req.estimatedTokens())
|
||||
if err != nil {
|
||||
continue
|
||||
|
||||
Reference in new issue
Block a user