Check the prompt fits the model that will actually serve it
CI / test (push) Successful in 1m40s

The context check ran against the model the caller asked for. The budget
fallback then swapped in a different one and nothing looked again, so a prompt
sized for a 32k model could be sent to a 512-token one. Measured before the
fix: 10002 prompt tokens delivered to a model declaring 512.

The caller sees that as a truncated reply or an engine error naming a model
they never chose, which is a worse outcome than the refusal the fallback
exists to avoid — it fails later and explains less.

Candidates that cannot hold the prompt are now skipped, and a fallback further
down the list that can hold it is used instead. When none fit, the request is
refused as it was before the fallback existed.

The history is deliberately not trimmed to make it fit. Dropping a caller's
messages to stay under a limit they never asked about would be a silent loss,
inside a request that was already being quietly moved to another model.

The tests drive the real request path, because a unit test on the fallback
would have asserted the same wrong thing the code did. Both were checked
against the unfixed code and fail there.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 11:01:23 +02:00
1 parent 1b7e11b91e
commit c9c841387a
2 files changed
+221

No files matched your search

+14
View File
@@ -31,6 +31,20 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
if !r.permits(p, name) {
continue
}
// And it has to be able to hold the prompt. The fit was checked against
// the model the caller asked for, before anyone knew a different one
// would serve — so without this a 20k prompt sent to a 32k model lands
// on a 2k one, and the engine truncates it or refuses. Either way the
// caller sees a failure naming a model they never chose.
//
// The history is not trimmed to make it fit. Dropping a caller's
// messages to stay under a limit they did not ask about is a worse
// answer than saying the prompt is too large, and it would be done
// silently, to a request that was already being quietly moved.
_, cand := r.resolve(name)
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
continue
}
res, err := r.acct.Check(p, name, req.estimatedTokens())
if err != nil {
continue