Ilker ManapandClaude Opus 5
e9fe3ad379
Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
61 lines
2.5 KiB
Go
61 lines
2.5 KiB
Go
package router
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
|
)
|
|
|
|
// checkContextFits reports why a request cannot fit in a model's context, or
|
|
// "" when it can.
|
|
//
|
|
// Refused here rather than at the engine. An over-long prompt used to be
|
|
// admitted, scheduled, given a replica and a slot, and only then rejected —
|
|
// paying the whole cost of the request to learn something knowable from the
|
|
// request itself. The engine's error also arrives in whatever shape that engine
|
|
// chose, which is not something a client can act on.
|
|
//
|
|
// The prompt estimate is bytes over four, the same crude stand-in used for
|
|
// scoring, so the comparison is approximate in one direction: it can let
|
|
// through a request the engine will refuse. That is the right way round. A
|
|
// tokenizer would make it exact, and needing one is why plan §7.5 tier 2 is v2.
|
|
// To keep an estimate that reads slightly low from refusing valid requests, a
|
|
// margin is allowed before anything is rejected.
|
|
func checkContextFits(m catalog.Model, promptTokens, maxTokens int) string {
|
|
if m.ContextLength <= 0 {
|
|
// Unknown, which is every model that has not declared one. Saying
|
|
// nothing beats guessing a limit and refusing work over it.
|
|
return ""
|
|
}
|
|
|
|
if m.MaxOutput > 0 && maxTokens > m.MaxOutput {
|
|
return fmt.Sprintf("max_tokens is %d, but %q allows at most %d",
|
|
maxTokens, m.Name, m.MaxOutput)
|
|
}
|
|
|
|
// Only the prompt is estimated; max_tokens is exact when given. A request
|
|
// that does not say how much it wants back is judged on its prompt alone,
|
|
// because assuming the whole remaining window would refuse most requests.
|
|
need := promptTokens + maxTokens
|
|
allowed := m.ContextLength + contextEstimateMargin(m.ContextLength)
|
|
if need <= allowed {
|
|
return ""
|
|
}
|
|
if maxTokens > 0 {
|
|
return fmt.Sprintf(
|
|
"about %d prompt tokens plus max_tokens %d exceeds the %d-token context of %q",
|
|
promptTokens, maxTokens, m.ContextLength, m.Name)
|
|
}
|
|
return fmt.Sprintf("the prompt is about %d tokens, over the %d-token context of %q",
|
|
promptTokens, m.ContextLength, m.Name)
|
|
}
|
|
|
|
// contextEstimateMargin is the slack allowed for the bytes-over-four estimate.
|
|
//
|
|
// That estimate is wrong in both directions depending on the text — code and
|
|
// non-Latin scripts tokenize very differently from English prose — and being
|
|
// wrong by a few percent must not turn a valid request into a refusal. Ten
|
|
// percent is enough to absorb that while still catching a prompt that is
|
|
// genuinely too big, which is the case worth catching.
|
|
func contextEstimateMargin(context int) int { return context / 10 }
|