Ilker ManapandClaude Opus 5
e9fe3ad379
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>