Commit Graph
2 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 3858114ada A picture in a prompt counts as what a model makes of it, not its base64
CI / test (push) Successful in 10m43s
A 1280x720 PNG sent to qwen3-vl-8b was estimated at 138 thousand prompt
tokens and refused against its 8k context before reaching the engine, which
makes about a thousand of it. Parts are counted by kind: text by its text,
a picture or a sound as a thousand tokens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-11 20:45:06 +02:00
Ilker ManapandClaude Opus 5 e9fe3ad379 Know how big a model's context is, and what it can do
CI / test (push) Successful in 1m32s
Three things the gateway did not know about its own models, all of which
surfaced this week as somebody else's bug.

Context length. Nothing read it, /v1/models did not publish it, and an
over-long prompt was not refused: it was admitted, scheduled, given a
replica and a slot, and rejected by the engine in whatever shape that
engine chose. A client sizing prompts from /v1/models had nothing to read.
It is now declared per model, published when known, and checked before
authentication work or a queue slot is spent — with a ten percent margin,
because the estimate is bytes over four and being a few percent out must
not turn a valid request into a refusal.

Capabilities. The gateway forwards a tools array to any model, and one that
cannot use them answers in prose; every OpenAI-compatible client then
behaves as though the model is broken, and the only way to find out was to
try it. Declared rather than discovered, because no engine will say — which
is why these are claims, and why the next commit makes engine-check verify
the tool-calling one instead of trusting it.

And a bug found while wiring the first: the token estimate was taken from
the routing prefix. That prefix stops at 2 KB by design, so every prompt
above it estimated as 512 tokens however long it really was, and TPM limits
undercounted exactly the requests that consume the most. Prefix routing
wants a bounded head of the conversation; a token estimate wants all of it.
They are now separate functions, and structured multimodal content counts
rather than measuring zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 23:07:49 +02:00