Know how big a model's context is, and what it can do

Three things the gateway did not know about its own models, all of which
surfaced this week as somebody else's bug.

Context length. Nothing read it, /v1/models did not publish it, and an
over-long prompt was not refused: it was admitted, scheduled, given a
replica and a slot, and rejected by the engine in whatever shape that
engine chose. A client sizing prompts from /v1/models had nothing to read.
It is now declared per model, published when known, and checked before
authentication work or a queue slot is spent — with a ten percent margin,
because the estimate is bytes over four and being a few percent out must
not turn a valid request into a refusal.

Capabilities. The gateway forwards a tools array to any model, and one that
cannot use them answers in prose; every OpenAI-compatible client then
behaves as though the model is broken, and the only way to find out was to
try it. Declared rather than discovered, because no engine will say — which
is why these are claims, and why the next commit makes engine-check verify
the tool-calling one instead of trusting it.

And a bug found while wiring the first: the token estimate was taken from
the routing prefix. That prefix stops at 2 KB by design, so every prompt
above it estimated as 512 tokens however long it really was, and TPM limits
undercounted exactly the requests that consume the most. Prefix routing
wants a bounded head of the conversation; a token estimate wants all of it.
They are now separate functions, and structured multimodal content counts
rather than measuring zero.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 23:07:49 +02:00
1 parent ea87021b5c
commit e9fe3ad379
5 files changed
+349 -4

No files matched your search

+33
View File
@@ -87,6 +87,39 @@ type Model struct {
// RoutingProfile selects how the router scores endpoints for this model.
RoutingProfile string `koanf:"routing_profile"`
SLO SLO `koanf:"slo"`
// ContextLength is how many tokens one request may use, prompt and
// completion together, and MaxOutput bounds the completion alone. Zero
// means unknown, which is what every model was until now.
//
// The gateway knew neither, so an over-long prompt was not refused: it was
// admitted, given a replica, and failed at the engine, and a client sizing
// its prompts from /v1/models had nothing to read. Worse, the number is not
// the one an operator writes in the engine's arguments — llama.cpp divides
// --ctx-size across its parallel slots, so a pool of 8192 across four slots
// is 2048 a request, and nothing said so.
ContextLength int `koanf:"context_length"`
MaxOutput int `koanf:"max_output"`
// Capabilities is what this model can do beyond plain completion.
Capabilities Capabilities `koanf:"capabilities"`
}
// Capabilities describes what a client may ask a model for.
//
// Declared by the operator rather than discovered, because there is no way to
// ask an engine what its model can do. That makes these claims, and a wrong
// claim is worse than none — which is why `ctl engine-check` verifies tool
// calling against a running engine rather than trusting this.
//
// The reason it exists: the gateway forwards a tools array to any model, and a
// model that cannot use them answers in prose. Every OpenAI-compatible client
// then behaves as though the model is broken, and the only way to find out was
// to try it. That cost an afternoon.
type Capabilities struct {
Tools bool `koanf:"tools"`
Vision bool `koanf:"vision"`
Embeddings bool `koanf:"embeddings"`
}
// Catalog is the set of models the cluster should serve.