Three things the gateway did not know about its own models, all of which surfaced this week as somebody else's bug. Context length. Nothing read it, /v1/models did not publish it, and an over-long prompt was not refused: it was admitted, scheduled, given a replica and a slot, and rejected by the engine in whatever shape that engine chose. A client sizing prompts from /v1/models had nothing to read. It is now declared per model, published when known, and checked before authentication work or a queue slot is spent — with a ten percent margin, because the estimate is bytes over four and being a few percent out must not turn a valid request into a refusal. Capabilities. The gateway forwards a tools array to any model, and one that cannot use them answers in prose; every OpenAI-compatible client then behaves as though the model is broken, and the only way to find out was to try it. Declared rather than discovered, because no engine will say — which is why these are claims, and why the next commit makes engine-check verify the tool-calling one instead of trusting it. And a bug found while wiring the first: the token estimate was taken from the routing prefix. That prefix stops at 2 KB by design, so every prompt above it estimated as 512 tokens however long it really was, and TPM limits undercounted exactly the requests that consume the most. Prefix routing wants a bounded head of the conversation; a token estimate wants all of it. They are now separate functions, and structured multimodal content counts rather than measuring zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ea87021b5c
commit
e9fe3ad379
5 files changed
+349
-4
No files matched your search
@@ -87,6 +87,39 @@ type Model struct {
|
||||
// RoutingProfile selects how the router scores endpoints for this model.
|
||||
RoutingProfile string `koanf:"routing_profile"`
|
||||
SLO SLO `koanf:"slo"`
|
||||
|
||||
// ContextLength is how many tokens one request may use, prompt and
|
||||
// completion together, and MaxOutput bounds the completion alone. Zero
|
||||
// means unknown, which is what every model was until now.
|
||||
//
|
||||
// The gateway knew neither, so an over-long prompt was not refused: it was
|
||||
// admitted, given a replica, and failed at the engine, and a client sizing
|
||||
// its prompts from /v1/models had nothing to read. Worse, the number is not
|
||||
// the one an operator writes in the engine's arguments — llama.cpp divides
|
||||
// --ctx-size across its parallel slots, so a pool of 8192 across four slots
|
||||
// is 2048 a request, and nothing said so.
|
||||
ContextLength int `koanf:"context_length"`
|
||||
MaxOutput int `koanf:"max_output"`
|
||||
|
||||
// Capabilities is what this model can do beyond plain completion.
|
||||
Capabilities Capabilities `koanf:"capabilities"`
|
||||
}
|
||||
|
||||
// Capabilities describes what a client may ask a model for.
|
||||
//
|
||||
// Declared by the operator rather than discovered, because there is no way to
|
||||
// ask an engine what its model can do. That makes these claims, and a wrong
|
||||
// claim is worse than none — which is why `ctl engine-check` verifies tool
|
||||
// calling against a running engine rather than trusting this.
|
||||
//
|
||||
// The reason it exists: the gateway forwards a tools array to any model, and a
|
||||
// model that cannot use them answers in prose. Every OpenAI-compatible client
|
||||
// then behaves as though the model is broken, and the only way to find out was
|
||||
// to try it. That cost an afternoon.
|
||||
type Capabilities struct {
|
||||
Tools bool `koanf:"tools"`
|
||||
Vision bool `koanf:"vision"`
|
||||
Embeddings bool `koanf:"embeddings"`
|
||||
}
|
||||
|
||||
// Catalog is the set of models the cluster should serve.
|
||||
|
||||
Reference in new issue
Block a user