4 Commits

Include renames

Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	92733317a9	A batch of embeddings is checked text by text against the model's context ... llmgw-kb sends many passages in one /v1/embeddings request. Their sizes were added together and compared with the context, so a batch of passages that each fit was refused as one over-long prompt. The longest input is what has to fit; the total still counts for usage and limits. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-09 11:43:21 +02:00
Ilker ManapandClaude Opus 5	4535bf362f	Price a request by the model that serves it ... The price table is written against the catalog. The name reaching the accountant was the one the caller typed, so every request made by an alias missed the table and fell to the default price — the same request charged two different rates depending on which of a model's names the client happened to be given. With no default price that is zero, which is silent. With one it is a wrong number, which is worse: a budget spent at the rate of whatever the default says rather than at the model's own. The router already resolves the alias, two lines above, to decide whether the key is allowed the model at all. It now passes the resolved name to the accountant as well, in the ordinary path and in the budget fallback. The test charges a real request through an aliased name and compares the figure with arithmetic it states itself, because the first version of it asserted "more than the fallback" and passed with the bug in place. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-28 00:15:40 +02:00
Ilker ManapandClaude Opus 5	c9c841387a	Check the prompt fits the model that will actually serve it ... The context check ran against the model the caller asked for. The budget fallback then swapped in a different one and nothing looked again, so a prompt sized for a 32k model could be sent to a 512-token one. Measured before the fix: 10002 prompt tokens delivered to a model declaring 512. The caller sees that as a truncated reply or an engine error naming a model they never chose, which is a worse outcome than the refusal the fallback exists to avoid — it fails later and explains less. Candidates that cannot hold the prompt are now skipped, and a fallback further down the list that can hold it is used instead. When none fit, the request is refused as it was before the fallback existed. The history is deliberately not trimmed to make it fit. Dropping a caller's messages to stay under a limit they never asked about would be a silent loss, inside a request that was already being quietly moved to another model. The tests drive the real request path, because a unit test on the fallback would have asserted the same wrong thing the code did. Both were checked against the unfixed code and fail there. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-18 11:01:23 +02:00
Ilker ManapandClaude Opus 5	28967edab3	Serve a spent budget something it can afford ... The accounting layer has always exempted a model that costs nothing, with a comment saying it is "what lets a key with no budget left still be served a cheap fallback". That described a caller nobody had written: the budget was checked once, against the requested model, and a refusal was the end of it, so the exemption could never fire. A caller that has run out is now offered the first model in its fallbacks it can still afford. The allowlist still applies — running out of money is not a way to reach a model a key was never allowed — and a model with no fallback is refused exactly as before, or a budget would stop meaning anything. Only a spent budget falls back: a rate limit is answered by waiting a second, and moving the request elsewhere would spend someone else's capacity to avoid a delay nobody asked to avoid. Which uncovered a header that has never worked. x-llmgw-served-model was set in finish, and finish runs after the proxy has written the response — so the one mechanism for telling a client that something else answered has been announcing it to nobody since it was written. Nothing failed, because the only thing asserting it was a comment. It is now set before dispatch, for the capacity fallback as well as this one. Finding that took bisecting a test I had written wrong: it polled for a 429 that never came, because the budget had run out and the fallback had served the request — the feature working, reported as the feature failing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-17 18:26:03 +02:00