Merge C17 response cache

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:51:19 +02:00
commit f48f55a866
27 files changed
+2066 -11

No files matched your search

+55
View File
@@ -0,0 +1,55 @@
## CHANGELOG
### Reusing answers
- **A response cache, per project, off until asked for.** `llmgw keys cache
--project P --ttl 1h` has the same request — same project, endpoint,
model by its catalog name, and body with `stream`, `stream_options`,
`user`, `metadata`, `store` and `service_tier` left out and its keys in
any order — answered from the gateway's memory instead of the model, for
the TTL. `--off` stops it; without either it says how it stands. The
console has it as **Cache answers…** on a project.
- **Only answers that come out the same are reused:** chat and completions
asked at temperature 0 or with a seed, and embeddings, rerank and score.
`--any` reuses sampled answers too. Only a whole 200 from the model asked
for is kept — not an error, a fallback's or a canary's answer, one a
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
- **Streams both ways:** a streamed answer is kept put back together into
one completion (tool calls and reasoning included), and a kept answer is
streamed as chunks — with the usage chunk if the client asked — to a
client that streams. `/v1/messages` and `/v1/responses` get it too.
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
- **Policies still apply:** limits and the input guardrails run before the
cache is looked in, the output policy judges a reused answer as it would
a new one, and a request under `no_store` or continuing a stored
conversation is never served from it nor kept in it. A project's answers
are never looked up by another project's keys.
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
marked `cache_hit` on the usage record (migration 0027), and shown by
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
together in each gateway process, least recently used dropped first;
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
`llmgw_cache_bytes` and `llmgw_cache_entries`.
## BACKLOG
**Done (branch `features-c2`):** exact-match answer cache per project
(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects`
columns, 30-second setting cache), reproducible answers only unless
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
`max_entry_bytes`, streamed answers kept and replayed as chunks,
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and
the output policy on it, `no_store` and threads bypass, a hit recorded as
`cache_hit` with no tokens and no cost (migration 0027), metrics, docs.
Left: semantic (embedding-similarity) caching through the embeddings
models; a cache shared across control planes (each process keeps its own,
deliberately: a shared one would write every answer to the store); a hit
still waits its turn in admission, because the guardrails judge the request
after admission and the key is the request they let through; logprobs in a
streamed answer leave it unkept; `users forget` does not reach into the
cache (answers age out by the TTL); no listing or flushing of what is
cached (`--off` stops it being served at once, the memory is freed by TTL
and LRU).