Files
llmgw/notes/c17.md
T
Ilker ManapandClaude Opus 5.5 44f3a9e31d Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 23:50:53 +02:00

56 lines
3.3 KiB
Markdown

## CHANGELOG
### Reusing answers
- **A response cache, per project, off until asked for.** `llmgw keys cache
--project P --ttl 1h` has the same request — same project, endpoint,
model by its catalog name, and body with `stream`, `stream_options`,
`user`, `metadata`, `store` and `service_tier` left out and its keys in
any order — answered from the gateway's memory instead of the model, for
the TTL. `--off` stops it; without either it says how it stands. The
console has it as **Cache answers…** on a project.
- **Only answers that come out the same are reused:** chat and completions
asked at temperature 0 or with a seed, and embeddings, rerank and score.
`--any` reuses sampled answers too. Only a whole 200 from the model asked
for is kept — not an error, a fallback's or a canary's answer, one a
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
- **Streams both ways:** a streamed answer is kept put back together into
one completion (tool calls and reasoning included), and a kept answer is
streamed as chunks — with the usage chunk if the client asked — to a
client that streams. `/v1/messages` and `/v1/responses` get it too.
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
- **Policies still apply:** limits and the input guardrails run before the
cache is looked in, the output policy judges a reused answer as it would
a new one, and a request under `no_store` or continuing a stored
conversation is never served from it nor kept in it. A project's answers
are never looked up by another project's keys.
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
marked `cache_hit` on the usage record (migration 0027), and shown by
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
together in each gateway process, least recently used dropped first;
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
`llmgw_cache_bytes` and `llmgw_cache_entries`.
## BACKLOG
**Done (branch `features-c2`):** exact-match answer cache per project
(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects`
columns, 30-second setting cache), reproducible answers only unless
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
`max_entry_bytes`, streamed answers kept and replayed as chunks,
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and
the output policy on it, `no_store` and threads bypass, a hit recorded as
`cache_hit` with no tokens and no cost (migration 0027), metrics, docs.
Left: semantic (embedding-similarity) caching through the embeddings
models; a cache shared across control planes (each process keeps its own,
deliberately: a shared one would write every answer to the store); a hit
still waits its turn in admission, because the guardrails judge the request
after admission and the key is the request they let through; logprobs in a
streamed answer leave it unkept; `users forget` does not reach into the
cache (answers age out by the TTL); no listing or flushing of what is
cached (`--off` stops it being served at once, the memory is freed by TTL
and LRU).