llmgw/notes/c17.md

CHANGELOG

Reusing answers

• A response cache, per project, off until asked for. llmgw keys cache --project P --ttl 1h has the same request — same project, endpoint, model by its catalog name, and body with stream, stream_options, user, metadata, store and service_tier left out and its keys in any order — answered from the gateway's memory instead of the model, for the TTL. --off stops it; without either it says how it stands. The console has it as Cache answers… on a project.
• Only answers that come out the same are reused: chat and completions asked at temperature 0 or with a seed, and embeddings, rerank and score. --any reuses sampled answers too. Only a whole 200 from the model asked for is kept — not an error, a fallback's or a canary's answer, one a guardrail changed or stopped, or one over router.cache.max_entry_bytes.
• Streams both ways: a streamed answer is kept put back together into one completion (tool calls and reasoning included), and a kept answer is streamed as chunks — with the usage chunk if the client asked — to a client that streams. /v1/messages and /v1/responses get it too.
• Headers: x-llmgw-cache: hit|miss|bypass on every answer of a project that reuses answers, Age on a hit. Cache-Control: no-cache or no-store, or x-llmgw-cache: no, asks the model anew.
• Policies still apply: limits and the input guardrails run before the cache is looked in, the output policy judges a reused answer as it would a new one, and a request under no_store or continuing a stored conversation is never served from it nor kept in it. A project's answers are never looked up by another project's keys.
• A hit costs nothing: recorded as a 200 with no tokens and no cost, marked cache_hit on the usage record (migration 0027), and shown by llmgw ctl request; it counts towards the requests-per-minute limit only.
• In memory only: router.cache.max_bytes (128 MB) for all projects together in each gateway process, least recently used dropped first; nothing is written to the store. llmgw_cache_requests_total{model,result}, llmgw_cache_bytes and llmgw_cache_entries.

BACKLOG

Done (branch features-c2): exact-match answer cache per project (keys cache --ttl/--any/--off, console Cache answers…, projects columns, 30-second setting cache), reproducible answers only unless --any, LRU in memory bounded by router.cache.max_bytes and max_entry_bytes, streamed answers kept and replayed as chunks, x-llmgw-cache and Cache-Control: no-cache, guardrails before a hit and the output policy on it, no_store and threads bypass, a hit recorded as cache_hit with no tokens and no cost (migration 0027), metrics, docs. Left: semantic (embedding-similarity) caching through the embeddings models; a cache shared across control planes (each process keeps its own, deliberately: a shared one would write every answer to the store); a hit still waits its turn in admission, because the guardrails judge the request after admission and the key is the request they let through; logprobs in a streamed answer leave it unkept; users forget does not reach into the cache (answers age out by the TTL); no listing or flushing of what is cached (--off stops it being served at once, the memory is freed by TTL and LRU).