Reusing answers: a response cache per project, exact match, opt-in with a TTL

llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:50:53 +02:00
1 parent 30f7f3474a
commit 44f3a9e31d
27 files changed
+2066 -11

No files matched your search

+10
View File
@@ -67,10 +67,20 @@ until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
them can read them, and a project under a `no_store` policy cannot make
them.
**Reused answers** — only for a project that asked (`llmgw keys cache`):
each answer that may be reused, as the client got it, and a hash of the
request it answers, in the gateway process's memory — never in the store,
never on disk — for the project's TTL or until pushed out by newer ones
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
policy. Forgetting a person does not reach into it; the answers age out by
the TTL. The store keeps only the setting, and a usage record's mark that it
was answered from the cache.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations, notes or kept requests, or sent a batch.
(Reused answers are held in memory only, as above.)
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,