llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on the console's project menu. The same request — project, endpoint, canonical model and the body without stream, stream_options, user, metadata, store and service_tier, keys sorted — is answered from the gateway's memory for the TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank, score) unless --any; only a whole 200 from the model asked for, not a fallback's, a canary's, or one a guardrail changed. Looked up after the limits and the input guardrails; the output policy judges a reused answer; no_store and stored conversations bypass it. Streamed answers are kept put back together and streamed back as chunks. x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost, marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total, llmgw_cache_bytes, llmgw_cache_entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
56 lines
3.3 KiB
Markdown
56 lines
3.3 KiB
Markdown
## CHANGELOG
|
|
|
|
### Reusing answers
|
|
|
|
- **A response cache, per project, off until asked for.** `llmgw keys cache
|
|
--project P --ttl 1h` has the same request — same project, endpoint,
|
|
model by its catalog name, and body with `stream`, `stream_options`,
|
|
`user`, `metadata`, `store` and `service_tier` left out and its keys in
|
|
any order — answered from the gateway's memory instead of the model, for
|
|
the TTL. `--off` stops it; without either it says how it stands. The
|
|
console has it as **Cache answers…** on a project.
|
|
- **Only answers that come out the same are reused:** chat and completions
|
|
asked at temperature 0 or with a seed, and embeddings, rerank and score.
|
|
`--any` reuses sampled answers too. Only a whole 200 from the model asked
|
|
for is kept — not an error, a fallback's or a canary's answer, one a
|
|
guardrail changed or stopped, or one over `router.cache.max_entry_bytes`.
|
|
- **Streams both ways:** a streamed answer is kept put back together into
|
|
one completion (tool calls and reasoning included), and a kept answer is
|
|
streamed as chunks — with the usage chunk if the client asked — to a
|
|
client that streams. `/v1/messages` and `/v1/responses` get it too.
|
|
- **Headers:** `x-llmgw-cache: hit|miss|bypass` on every answer of a
|
|
project that reuses answers, `Age` on a hit. `Cache-Control: no-cache`
|
|
or `no-store`, or `x-llmgw-cache: no`, asks the model anew.
|
|
- **Policies still apply:** limits and the input guardrails run before the
|
|
cache is looked in, the output policy judges a reused answer as it would
|
|
a new one, and a request under `no_store` or continuing a stored
|
|
conversation is never served from it nor kept in it. A project's answers
|
|
are never looked up by another project's keys.
|
|
- **A hit costs nothing:** recorded as a 200 with no tokens and no cost,
|
|
marked `cache_hit` on the usage record (migration 0027), and shown by
|
|
`llmgw ctl request`; it counts towards the requests-per-minute limit only.
|
|
- **In memory only:** `router.cache.max_bytes` (128 MB) for all projects
|
|
together in each gateway process, least recently used dropped first;
|
|
nothing is written to the store. `llmgw_cache_requests_total{model,result}`,
|
|
`llmgw_cache_bytes` and `llmgw_cache_entries`.
|
|
|
|
## BACKLOG
|
|
|
|
**Done (branch `features-c2`):** exact-match answer cache per project
|
|
(`keys cache --ttl/--any/--off`, console **Cache answers…**, `projects`
|
|
columns, 30-second setting cache), reproducible answers only unless
|
|
`--any`, LRU in memory bounded by `router.cache.max_bytes` and
|
|
`max_entry_bytes`, streamed answers kept and replayed as chunks,
|
|
`x-llmgw-cache` and `Cache-Control: no-cache`, guardrails before a hit and
|
|
the output policy on it, `no_store` and threads bypass, a hit recorded as
|
|
`cache_hit` with no tokens and no cost (migration 0027), metrics, docs.
|
|
Left: semantic (embedding-similarity) caching through the embeddings
|
|
models; a cache shared across control planes (each process keeps its own,
|
|
deliberately: a shared one would write every answer to the store); a hit
|
|
still waits its turn in admission, because the guardrails judge the request
|
|
after admission and the key is the request they let through; logprobs in a
|
|
streamed answer leave it unkept; `users forget` does not reach into the
|
|
cache (answers age out by the TTL); no listing or flushing of what is
|
|
cached (`--off` stops it being served at once, the memory is freed by TTL
|
|
and LRU).
|