Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on the console's project menu. The same request — project, endpoint, canonical model and the body without stream, stream_options, user, metadata, store and service_tier, keys sorted — is answered from the gateway's memory for the TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank, score) unless --any; only a whole 200 from the model asked for, not a fallback's, a canary's, or one a guardrail changed. Looked up after the limits and the input guardrails; the output policy judges a reused answer; no_store and stored conversations bypass it. Streamed answers are kept put back together and streamed back as chunks. x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost, marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total, llmgw_cache_bytes, llmgw_cache_entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
44f3a9e31d
27 files changed
+2066
-11
No files matched your search
@@ -1208,6 +1208,68 @@ llmgw keys requests --project Acme/ml/train --off
|
||||
requests past their hours, or of a project turned off, are removed hourly.
|
||||
The console has it as **Keep requests…** on a project.
|
||||
|
||||
## Reusing answers
|
||||
|
||||
A project that asks the same thing again and again — an evaluation run over
|
||||
the same prompts, embeddings of documents that do not change, an agent that
|
||||
retries — can have the same request answered from memory rather than by the
|
||||
model:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
|
||||
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
|
||||
llmgw keys cache --project Acme/ml/train # how it stands
|
||||
llmgw keys cache --project Acme/ml/train --off
|
||||
```
|
||||
|
||||
- **What is the same request:** the same project, the same endpoint, the same
|
||||
model (by its catalog name, whichever alias was used) and the same body,
|
||||
with `stream`, `stream_options`, `user`, `metadata`, `store` and
|
||||
`service_tier` left out and the keys in any order. Keys of another project
|
||||
never see the answer, whatever they send.
|
||||
- **Which answers are reused:** a chat or completion asked at `temperature`
|
||||
0 or with a `seed`, and every embedding, rerank and score, which sample
|
||||
nothing. A request sampled above temperature 0 with no seed asks for a new
|
||||
answer, and is asked anew — unless the project says `--any`. Only a whole
|
||||
200 from the model that was asked for is kept: not an error, not a
|
||||
fallback model's or a canary's answer, not one a guardrail changed or
|
||||
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
|
||||
- **Streams:** an answer given whole is streamed to a client that asks for a
|
||||
stream, as one chunk per choice with the usage chunk if it asked for that;
|
||||
a streamed answer is kept put back together, and served whole to a client
|
||||
that does not stream. Through `/v1/messages` and `/v1/responses` too.
|
||||
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
|
||||
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
|
||||
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
|
||||
can be reused), and a hit says its `Age` in seconds.
|
||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||
the request before the cache is looked in, and the output policy judges a
|
||||
reused answer as it would a new one. A request continuing a stored
|
||||
conversation, or under a `no_store` policy, is never answered from the
|
||||
cache nor kept in it.
|
||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||
budget and no token limit. The answer itself carries the usage the model
|
||||
reported when it was first given.
|
||||
- **Where it lives:** in each gateway process's memory, at most
|
||||
`router.cache.max_bytes` (128 MB) of answers for all projects together,
|
||||
those used longest ago dropped first; nothing is written to the store, and
|
||||
a restart begins it again. With two control planes behind a balancer each
|
||||
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
|
||||
applies to what is already kept.
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
max_bytes: 268435456 # 256 MB
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
|
||||
Reference in new issue
Block a user