Reusing answers: a response cache per project, exact match, opt-in with a TTL

llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on
the console's project menu. The same request — project, endpoint, canonical
model and the body without stream, stream_options, user, metadata, store and
service_tier, keys sorted — is answered from the gateway's memory for the
TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank,
score) unless --any; only a whole 200 from the model asked for, not a
fallback's, a canary's, or one a guardrail changed.

Looked up after the limits and the input guardrails; the output policy
judges a reused answer; no_store and stored conversations bypass it.
Streamed answers are kept put back together and streamed back as chunks.
x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or
x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost,
marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded
by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total,
llmgw_cache_bytes, llmgw_cache_entries.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:50:53 +02:00
1 parent 30f7f3474a
commit 44f3a9e31d
27 files changed
+2066 -11

No files matched your search

+3 -1
View File
@@ -195,7 +195,9 @@ row: add a team or project inside it, rename, move, limits for the level and
for each of its members, its capacity quota, and for a project how long
conversations and agent notes are kept, and whether its requests are
(**Keep requests…**: for how long, and what share), and whether they stay in
the cluster (**Requests stay in the cluster…**: never an outside provider). **Longest key lifetime…** caps how
the cluster (**Requests stay in the cluster…**: never an outside provider),
and whether the same request is answered again from memory (**Cache
answers…**: for how long, and whether only reproducible answers or any). **Longest key lifetime…** caps how
long a key made there may live, and is the expiry of one made without any
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
**Remove** takes away only something empty.
+10
View File
@@ -67,10 +67,20 @@ until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
them can read them, and a project under a `no_store` policy cannot make
them.
**Reused answers** — only for a project that asked (`llmgw keys cache`):
each answer that may be reused, as the client got it, and a hash of the
request it answers, in the gateway process's memory — never in the store,
never on disk — for the project's TTL or until pushed out by newer ones
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
policy. Forgetting a person does not reach into it; the answers age out by
the TTL. The store keeps only the setting, and a usage record's mark that it
was answered from the cache.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations, notes or kept requests, or sent a batch.
(Reused answers are held in memory only, as above.)
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
+62
View File
@@ -1208,6 +1208,68 @@ llmgw keys requests --project Acme/ml/train --off
requests past their hours, or of a project turned off, are removed hourly.
The console has it as **Keep requests…** on a project.
## Reusing answers
A project that asks the same thing again and again — an evaluation run over
the same prompts, embeddings of documents that do not change, an agent that
retries — can have the same request answered from memory rather than by the
model:
```
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
llmgw keys cache --project Acme/ml/train # how it stands
llmgw keys cache --project Acme/ml/train --off
```
- **What is the same request:** the same project, the same endpoint, the same
model (by its catalog name, whichever alias was used) and the same body,
with `stream`, `stream_options`, `user`, `metadata`, `store` and
`service_tier` left out and the keys in any order. Keys of another project
never see the answer, whatever they send.
- **Which answers are reused:** a chat or completion asked at `temperature`
0 or with a `seed`, and every embedding, rerank and score, which sample
nothing. A request sampled above temperature 0 with no seed asks for a new
answer, and is asked anew — unless the project says `--any`. Only a whole
200 from the model that was asked for is kept: not an error, not a
fallback model's or a canary's answer, not one a guardrail changed or
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
- **Streams:** an answer given whole is streamed to a client that asks for a
stream, as one chunk per choice with the usage chunk if it asked for that;
a streamed answer is kept put back together, and served whole to a client
that does not stream. Through `/v1/messages` and `/v1/responses` too.
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
can be reused), and a hit says its `Age` in seconds.
- **Policies still apply:** the limits are checked and the guardrails judge
the request before the cache is looked in, and the output policy judges a
reused answer as it would a new one. A request continuing a stored
conversation, or under a `no_store` policy, is never answered from the
cache nor kept in it.
- **What a hit costs:** nothing. It is recorded as a request — status 200,
the model, the key, the rate limit's count — with no tokens and no cost,
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
budget and no token limit. The answer itself carries the usage the model
reported when it was first given.
- **Where it lives:** in each gateway process's memory, at most
`router.cache.max_bytes` (128 MB) of answers for all projects together,
those used longest ago dropped first; nothing is written to the store, and
a restart begins it again. With two control planes behind a balancer each
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
applies to what is already kept.
```yaml
router:
cache:
max_bytes: 268435456 # 256 MB
max_entry_bytes: 2097152 # 2 MB
```
`llmgw_cache_requests_total{model,result}` counts hits, misses and
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
console has it as **Cache answers…** on a project.
## Batches
Work that can wait — a night's classification, embeddings for a document
+29
View File
@@ -47,6 +47,8 @@ rather than repeated.
| `max_lines` | `int` | [Batch](#batch) |
| `concurrency` | `int` | [Batch](#batch) |
| `retain_days` | `int` | [Batch](#batch) |
| `max_bytes` | `int64` | [Cache](#cache) |
| `max_entry_bytes` | `int64` | [Cache](#cache) |
| `model` | `string` | [Canary](#canary) |
| `percent` | `int` | [Canary](#canary) |
| `min_requests` | `int` | [Canary](#canary) |
@@ -204,6 +206,7 @@ rather than repeated.
| `tag_labels` | `[]string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `batch` | `Batch` | [Router](#router) |
| `cache` | `Cache` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
@@ -570,6 +573,22 @@ every interactive request.
RetainDays is how long a batch's answer and error files are kept;
30 unless set. Uploaded files are kept until deleted.
## Cache
### `max_bytes`
`int64`
MaxBytes bounds every project's answers together; 128 MB unless set.
Past it, the answers used longest ago go first.
### `max_entry_bytes`
`int64`
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
unless set.
## Canary
### `model`
@@ -1692,6 +1711,16 @@ another. Needs the store.
Takes a [Batch](#batch) block.
### `cache`
`Cache`
Cache bounds the answers kept for the projects that reuse them
(llmgw keys cache). Which projects do, and for how long, is theirs
to set; this is only how much memory it may take.
Takes a [Cache](#cache) block.
### `trusted_proxies`
`[]string`