Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on the console's project menu. The same request — project, endpoint, canonical model and the body without stream, stream_options, user, metadata, store and service_tier, keys sorted — is answered from the gateway's memory for the TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank, score) unless --any; only a whole 200 from the model asked for, not a fallback's, a canary's, or one a guardrail changed. Looked up after the limits and the input guardrails; the output policy judges a reused answer; no_store and stored conversations bypass it. Streamed answers are kept put back together and streamed back as chunks. x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost, marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total, llmgw_cache_bytes, llmgw_cache_entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
44f3a9e31d
27 files changed
+2066
-11
No files matched your search
@@ -47,6 +47,8 @@ rather than repeated.
|
||||
| `max_lines` | `int` | [Batch](#batch) |
|
||||
| `concurrency` | `int` | [Batch](#batch) |
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -204,6 +206,7 @@ rather than repeated.
|
||||
| `tag_labels` | `[]string` | [Router](#router) |
|
||||
| `max_body_bytes` | `int64` | [Router](#router) |
|
||||
| `batch` | `Batch` | [Router](#router) |
|
||||
| `cache` | `Cache` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
| `scoring` | `Scoring` | [Router](#router) |
|
||||
@@ -570,6 +573,22 @@ every interactive request.
|
||||
RetainDays is how long a batch's answer and error files are kept;
|
||||
30 unless set. Uploaded files are kept until deleted.
|
||||
|
||||
## Cache
|
||||
|
||||
### `max_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
MaxBytes bounds every project's answers together; 128 MB unless set.
|
||||
Past it, the answers used longest ago go first.
|
||||
|
||||
### `max_entry_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -1692,6 +1711,16 @@ another. Needs the store.
|
||||
|
||||
Takes a [Batch](#batch) block.
|
||||
|
||||
### `cache`
|
||||
|
||||
`Cache`
|
||||
|
||||
Cache bounds the answers kept for the projects that reuse them
|
||||
(llmgw keys cache). Which projects do, and for how long, is theirs
|
||||
to set; this is only how much memory it may take.
|
||||
|
||||
Takes a [Cache](#cache) block.
|
||||
|
||||
### `trusted_proxies`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user