llmgw keys cache --similar 0.95 also gives a chat question the answer to one like it: the last user message embedded by router.cache.embedding_model (asked on the gateway's account, as a judge is, within 2 s), compared by cosine with the questions answered in the same conversation — system prompt, earlier turns, tools and sampling the same. The exact cache is asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar with x-llmgw-cache-similarity, charged as a hit. The vectors live with their answers in the gateway's memory, counted in max_bytes and evicted with them, not in a pgvector table as BACKLOG I1 planned: the cache keeps answer text out of the store on purpose. A question over 2,000 characters is reused only when the same. Migration 0051 holds the setting. BACKLOG I1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed0d1ec0ad
commit
3a3db8cd20
19 files changed
+514
-33
No files matched your search
+40
-2
@@ -2158,8 +2158,46 @@ router:
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
### Similar questions
|
||||
|
||||
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||
model forty times under the exact cache. `--similar` also gives a chat
|
||||
question the answer to one like it:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||
```
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
embedding_model: bge-m3 # an embedding model in the catalog
|
||||
```
|
||||
|
||||
- **What is compared:** the last user message, embedded by
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, or after a tool's
|
||||
result, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
to meet. A question that differs only in a number or a name ("2+2" and
|
||||
"2+3") can be closer than that, so a project whose questions turn on
|
||||
figures should not use it. A question over 2,000 characters is reused only
|
||||
when it is the same.
|
||||
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||
which answers are reused and kept; guardrails judge the request before.
|
||||
- **The embedding** is asked of the model by the gateway on its own account,
|
||||
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||
model, or when it does not answer, only the same request is reused. The
|
||||
vectors are kept in memory with their answers and counted in
|
||||
`router.cache.max_bytes`.
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Reference in new issue
Block a user