Answers to similar questions, not only the same one
CI / test (push) Successful in 8m52s

llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 17:47:11 +01:00
1 parent ed0d1ec0ad
commit 3a3db8cd20
19 files changed
+514 -33

No files matched your search

+40 -2
View File
@@ -2158,8 +2158,46 @@ router:
max_entry_bytes: 2097152 # 2 MB
```
`llmgw_cache_requests_total{model,result}` counts hits, misses and
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
### Similar questions
A help desk asked "how do I reset my password" in forty wordings asks the
model forty times under the exact cache. `--similar` also gives a chat
question the answer to one like it:
```
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
```
```yaml
router:
cache:
embedding_model: bge-m3 # an embedding model in the catalog
```
- **What is compared:** the last user message, embedded by
`router.cache.embedding_model`, against the questions already answered
in the same conversation — the same system prompt, the same earlier turns,
tools, `response_format` and sampling settings, the same model and
project. A like question in another conversation, or after a tool's
result, is not taken for this one. The exact cache is asked first, and an
exact hit embeds nothing.
- **How alike:** the cosine similarity of the two embeddings, at least the
project's `--similar`. 0.95 is strict; below 0.9 different questions start
to meet. A question that differs only in a number or a name ("2+2" and
"2+3") can be closer than that, so a project whose questions turn on
figures should not use it. A question over 2,000 characters is reused only
when it is the same.
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
which answers are reused and kept; guardrails judge the request before.
- **The embedding** is asked of the model by the gateway on its own account,
as a judge is, within 2 seconds; every miss waits for it. Without the
model, or when it does not answer, only the same request is reused. The
vectors are kept in memory with their answers and counted in
`router.cache.max_bytes`.
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
console has it as **Cache answers…** on a project.
## Batches