Answers to similar questions, not only the same one
llmgw keys cache --similar 0.95 also gives a chat question the answer to one like it: the last user message embedded by router.cache.embedding_model (asked on the gateway's account, as a judge is, within 2 s), compared by cosine with the questions answered in the same conversation — system prompt, earlier turns, tools and sampling the same. The exact cache is asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar with x-llmgw-cache-similarity, charged as a hit. The vectors live with their answers in the gateway's memory, counted in max_bytes and evicted with them, not in a pgvector table as BACKLOG I1 planned: the cache keeps answer text out of the store on purpose. A question over 2,000 characters is reused only when the same. Migration 0051 holds the setting. BACKLOG I1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed0d1ec0ad
commit
3a3db8cd20
19 files changed
+514
-33
No files matched your search
+3
-1
@@ -72,7 +72,9 @@ each answer that may be reused, as the client got it, and a hash of the
|
||||
request it answers, in the gateway process's memory — never in the store,
|
||||
never on disk — for the project's TTL or until pushed out by newer ones
|
||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||
policy. Forgetting a person does not reach into it; the answers age out by
|
||||
policy. A project that reuses answers to similar questions (`--similar`)
|
||||
also has each question's last user message embedded, and the vector kept
|
||||
beside its answer, in the same memory and for as long. Forgetting a person does not reach into it; the answers age out by
|
||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||
was answered from the cache.
|
||||
|
||||
|
||||
+40
-2
@@ -2158,8 +2158,46 @@ router:
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
### Similar questions
|
||||
|
||||
A help desk asked "how do I reset my password" in forty wordings asks the
|
||||
model forty times under the exact cache. `--similar` also gives a chat
|
||||
question the answer to one like it:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/support/desk --ttl 24h --similar 0.95
|
||||
```
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
embedding_model: bge-m3 # an embedding model in the catalog
|
||||
```
|
||||
|
||||
- **What is compared:** the last user message, embedded by
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, or after a tool's
|
||||
result, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
to meet. A question that differs only in a number or a name ("2+2" and
|
||||
"2+3") can be closer than that, so a project whose questions turn on
|
||||
figures should not use it. A question over 2,000 characters is reused only
|
||||
when it is the same.
|
||||
- **Said:** `x-llmgw-cache: similar`, with `x-llmgw-cache-similarity`
|
||||
and `Age`. Recorded and charged as a hit. The same rules as a hit decide
|
||||
which answers are reused and kept; guardrails judge the request before.
|
||||
- **The embedding** is asked of the model by the gateway on its own account,
|
||||
as a judge is, within 2 seconds; every miss waits for it. Without the
|
||||
model, or when it does not answer, only the same request is reused. The
|
||||
vectors are kept in memory with their answers and counted in
|
||||
`router.cache.max_bytes`.
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, similar hits, misses
|
||||
and bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
@@ -53,6 +53,7 @@ rather than repeated.
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `embedding_model` | `string` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -769,6 +770,15 @@ Past it, the answers used longest ago go first.
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
### `embedding_model`
|
||||
|
||||
`string`
|
||||
|
||||
EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
catalog. Unset, those projects reuse only the same request's answer.
|
||||
The gateway asks it on its own account, as it asks a judge.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
|
||||
Reference in new issue
Block a user