Answers to similar questions, not only the same one
llmgw keys cache --similar 0.95 also gives a chat question the answer to one like it: the last user message embedded by router.cache.embedding_model (asked on the gateway's account, as a judge is, within 2 s), compared by cosine with the questions answered in the same conversation — system prompt, earlier turns, tools and sampling the same. The exact cache is asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar with x-llmgw-cache-similarity, charged as a hit. The vectors live with their answers in the gateway's memory, counted in max_bytes and evicted with them, not in a pgvector table as BACKLOG I1 planned: the cache keeps answer text out of the store on purpose. A question over 2,000 characters is reused only when the same. Migration 0051 holds the setting. BACKLOG I1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed0d1ec0ad
commit
3a3db8cd20
19 files changed
+514
-33
No files matched your search
+7
-1
@@ -1496,7 +1496,13 @@ Azure/Anthropic/Bedrock/Vertex adapters, request hooks a customer plugs in,
|
||||
routing by what a request holds. Not taken: a realtime (WebSocket audio)
|
||||
API, prompt management, an `npx` install.
|
||||
|
||||
### I1. Answers to similar questions, not only the same one (M)
|
||||
### I1. Answers to similar questions, not only the same one (M) — done 2026-10-25
|
||||
`keys cache --similar`, `router.cache.embedding_model`, `internal/router/similar.go`. Changed
|
||||
from the plan: the vectors are in the gateway's memory beside the exact answers, not a pgvector
|
||||
table. The exact cache keeps answer text out of the store on purpose (data.md), and a table of
|
||||
answers would undo that; it also works on SQLite. The usage row marks a similar hit as a hit
|
||||
(no column of its own). Open: the 40-rewordings and eval-suite runs on the PoC.
|
||||
The original item follows.
|
||||
The answer cache (`internal/router/cache.go`, C17) answers a request whose
|
||||
body is the same byte for byte. A help desk asked "how do I reset my
|
||||
password" in forty wordings asks the model forty times.
|
||||
|
||||
Reference in new issue
Block a user