Answers to similar questions, not only the same one
CI / test (push) Successful in 8m52s

llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.

The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 17:47:11 +01:00
1 parent ed0d1ec0ad
commit 3a3db8cd20
19 files changed
+514 -33

No files matched your search

+1
View File
@@ -53,6 +53,7 @@ var Reference = []Block{
{Package: "config", Name: "Cache", Title: "Cache", Doc: "", Fields: []Field{
{Key: "max_bytes", Type: "int64", Doc: "MaxBytes bounds every project's answers together; 128 MB unless set.\nPast it, the answers used longest ago go first.", Nested: ""},
{Key: "max_entry_bytes", Type: "int64", Doc: "MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB\nunless set.", Nested: ""},
{Key: "embedding_model", Type: "string", Doc: "EmbeddingModel embeds the questions of projects that reuse answers to\nsimilar ones (llmgw keys cache --similar): an embedding model in the\ncatalog. Unset, those projects reuse only the same request's answer.\nThe gateway asks it on its own account, as it asks a judge.", Nested: ""},
}},
{Package: "catalog", Name: "Canary", Title: "Canary", Doc: "", Fields: []Field{
{Key: "model", Type: "string", Doc: "Model is the candidate: a catalog model, usually this one at newer\nweights, which clients cannot name themselves.", Nested: ""},