llmgw keys cache --similar 0.95 also gives a chat question the answer to one like it: the last user message embedded by router.cache.embedding_model (asked on the gateway's account, as a judge is, within 2 s), compared by cosine with the questions answered in the same conversation — system prompt, earlier turns, tools and sampling the same. The exact cache is asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar with x-llmgw-cache-similarity, charged as a hit. The vectors live with their answers in the gateway's memory, counted in max_bytes and evicted with them, not in a pgvector table as BACKLOG I1 planned: the cache keeps answer text out of the store on purpose. A question over 2,000 characters is reused only when the same. Migration 0051 holds the setting. BACKLOG I1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
ed0d1ec0ad
commit
3a3db8cd20
19 files changed
+514
-33
No files matched your search
@@ -798,6 +798,11 @@ type Cache struct {
|
||||
// MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
// unless set.
|
||||
MaxEntryBytes int64 `koanf:"max_entry_bytes"`
|
||||
// EmbeddingModel embeds the questions of projects that reuse answers to
|
||||
// similar ones (llmgw keys cache --similar): an embedding model in the
|
||||
// catalog. Unset, those projects reuse only the same request's answer.
|
||||
// The gateway asks it on its own account, as it asks a judge.
|
||||
EmbeddingModel string `koanf:"embedding_model"`
|
||||
}
|
||||
|
||||
// Default returns the configuration used when nothing is specified.
|
||||
|
||||
Reference in new issue
Block a user