Passages of one organization never share a vLLM prefix cache with another's

A request given passages carries a cache_salt derived from its organization
when the replica chosen is vLLM, so how fast an answer starts cannot say what
another tenant's documents held. Other engines are not sent the field.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 16:02:19 +02:00
1 parent 5ee8e74de1
commit 26630274f4
6 files changed
+108 -4

No files matched your search

+2 -4
View File
@@ -975,10 +975,8 @@ the model's context; added to the one leading system message (or a new
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
portal chat's Documents picker (collection, default or none; sources listed
under the answer; a header the page sends itself is dropped). Left:
`cache_salt` per organization (not sent: llama.cpp may refuse an unknown
field, and the router does not know the engine before choosing the
replica); returned passages
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
replicas only (chosen replica's engine). Left: returned passages
are not audited per request (the header and the kept request record them).
### F4. Adapter training (L, after F1–F3)