Passages of one organization never share a vLLM prefix cache with another's
A request given passages carries a cache_salt derived from its organization when the replica chosen is vLLM, so how fast an answer starts cannot say what another tenant's documents held. Other engines are not sent the field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5ee8e74de1
commit
26630274f4
6 files changed
+108
-4
No files matched your search
+2
-4
@@ -975,10 +975,8 @@ the model's context; added to the one leading system message (or a new
|
||||
one); `x-llmgw-sources`; before guardInput; never cached; metrics. Default collections built (kb_defaults on org/team/project/key, narrowest
|
||||
wins, 30 s cache, `ctl kb default`, console action; never refuse); the
|
||||
portal chat's Documents picker (collection, default or none; sources listed
|
||||
under the answer; a header the page sends itself is dropped). Left:
|
||||
`cache_salt` per organization (not sent: llama.cpp may refuse an unknown
|
||||
field, and the router does not know the engine before choosing the
|
||||
replica); returned passages
|
||||
under the answer; a header the page sends itself is dropped). `cache_salt` per organization on requests given passages, sent to vLLM
|
||||
replicas only (chosen replica's engine). Left: returned passages
|
||||
are not audited per request (the header and the kept request record them).
|
||||
|
||||
### F4. Adapter training (L, after F1–F3)
|
||||
|
||||
Reference in new issue
Block a user