Passages of one organization never share a vLLM prefix cache with another's

A request given passages carries a cache_salt derived from its organization
when the replica chosen is vLLM, so how fast an answer starts cannot say what
another tenant's documents held. Other engines are not sent the field.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 16:02:19 +02:00
1 parent 5ee8e74de1
commit 26630274f4
6 files changed
+108 -4

No files matched your search

+4
View File
@@ -316,6 +316,10 @@ Behaviour to know about:
- **Other APIs.** Requests through `/v1/responses` and `/v1/messages`
(Codex, Claude Code) work the same way, since they are answered as chat
completions.
- **Separate prefix caches.** On vLLM replicas, a request given passages
carries a `cache_salt` derived from its organization. One organization's
passages then never share cached blocks with another's, so how fast an
answer starts says nothing about what another tenant asked.
- **No answer cache.** Answers from knowledge bases are never served from
the answer cache, because two keys of one project can read different
collections.