Passages of one organization never share a vLLM prefix cache with another's
A request given passages carries a cache_salt derived from its organization when the replica chosen is vLLM, so how fast an answer starts cannot say what another tenant's documents held. Other engines are not sent the field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5ee8e74de1
commit
26630274f4
6 files changed
+108
-4
No files matched your search
@@ -316,6 +316,10 @@ Behaviour to know about:
|
||||
- **Other APIs.** Requests through `/v1/responses` and `/v1/messages`
|
||||
(Codex, Claude Code) work the same way, since they are answered as chat
|
||||
completions.
|
||||
- **Separate prefix caches.** On vLLM replicas, a request given passages
|
||||
carries a `cache_salt` derived from its organization. One organization's
|
||||
passages then never share cached blocks with another's, so how fast an
|
||||
answer starts says nothing about what another tenant asked.
|
||||
- **No answer cache.** Answers from knowledge bases are never served from
|
||||
the answer cache, because two keys of one project can read different
|
||||
collections.
|
||||
|
||||
Reference in new issue
Block a user