A reused answer never carries another caller's values
With restore: true and the answer cache on, alice's "write to alice@example.com" and bob's "write to bob@example.com" redact to the same request, and bob was given the answer kept for alice, her address put back in it. A request with restore values is now never looked up in the cache nor kept in it. The similar-question cache compared the last message's text and blanked its parts: two pictures asked about in the same words were one question. A last message with any part that is not text is now reused only when the request is the same. Found in review; both tests fail without the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
03d3bcfd2d
commit
7a04248148
6 files changed
+87
-4
No files matched your search
+6
-4
@@ -2155,8 +2155,10 @@ llmgw keys cache --project Acme/ml/train --off
|
||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||
the request before the cache is looked in, and the output policy judges a
|
||||
reused answer as it would a new one. A request continuing a stored
|
||||
conversation, or under a `no_store` policy, is never answered from the
|
||||
cache nor kept in it.
|
||||
conversation, under a `no_store` policy, or whose redacted values are put
|
||||
back in its answer (`restore: true`) is never answered from the cache nor
|
||||
kept in it: redacted, two callers' requests read the same, and the answer
|
||||
has the first caller's values in it.
|
||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||
@@ -2196,8 +2198,8 @@ router:
|
||||
`router.cache.embedding_model`, against the questions already answered
|
||||
in the same conversation — the same system prompt, the same earlier turns,
|
||||
tools, `response_format` and sampling settings, the same model and
|
||||
project. A like question in another conversation, or after a tool's
|
||||
result, is not taken for this one. The exact cache is asked first, and an
|
||||
project. A like question in another conversation, after a tool's result,
|
||||
or with a picture, a sound or a file in it, is not taken for this one. The exact cache is asked first, and an
|
||||
exact hit embeds nothing.
|
||||
- **How alike:** the cosine similarity of the two embeddings, at least the
|
||||
project's `--similar`. 0.95 is strict; below 0.9 different questions start
|
||||
|
||||
Reference in new issue
Block a user