A reused answer never carries another caller's values

With restore: true and the answer cache on, alice's "write to
alice@example.com" and bob's "write to bob@example.com" redact to the
same request, and bob was given the answer kept for alice, her address put
back in it. A request with restore values is now never looked up in the
cache nor kept in it.

The similar-question cache compared the last message's text and blanked
its parts: two pictures asked about in the same words were one question.
A last message with any part that is not text is now reused only when the
request is the same.

Found in review; both tests fail without the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-28 22:03:32 +01:00
1 parent 03d3bcfd2d
commit 7a04248148
6 files changed
+87 -4

No files matched your search

+6 -4
View File
@@ -2155,8 +2155,10 @@ llmgw keys cache --project Acme/ml/train --off
- **Policies still apply:** the limits are checked and the guardrails judge
the request before the cache is looked in, and the output policy judges a
reused answer as it would a new one. A request continuing a stored
conversation, or under a `no_store` policy, is never answered from the
cache nor kept in it.
conversation, under a `no_store` policy, or whose redacted values are put
back in its answer (`restore: true`) is never answered from the cache nor
kept in it: redacted, two callers' requests read the same, and the answer
has the first caller's values in it.
- **What a hit costs:** nothing. It is recorded as a request — status 200,
the model, the key, the rate limit's count — with no tokens and no cost,
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
@@ -2196,8 +2198,8 @@ router:
`router.cache.embedding_model`, against the questions already answered
in the same conversation — the same system prompt, the same earlier turns,
tools, `response_format` and sampling settings, the same model and
project. A like question in another conversation, or after a tool's
result, is not taken for this one. The exact cache is asked first, and an
project. A like question in another conversation, after a tool's result,
or with a picture, a sound or a file in it, is not taken for this one. The exact cache is asked first, and an
exact hit embeds nothing.
- **How alike:** the cosine similarity of the two embeddings, at least the
project's `--similar`. 0.95 is strict; below 0.9 different questions start