Reusing answers: a response cache per project, exact match, opt-in with a TTL
llmgw keys cache --project P --ttl 1h [--any] | --off, and Cache answers… on the console's project menu. The same request — project, endpoint, canonical model and the body without stream, stream_options, user, metadata, store and service_tier, keys sorted — is answered from the gateway's memory for the TTL. Only reproducible answers (temperature 0 or a seed; embeddings, rerank, score) unless --any; only a whole 200 from the model asked for, not a fallback's, a canary's, or one a guardrail changed. Looked up after the limits and the input guardrails; the output policy judges a reused answer; no_store and stored conversations bypass it. Streamed answers are kept put back together and streamed back as chunks. x-llmgw-cache: hit|miss|bypass, Age on a hit, Cache-Control: no-cache or x-llmgw-cache: no to bypass. A hit is recorded with no tokens and no cost, marked cache_hit (migration 0027) and shown by ctl request. An LRU bounded by router.cache.max_bytes and max_entry_bytes; llmgw_cache_requests_total, llmgw_cache_bytes, llmgw_cache_entries. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
30f7f3474a
commit
44f3a9e31d
27 files changed
+2066
-11
No files matched your search
@@ -1419,6 +1419,8 @@ type RequestView struct {
|
||||
// What the client said about it: OpenAI's user field, and its tags.
|
||||
EndUser string `json:"end_user,omitempty"`
|
||||
Tags map[string]string `json:"tags,omitempty"`
|
||||
// CacheHit is an answer reused from the project's answer cache.
|
||||
CacheHit bool `json:"cache_hit,omitempty"`
|
||||
}
|
||||
|
||||
func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
@@ -1442,7 +1444,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user