Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
841ee21c12
commit
32ff1dbd6f
22 files changed
+757
-10
No files matched your search
@@ -1130,6 +1130,29 @@ curl http://gateway:8080/v1/chat/completions \
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Keeping requests, and replaying them
|
||||
|
||||
To follow a request up — what exactly was sent, what came back — or to try
|
||||
real traffic on another model, a project can keep its requests:
|
||||
|
||||
```
|
||||
llmgw keys requests --project Acme/ml/train --retain-hours 72 # every one
|
||||
llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05
|
||||
llmgw ctl request req_3f2a… --body # what was sent and answered
|
||||
llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct # the same, to another model
|
||||
llmgw keys requests --project Acme/ml/train --off
|
||||
```
|
||||
|
||||
- **What is kept:** the request as the model was sent it — after the
|
||||
guardrails' redaction, with any stored conversation spliced in — and the
|
||||
answer as the client got it, up to 128 KB of it. A share (`--sample`) is
|
||||
the same requests every time for the same ids.
|
||||
- **Replay** sends it as the gateway itself: past the limits, the budget and
|
||||
the guardrails, which judged it the first time, and charged to nobody.
|
||||
- **Never** for a project under a guardrails policy with `no_store`. Kept
|
||||
requests past their hours, or of a project turned off, are removed hourly.
|
||||
The console has it as **Keep requests…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
|
||||
Reference in new issue
Block a user