Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 22:51:21 +02:00
1 parent 841ee21c12
commit 32ff1dbd6f
22 files changed
+757 -10

No files matched your search

+2 -1
View File
@@ -192,7 +192,8 @@ with anything secret taken out. `llmgw ctl audit` is the same, with filters.
projects, with members, keys and limits. **A** adds an organization. On a
row: add a team or project inside it, rename, move, limits for the level and
for each of its members, its capacity quota, and for a project how long
conversations and agent notes are kept. **Longest key lifetime…** caps how
conversations and agent notes are kept, and whether its requests are
(**Keep requests…**: for how long, and what share). **Longest key lifetime…** caps how
long a key made there may live, and is the expiry of one made without any
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
**Remove** takes away only something empty.
+6 -1
View File
@@ -51,6 +51,11 @@ limits attached to it. Passwords are hashed.
**Conversations and agent notes** — only for a project that asked. See below.
**Kept requests** — only for a project that asked (`llmgw keys requests`):
what each request sent the model, after the guardrails' redaction and with
any stored conversation in it, and the answer the client got, for the hours
it said, all of them or a share. Never under a `no_store` policy.
**Batch files** — when a client uses the Batch API: the file of requests it
uploaded (prompts) and the files the batch wrote (answers), in the store,
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
@@ -61,7 +66,7 @@ them.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations or notes, or sent a batch.
project turned on conversations, notes or kept requests, or sent a batch.
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
+23
View File
@@ -1130,6 +1130,29 @@ curl http://gateway:8080/v1/chat/completions \
100 values of each apart and the rest as `other`: a tag is the client's
to set, and every value is a series.
## Keeping requests, and replaying them
To follow a request up — what exactly was sent, what came back — or to try
real traffic on another model, a project can keep its requests:
```
llmgw keys requests --project Acme/ml/train --retain-hours 72 # every one
llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05
llmgw ctl request req_3f2a… --body # what was sent and answered
llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct # the same, to another model
llmgw keys requests --project Acme/ml/train --off
```
- **What is kept:** the request as the model was sent it — after the
guardrails' redaction, with any stored conversation spliced in — and the
answer as the client got it, up to 128 KB of it. A share (`--sample`) is
the same requests every time for the same ids.
- **Replay** sends it as the gateway itself: past the limits, the budget and
the guardrails, which judged it the first time, and charged to nobody.
- **Never** for a project under a guardrails policy with `no_store`. Kept
requests past their hours, or of a project turned off, are removed hourly.
The console has it as **Keep requests…** on a project.
## Batches
Work that can wait — a night's classification, embeddings for a document