Requests kept and replayed: keys requests per project with retention and a share, ctl request --body and --replay
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
841ee21c12
commit
32ff1dbd6f
22 files changed
+757
-10
No files matched your search
+2
-1
@@ -192,7 +192,8 @@ with anything secret taken out. `llmgw ctl audit` is the same, with filters.
|
||||
projects, with members, keys and limits. **A** adds an organization. On a
|
||||
row: add a team or project inside it, rename, move, limits for the level and
|
||||
for each of its members, its capacity quota, and for a project how long
|
||||
conversations and agent notes are kept. **Longest key lifetime…** caps how
|
||||
conversations and agent notes are kept, and whether its requests are
|
||||
(**Keep requests…**: for how long, and what share). **Longest key lifetime…** caps how
|
||||
long a key made there may live, and is the expiry of one made without any
|
||||
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
|
||||
**Remove** takes away only something empty.
|
||||
|
||||
+6
-1
@@ -51,6 +51,11 @@ limits attached to it. Passwords are hashed.
|
||||
|
||||
**Conversations and agent notes** — only for a project that asked. See below.
|
||||
|
||||
**Kept requests** — only for a project that asked (`llmgw keys requests`):
|
||||
what each request sent the model, after the guardrails' redaction and with
|
||||
any stored conversation in it, and the answer the client got, for the hours
|
||||
it said, all of them or a share. Never under a `no_store` policy.
|
||||
|
||||
**Batch files** — when a client uses the Batch API: the file of requests it
|
||||
uploaded (prompts) and the files the batch wrote (answers), in the store,
|
||||
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
|
||||
@@ -61,7 +66,7 @@ them.
|
||||
## What is not stored
|
||||
|
||||
Prompts. Completions. Anything the model was shown or produced, unless the
|
||||
project turned on conversations or notes, or sent a batch.
|
||||
project turned on conversations, notes or kept requests, or sent a batch.
|
||||
|
||||
The chat page in llmgw's own web interface keeps the conversation in a
|
||||
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
||||
|
||||
@@ -1130,6 +1130,29 @@ curl http://gateway:8080/v1/chat/completions \
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Keeping requests, and replaying them
|
||||
|
||||
To follow a request up — what exactly was sent, what came back — or to try
|
||||
real traffic on another model, a project can keep its requests:
|
||||
|
||||
```
|
||||
llmgw keys requests --project Acme/ml/train --retain-hours 72 # every one
|
||||
llmgw keys requests --project Acme/ml/train --retain-hours 168 --sample 0.05
|
||||
llmgw ctl request req_3f2a… --body # what was sent and answered
|
||||
llmgw ctl request req_3f2a… --replay qwen2.5-14b-instruct # the same, to another model
|
||||
llmgw keys requests --project Acme/ml/train --off
|
||||
```
|
||||
|
||||
- **What is kept:** the request as the model was sent it — after the
|
||||
guardrails' redaction, with any stored conversation spliced in — and the
|
||||
answer as the client got it, up to 128 KB of it. A share (`--sample`) is
|
||||
the same requests every time for the same ids.
|
||||
- **Replay** sends it as the gateway itself: past the limits, the budget and
|
||||
the guardrails, which judged it the first time, and charged to nobody.
|
||||
- **Never** for a project under a guardrails policy with `no_store`. Kept
|
||||
requests past their hours, or of a project turned off, are removed hourly.
|
||||
The console has it as **Keep requests…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
|
||||
Reference in new issue
Block a user