Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
889be70b8f
commit
841ee21c12
19 files changed
+1775
-10
No files matched your search
+8
-1
@@ -51,10 +51,17 @@ limits attached to it. Passwords are hashed.
|
||||
|
||||
**Conversations and agent notes** — only for a project that asked. See below.
|
||||
|
||||
**Batch files** — when a client uses the Batch API: the file of requests it
|
||||
uploaded (prompts) and the files the batch wrote (answers), in the store,
|
||||
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
|
||||
`router.batch.retain_days` (30) pass. Only keys of the project that made
|
||||
them can read them, and a project under a `no_store` policy cannot make
|
||||
them.
|
||||
|
||||
## What is not stored
|
||||
|
||||
Prompts. Completions. Anything the model was shown or produced, unless the
|
||||
project turned on conversations or notes.
|
||||
project turned on conversations or notes, or sent a batch.
|
||||
|
||||
The chat page in llmgw's own web interface keeps the conversation in a
|
||||
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
||||
|
||||
@@ -1130,6 +1130,42 @@ curl http://gateway:8080/v1/chat/completions \
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-F purpose=batch -F file=@requests.jsonl
|
||||
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
|
||||
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
Each line of the file is `{"custom_id": …, "method": "POST", "url":
|
||||
"/v1/chat/completions", "body": {…}}`, at most 50 000 of them
|
||||
(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`).
|
||||
|
||||
- **How it is served:** each line as a request of the key that made the
|
||||
batch — its limits, its budget, its guardrails — in the lowest admission
|
||||
band, behind every interactive request, `router.batch.concurrency` (4) at
|
||||
a time. A rate limit or a model still loading is waited out; a spent
|
||||
budget fails the lines left.
|
||||
- **What comes back:** answers in an output file, failures in an error
|
||||
file, both OpenAI's form, kept `router.batch.retain_days` (30). Each
|
||||
request is a usage record tagged `batch=<id>`, so `llmgw ctl request`
|
||||
finds it.
|
||||
- **Who sees it:** files and batches belong to the key's project; every key
|
||||
of the project sees them, no other does.
|
||||
- **Surviving a restart:** every answered line is kept as it is answered,
|
||||
and whichever gateway holds the batch's lease carries on from there; a
|
||||
control plane that stops hands its batches to another within two minutes.
|
||||
- **What it keeps:** a batch is prompts and answers stored until deleted
|
||||
or past their retention — a project under a guardrails policy with
|
||||
`no_store` is refused it.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
@@ -43,6 +43,10 @@ rather than repeated.
|
||||
| `batch_size` | `int` | [Auth](#auth) |
|
||||
| `buffer_size` | `int` | [Auth](#auth) |
|
||||
| `usage_retention` | `time.Duration` | [Auth](#auth) |
|
||||
| `max_file_bytes` | `int64` | [Batch](#batch) |
|
||||
| `max_lines` | `int` | [Batch](#batch) |
|
||||
| `concurrency` | `int` | [Batch](#batch) |
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -198,6 +202,7 @@ rather than repeated.
|
||||
| `tenant_label` | `string` | [Router](#router) |
|
||||
| `tag_labels` | `[]string` | [Router](#router) |
|
||||
| `max_body_bytes` | `int64` | [Router](#router) |
|
||||
| `batch` | `Batch` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
| `scoring` | `Scoring` | [Router](#router) |
|
||||
@@ -517,6 +522,35 @@ five-minute sums kept beside the records, which are not pruned, so this
|
||||
bounds the table that grows by a row a request without changing any
|
||||
figure a page shows. At least a day when set.
|
||||
|
||||
## Batch
|
||||
|
||||
### `max_file_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
MaxFileBytes bounds one uploaded file; 100 MB unless set.
|
||||
|
||||
### `max_lines`
|
||||
|
||||
`int`
|
||||
|
||||
MaxLines bounds the requests in one batch; 50 000 unless set.
|
||||
|
||||
### `concurrency`
|
||||
|
||||
`int`
|
||||
|
||||
Concurrency is how many of one batch's requests are in flight at
|
||||
once; 4 unless set. They queue in the lowest admission band, behind
|
||||
every interactive request.
|
||||
|
||||
### `retain_days`
|
||||
|
||||
`int`
|
||||
|
||||
RetainDays is how long a batch's answer and error files are kept;
|
||||
30 unless set. Uploaded files are kept until deleted.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -1622,6 +1656,16 @@ a series. The first 100 values of each are counted apart, the rest as
|
||||
|
||||
MaxBodyBytes bounds the request body the router will read.
|
||||
|
||||
### `batch`
|
||||
|
||||
`Batch`
|
||||
|
||||
Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of
|
||||
requests served at the lowest priority, its answers written to
|
||||
another. Needs the store.
|
||||
|
||||
Takes a [Batch](#batch) block.
|
||||
|
||||
### `trusted_proxies`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user