Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 21:00:01 +02:00
1 parent 889be70b8f
commit 841ee21c12
19 files changed
+1775 -10

No files matched your search

+8 -1
View File
@@ -51,10 +51,17 @@ limits attached to it. Passwords are hashed.
**Conversations and agent notes** — only for a project that asked. See below.
**Batch files** — when a client uses the Batch API: the file of requests it
uploaded (prompts) and the files the batch wrote (answers), in the store,
until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
`router.batch.retain_days` (30) pass. Only keys of the project that made
them can read them, and a project under a `no_store` policy cannot make
them.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations or notes.
project turned on conversations or notes, or sent a batch.
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
+36
View File
@@ -1130,6 +1130,42 @@ curl http://gateway:8080/v1/chat/completions \
100 values of each apart and the rest as `other`: a tag is the client's
to set, and every value is a series.
## Batches
Work that can wait — a night's classification, embeddings for a document
store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.
```
curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
-F purpose=batch -F file=@requests.jsonl
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
-d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"
```
Each line of the file is `{"custom_id": …, "method": "POST", "url":
"/v1/chat/completions", "body": {…}}`, at most 50 000 of them
(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`).
- **How it is served:** each line as a request of the key that made the
batch — its limits, its budget, its guardrails — in the lowest admission
band, behind every interactive request, `router.batch.concurrency` (4) at
a time. A rate limit or a model still loading is waited out; a spent
budget fails the lines left.
- **What comes back:** answers in an output file, failures in an error
file, both OpenAI's form, kept `router.batch.retain_days` (30). Each
request is a usage record tagged `batch=<id>`, so `llmgw ctl request`
finds it.
- **Who sees it:** files and batches belong to the key's project; every key
of the project sees them, no other does.
- **Surviving a restart:** every answered line is kept as it is answered,
and whichever gateway holds the batch's lease carries on from there; a
control plane that stops hands its batches to another within two minutes.
- **What it keeps:** a batch is prompts and answers stored until deleted
or past their retention — a project under a guardrails policy with
`no_store` is refused it.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content
+44
View File
@@ -43,6 +43,10 @@ rather than repeated.
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `max_file_bytes` | `int64` | [Batch](#batch) |
| `max_lines` | `int` | [Batch](#batch) |
| `concurrency` | `int` | [Batch](#batch) |
| `retain_days` | `int` | [Batch](#batch) |
| `model` | `string` | [Canary](#canary) |
| `percent` | `int` | [Canary](#canary) |
| `min_requests` | `int` | [Canary](#canary) |
@@ -198,6 +202,7 @@ rather than repeated.
| `tenant_label` | `string` | [Router](#router) |
| `tag_labels` | `[]string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `batch` | `Batch` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
@@ -517,6 +522,35 @@ five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Batch
### `max_file_bytes`
`int64`
MaxFileBytes bounds one uploaded file; 100 MB unless set.
### `max_lines`
`int`
MaxLines bounds the requests in one batch; 50 000 unless set.
### `concurrency`
`int`
Concurrency is how many of one batch's requests are in flight at
once; 4 unless set. They queue in the lowest admission band, behind
every interactive request.
### `retain_days`
`int`
RetainDays is how long a batch's answer and error files are kept;
30 unless set. Uploaded files are kept until deleted.
## Canary
### `model`
@@ -1622,6 +1656,16 @@ a series. The first 100 values of each are counted apart, the rest as
MaxBodyBytes bounds the request body the router will read.
### `batch`
`Batch`
Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of
requests served at the lowest priority, its answers written to
another. Needs the store.
Takes a [Batch](#batch) block.
### `trusted_proxies`
`[]string`