Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 21:00:01 +02:00
1 parent 889be70b8f
commit 841ee21c12
19 files changed
+1775 -10

No files matched your search

+36
View File
@@ -1130,6 +1130,42 @@ curl http://gateway:8080/v1/chat/completions \
100 values of each apart and the rest as `other`: a tag is the client's
to set, and every value is a series.
## Batches
Work that can wait — a night's classification, embeddings for a document
store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.
```
curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
-F purpose=batch -F file=@requests.jsonl
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
-d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"
```
Each line of the file is `{"custom_id": …, "method": "POST", "url":
"/v1/chat/completions", "body": {…}}`, at most 50 000 of them
(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`).
- **How it is served:** each line as a request of the key that made the
batch — its limits, its budget, its guardrails — in the lowest admission
band, behind every interactive request, `router.batch.concurrency` (4) at
a time. A rate limit or a model still loading is waited out; a spent
budget fails the lines left.
- **What comes back:** answers in an output file, failures in an error
file, both OpenAI's form, kept `router.batch.retain_days` (30). Each
request is a usage record tagged `batch=<id>`, so `llmgw ctl request`
finds it.
- **Who sees it:** files and batches belong to the key's project; every key
of the project sees them, no other does.
- **Surviving a restart:** every answered line is kept as it is answered,
and whichever gateway holds the batch's lease carries on from there; a
control plane that stops hands its batches to another within two minutes.
- **What it keeps:** a batch is prompts and answers stored until deleted
or past their retention — a project under a guardrails policy with
`no_store` is refused it.
## Conversations across models
Off everywhere until a project asks. llmgw otherwise stores no message content