Batches: OpenAI's Files and Batch APIs, served at the lowest priority under a store lease, each line a request of the key that made it
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
889be70b8f
commit
841ee21c12
19 files changed
+1775
-10
No files matched your search
@@ -1130,6 +1130,42 @@ curl http://gateway:8080/v1/chat/completions \
|
||||
100 values of each apart and the rest as `other`: a tag is the client's
|
||||
to set, and every value is a series.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
store — goes as a batch: OpenAI's Batch API, so its clients work unchanged.
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/files -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-F purpose=batch -F file=@requests.jsonl
|
||||
curl http://gateway:8080/v1/batches -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-d '{"input_file_id": "file-…", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
|
||||
curl http://gateway:8080/v1/batches/batch_… -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
Each line of the file is `{"custom_id": …, "method": "POST", "url":
|
||||
"/v1/chat/completions", "body": {…}}`, at most 50 000 of them
|
||||
(`router.batch.max_lines`) in at most 100 MB (`router.batch.max_file_bytes`).
|
||||
|
||||
- **How it is served:** each line as a request of the key that made the
|
||||
batch — its limits, its budget, its guardrails — in the lowest admission
|
||||
band, behind every interactive request, `router.batch.concurrency` (4) at
|
||||
a time. A rate limit or a model still loading is waited out; a spent
|
||||
budget fails the lines left.
|
||||
- **What comes back:** answers in an output file, failures in an error
|
||||
file, both OpenAI's form, kept `router.batch.retain_days` (30). Each
|
||||
request is a usage record tagged `batch=<id>`, so `llmgw ctl request`
|
||||
finds it.
|
||||
- **Who sees it:** files and batches belong to the key's project; every key
|
||||
of the project sees them, no other does.
|
||||
- **Surviving a restart:** every answered line is kept as it is answered,
|
||||
and whichever gateway holds the batch's lease carries on from there; a
|
||||
control plane that stops hands its batches to another within two minutes.
|
||||
- **What it keeps:** a batch is prompts and answers stored until deleted
|
||||
or past their retention — a project under a guardrails policy with
|
||||
`no_store` is refused it.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
Reference in new issue
Block a user