An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
@@ -28,6 +28,11 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
### Added
|
||||
|
||||
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
|
||||
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
|
||||
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
|
||||
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
|
||||
|
||||
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
|
||||
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
|
||||
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
|
||||
|
||||
Reference in new issue
Block a user