Several keys for one outside provider
CI / test (push) Successful in 8m52s

An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-17 08:56:30 +02:00
1 parent 688b205604
commit fe310576e7
14 files changed
+283 -15

No files matched your search

+5
View File
@@ -28,6 +28,11 @@ APIs change between minors without a deprecation period.
### Added
- **Several keys for one outside provider.** An external model takes `api_key_files`, each with an optional weight, in
place of `api_key_file`. Each key is an endpoint of its own (`external:MODEL#1`, `#2`), so a key answering 429 is
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
`external_key_refused` is sent, naming the endpoint, never the key. Usage records which key answered by that name.
- **`llmgw ctl bench overhead`**: what the gateway adds to a request, measured against an engine that answers at
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.