Several keys for one outside provider
An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
+8
-2
@@ -25,7 +25,7 @@ What is left, most needed first. The detail is in the item each names.
|
||||
judge by itself.
|
||||
3. ~~**I3**~~ — done 2026-10-16: `ctl bench overhead`, figures in the
|
||||
README.
|
||||
4. **I4** — several keys for one outside provider. (S)
|
||||
4. ~~**I4**~~ — done 2026-10-17: several keys for one outside provider.
|
||||
5. **J2** — a judge the leader adds by itself is sized as init's is. (S)
|
||||
6. **J3** — HA setup puts the judges on nodes of their own. (S)
|
||||
7. **J8** — a judge budget that fits the machine, measured at install. (S)
|
||||
@@ -1547,7 +1547,13 @@ answered with a guess.
|
||||
- Done when: the README states the overhead, with how it was measured, and
|
||||
CI runs the short version so a change that doubles it is seen.
|
||||
|
||||
### I4. Several keys for one outside provider (S)
|
||||
### I4. Several keys for one outside provider (S) — done 2026-10-17
|
||||
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
|
||||
each in `control.externalEndpoints`, 401 retried on another key and
|
||||
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
|
||||
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
|
||||
original item follows.
|
||||
|
||||
An external model has one `api_key_file` (`internal/control/control.go`,
|
||||
where external endpoints are made). A provider's limit is per key; one key
|
||||
answering 429 makes the whole fallback answer 429.
|
||||
|
||||
Reference in new issue
Block a user