Several keys for one outside provider

An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-17 08:56:30 +02:00
1 parent 688b205604
commit fe310576e7
14 files changed
+283 -15

No files matched your search

+8 -2
View File
@@ -25,7 +25,7 @@ What is left, most needed first. The detail is in the item each names.
judge by itself.
3. ~~**I3**~~ — done 2026-10-16: `ctl bench overhead`, figures in the
README.
4. **I4** — several keys for one outside provider. (S)
4. ~~**I4**~~ — done 2026-10-17: several keys for one outside provider.
5. **J2** — a judge the leader adds by itself is sized as init's is. (S)
6. **J3** — HA setup puts the judges on nodes of their own. (S)
7. **J8** — a judge budget that fits the machine, measured at install. (S)
@@ -1547,7 +1547,13 @@ answered with a guess.
- Done when: the README states the overhead, with how it was measured, and
CI runs the short version so a change that doubles it is seen.
### I4. Several keys for one outside provider (S)
### I4. Several keys for one outside provider (S) — done 2026-10-17
`external.api_key_files` (`catalog.ExternalKey`: file, weight), one endpoint
each in `control.externalEndpoints`, 401 retried on another key and
`external_key_refused` notified (`router.keyRefused`). With one key a 401 is
still the client's. `TestAKeyAtItsLimitOrRefusedIsPassedOverForAnother`. The
original item follows.
An external model has one `api_key_file` (`internal/control/control.go`,
where external endpoints are made). A provider's limit is per key; one key
answering 429 makes the whole fallback answer 429.