Several keys for one outside provider

An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/admin-api.openapi.json

@@ -9054,6 +9054,13 @@
             "type": "string",
             "description": "APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none."
           },
+          "api_key_files": {
+            "type": "array",
+            "items": {
+              "$ref": "#/components/schemas/catalog.ExternalKey"
+            },
+            "description": "APIKeyFiles are several keys for the provider, in place of APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that answers 429 or 401 is passed over while its breaker is open and the request goes to another. A provider's limits are per key."
+          },
           "peer": {
             "type": "string",
             "description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
@@ -9072,6 +9079,20 @@
           }
         }
       },
+      "catalog.ExternalKey": {
+        "type": "object",
+        "description": "ExternalKey is one of a provider's keys.",
+        "properties": {
+          "file": {
+            "type": "string",
+            "description": "File holds the key, on the control plane, as APIKeyFile does."
+          },
+          "weight": {
+            "type": "number",
+            "description": "Weight is its share of the requests against the others'; zero is 1."
+          }
+        }
+      },
       "catalog.GPUNeed": {
         "type": "object",
         "description": "GPUNeed is what card a profile needs.",

docs/install.md

@@ -1323,6 +1323,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
 file cannot be read is not offered. Everything else applies as to any model:
 prices and budgets, rate limits, usage records, and key allowlists.
 
+A provider's limits are per key. With several keys, one answering 429 need not
+hold up the model: give them all, each with an optional weight (its share
+against the others'), in place of `api_key_file`:
+
+```yaml
+  external:
+    url: https://api.openai.com
+    model: gpt-4o-mini
+    api_key_files:
+      - {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
+      - {file: /etc/llmgw/secrets/openai-2.key}
+```
+
+Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
+answers 429 is passed over while its breaker is open, and the request goes to
+another. One that answers 401 (revoked, expired, mistyped) is passed over too,
+and `external_key_refused` goes to the notification webhooks, naming the
+endpoint and never the key. The usage records say which key answered by the
+same name. A key whose file cannot be read is left out, and the others stay.
+
 **Its prompts leave the cluster.** Decide who may send them: a key or project
 whose allowlist leaves the external model out cannot reach it, as a fallback or
 by name. The cluster page and `ctl models` mark it `external`.

docs/reference.md

@@ -105,10 +105,13 @@ rather than repeated.
 | `url` | `string` | [External](#external) |
 | `model` | `string` | [External](#external) |
 | `api_key_file` | `string` | [External](#external) |
+| `api_key_files` | `[]ExternalKey` | [External](#external) |
 | `peer` | `string` | [External](#external) |
 | `api` | `string` | [External](#external) |
 | `workflow` | `string` | [External](#external) |
 | `output` | `string` | [External](#external) |
+| `file` | `string` | [ExternalKey](#externalkey) |
+| `weight` | `float64` | [ExternalKey](#externalkey) |
 | `budgets` | `[]FacilityBudget` | [Facility](#facility) |
 | `margin` | `float64` | [Facility](#facility) |
 | `gpu_watts` | `float64` | [Facility](#facility) |
@@ -1103,6 +1106,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
 rather than the key itself, so the key is in neither the catalog, the
 store nor the audit log; empty for a provider that asks for none.
 
+### `api_key_files`
+
+`[]ExternalKey`
+
+APIKeyFiles are several keys for the provider, in place of
+APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
+answers 429 or 401 is passed over while its breaker is open and the
+request goes to another. A provider's limits are per key.
+
 ### `peer`
 
 `string`
@@ -1135,6 +1147,20 @@ plane.
 
 Output is what it makes: pictures, or meshes (see ComfyUI.Output).
 
+## ExternalKey
+
+### `file`
+
+`string`
+
+File holds the key, on the control plane, as APIKeyFile does.
+
+### `weight`
+
+`float64`
+
+Weight is its share of the requests against the others'; zero is 1.
+
 ## Facility
 
 ### `budgets`