Several keys for one outside provider
CI / test (push) Successful in 8m52s

An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-17 08:56:30 +02:00
1 parent 688b205604
commit fe310576e7
14 files changed
+283 -15

No files matched your search

+21
View File
@@ -9054,6 +9054,13 @@
"type": "string",
"description": "APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none."
},
"api_key_files": {
"type": "array",
"items": {
"$ref": "#/components/schemas/catalog.ExternalKey"
},
"description": "APIKeyFiles are several keys for the provider, in place of APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that answers 429 or 401 is passed over while its breaker is open and the request goes to another. A provider's limits are per key."
},
"peer": {
"type": "string",
"description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
@@ -9072,6 +9079,20 @@
}
}
},
"catalog.ExternalKey": {
"type": "object",
"description": "ExternalKey is one of a provider's keys.",
"properties": {
"file": {
"type": "string",
"description": "File holds the key, on the control plane, as APIKeyFile does."
},
"weight": {
"type": "number",
"description": "Weight is its share of the requests against the others'; zero is 1."
}
}
},
"catalog.GPUNeed": {
"type": "object",
"description": "GPUNeed is what card a profile needs.",