An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
@@ -9054,6 +9054,13 @@
|
||||
"type": "string",
|
||||
"description": "APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none."
|
||||
},
|
||||
"api_key_files": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.ExternalKey"
|
||||
},
|
||||
"description": "APIKeyFiles are several keys for the provider, in place of APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that answers 429 or 401 is passed over while its breaker is open and the request goes to another. A provider's limits are per key."
|
||||
},
|
||||
"peer": {
|
||||
"type": "string",
|
||||
"description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
|
||||
@@ -9072,6 +9079,20 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.ExternalKey": {
|
||||
"type": "object",
|
||||
"description": "ExternalKey is one of a provider's keys.",
|
||||
"properties": {
|
||||
"file": {
|
||||
"type": "string",
|
||||
"description": "File holds the key, on the control plane, as APIKeyFile does."
|
||||
},
|
||||
"weight": {
|
||||
"type": "number",
|
||||
"description": "Weight is its share of the requests against the others'; zero is 1."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.GPUNeed": {
|
||||
"type": "object",
|
||||
"description": "GPUNeed is what card a profile needs.",
|
||||
|
||||
Reference in new issue
Block a user