Several keys for one outside provider
An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
@@ -105,10 +105,13 @@ rather than repeated.
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `api_key_files` | `[]ExternalKey` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `file` | `string` | [ExternalKey](#externalkey) |
|
||||
| `weight` | `float64` | [ExternalKey](#externalkey) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -1103,6 +1106,15 @@ APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
### `api_key_files`
|
||||
|
||||
`[]ExternalKey`
|
||||
|
||||
APIKeyFiles are several keys for the provider, in place of
|
||||
APIKeyFile: one endpoint each (external:MODEL#1, #2…), so a key that
|
||||
answers 429 or 401 is passed over while its breaker is open and the
|
||||
request goes to another. A provider's limits are per key.
|
||||
|
||||
### `peer`
|
||||
|
||||
`string`
|
||||
@@ -1135,6 +1147,20 @@ plane.
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## ExternalKey
|
||||
|
||||
### `file`
|
||||
|
||||
`string`
|
||||
|
||||
File holds the key, on the control plane, as APIKeyFile does.
|
||||
|
||||
### `weight`
|
||||
|
||||
`float64`
|
||||
|
||||
Weight is its share of the requests against the others'; zero is 1.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
|
||||
Reference in new issue
Block a user