An external model takes api_key_files, each a file and an optional weight, in place of api_key_file. Each key is an endpoint of its own (external:MODEL#1, #2), so the router's scoring, retries and breakers work per key. A key answering 429 is passed over while its breaker is open. A key answering 401 is passed over too, and external_key_refused is sent, naming the endpoint and never the key; with a single key the 401 is still the client's. A key whose file cannot be read is left out, and usage records which key answered by its endpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
688b205604
commit
fe310576e7
14 files changed
+283
-15
No files matched your search
@@ -1323,6 +1323,26 @@ catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
A provider's limits are per key. With several keys, one answering 429 need not
|
||||
hold up the model: give them all, each with an optional weight (its share
|
||||
against the others'), in place of `api_key_file`:
|
||||
|
||||
```yaml
|
||||
external:
|
||||
url: https://api.openai.com
|
||||
model: gpt-4o-mini
|
||||
api_key_files:
|
||||
- {file: /etc/llmgw/secrets/openai-1.key, weight: 2}
|
||||
- {file: /etc/llmgw/secrets/openai-2.key}
|
||||
```
|
||||
|
||||
Each key is an endpoint of its own, `external:cloud-mini#1`, `#2`. A key that
|
||||
answers 429 is passed over while its breaker is open, and the request goes to
|
||||
another. One that answers 401 (revoked, expired, mistyped) is passed over too,
|
||||
and `external_key_refused` goes to the notification webhooks, naming the
|
||||
endpoint and never the key. The usage records say which key answered by the
|
||||
same name. A key whose file cannot be read is left out, and the others stay.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
Reference in new issue
Block a user