Serve a model from an external OpenAI-compatible provider
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d5f02c0a4b
commit
94eec557ad
16 files changed
+394
-12
No files matched your search
@@ -78,6 +78,9 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -95,6 +98,7 @@ rather than repeated.
|
||||
| `scaling` | `Scaling` | [Model](#model) |
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
@@ -616,6 +620,28 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
URL is the provider's base address, without /v1: https://api.openai.com.
|
||||
|
||||
### `model`
|
||||
|
||||
`string`
|
||||
|
||||
Model is what the provider calls it; requests are rewritten to it.
|
||||
|
||||
### `api_key_file`
|
||||
|
||||
`string`
|
||||
|
||||
APIKeyFile holds the provider's key, on the control plane. A file
|
||||
rather than the key itself, so the key is in neither the catalog, the
|
||||
store nor the audit log; empty for a provider that asks for none.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -765,6 +791,20 @@ Owner is the part of the tenancy tree whose capacity quota this model
|
||||
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
||||
held by the cluster, counted against nobody.
|
||||
|
||||
### `external`
|
||||
|
||||
`*External`
|
||||
|
||||
External is a model served by someone else — an OpenAI-compatible API
|
||||
outside the cluster — rather than by replicas placed here. Such a model
|
||||
has no profiles and nothing is placed for it; the router sends its
|
||||
requests to the provider. Most useful named in another model's
|
||||
fallbacks: used only when the cluster has no capacity left. Its
|
||||
prompts leave the cluster, so which keys may name it is worth an
|
||||
allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `fallbacks`
|
||||
|
||||
`[]string`
|
||||
|
||||
Reference in new issue
Block a user