Serve a model from an external OpenAI-compatible provider
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d5f02c0a4b
commit
94eec557ad
16 files changed
+394
-12
No files matched your search
@@ -671,6 +671,40 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
|
||||
which names every candidate and why each was rejected, rather than leaving you
|
||||
to infer it.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
another gateway — rather than replicas here. Nothing is placed for it; the
|
||||
router sends its requests on, with the provider's key and the provider's name
|
||||
for the model:
|
||||
|
||||
```yaml
|
||||
- name: cloud-mini
|
||||
source: external
|
||||
external:
|
||||
url: https://api.openai.com # without /v1
|
||||
model: gpt-4o-mini # what the provider calls it
|
||||
api_key_file: /etc/llmgw/secrets/openai.key
|
||||
```
|
||||
|
||||
Named in another model's `fallbacks`, it is used only when the cluster has no
|
||||
capacity left for that model:
|
||||
|
||||
```yaml
|
||||
- name: qwen2.5-coder-7b-instruct
|
||||
fallbacks: [cloud-mini]
|
||||
...
|
||||
```
|
||||
|
||||
The key is read from its file on the control plane, never stored in the
|
||||
catalog, and a key rotated in the file is used within seconds. A model whose key
|
||||
file cannot be read is not offered. Everything else applies as to any model:
|
||||
prices and budgets, rate limits, usage records, and key allowlists.
|
||||
|
||||
**Its prompts leave the cluster.** Decide who may send them: a key or project
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
|
||||
Reference in new issue
Block a user