Serve a model from an external OpenAI-compatible provider

A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 23:29:01 +02:00
1 parent d5f02c0a4b
commit 94eec557ad
16 files changed
+394 -12

No files matched your search

+34
View File
@@ -671,6 +671,40 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
which names every candidate and why each was rejected, rather than leaving you
to infer it.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
another gateway — rather than replicas here. Nothing is placed for it; the
router sends its requests on, with the provider's key and the provider's name
for the model:
```yaml
- name: cloud-mini
source: external
external:
url: https://api.openai.com # without /v1
model: gpt-4o-mini # what the provider calls it
api_key_file: /etc/llmgw/secrets/openai.key
```
Named in another model's `fallbacks`, it is used only when the cluster has no
capacity left for that model:
```yaml
- name: qwen2.5-coder-7b-instruct
fallbacks: [cloud-mini]
...
```
The key is read from its file on the control plane, never stored in the
catalog, and a key rotated in the file is used within seconds. A model whose key
file cannot be read is not offered. Everything else applies as to any model:
prices and budgets, rate limits, usage records, and key allowlists.
**Its prompts leave the cluster.** Decide who may send them: a key or project
whose allowlist leaves the external model out cannot reach it, as a fallback or
by name. The cluster page and `ctl models` mark it `external`.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
+40
View File
@@ -78,6 +78,9 @@ rather than repeated.
| `pool` | `string` | [Endpoint](#endpoint) |
| `weight` | `float64` | [Endpoint](#endpoint) |
| `priority` | `int` | [Endpoint](#endpoint) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
@@ -95,6 +98,7 @@ rather than repeated.
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
@@ -616,6 +620,28 @@ called. Empty means vLLM's names.
`int`
## External
### `url`
`string`
URL is the provider's base address, without /v1: https://api.openai.com.
### `model`
`string`
Model is what the provider calls it; requests are rewritten to it.
### `api_key_file`
`string`
APIKeyFile holds the provider's key, on the control plane. A file
rather than the key itself, so the key is in neither the catalog, the
store nor the audit log; empty for a provider that asks for none.
## Landing
### `contact`
@@ -765,6 +791,20 @@ Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `external`
`*External`
External is a model served by someone else — an OpenAI-compatible API
outside the cluster — rather than by replicas placed here. Such a model
has no profiles and nothing is placed for it; the router sends its
requests to the provider. Most useful named in another model's
fallbacks: used only when the cluster has no capacity left. Its
prompts leave the cluster, so which keys may name it is worth an
allowlist.
Takes a [External](#external) block.
### `fallbacks`
`[]string`