Decide engines per node, and let a node join without one it cannot run

engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.

Refs #2 (items 4 and 7).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 12:07:25 +02:00
1 parent cba3e329fc
commit e313f32696
21 files changed
+532 -62

No files matched your search

+41
View File
@@ -81,6 +81,10 @@ rather than repeated.
| `pool` | `string` | [Endpoint](#endpoint) |
| `weight` | `float64` | [Endpoint](#endpoint) |
| `priority` | `int` | [Endpoint](#endpoint) |
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
@@ -183,6 +187,7 @@ rather than repeated.
| `tracing` | `Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
@@ -645,6 +650,32 @@ called. Empty means vLLM's names.
`int`
## EngineOverride
### `gpu_class`
`string`
GPUClass picks nodes with a GPU of this class ("v100-32g").
### `nodes`
`[]string`
Nodes picks nodes by id.
### `labels`
`map[string]string`
Labels picks nodes carrying all of these.
### `engines`
`map[string]string`
Engines are laid over the cluster's: a version, or "" for none.
## External
### `url`
@@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `engine_overrides`
`[]EngineOverride`
EngineOverrides give some nodes different engines from the rest: a
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
all on nodes meant only for llama.cpp. Each picks nodes and lays its
engines over the cluster's, in order, later ones last; an empty version
removes that engine from the nodes it picks.
### `backup_dir`
`string`