Decide engines per node, and let a node join without one it cannot run
engine_overrides give nodes picked by GPU class, id or labels their own engine versions; node add --disable-engine keeps one off a node; what vLLM cannot do on a node is a warning rather than a refused node; the CUDA requirement follows the version; a hand-named binary is no longer replaced by an installed one. Refs #2 (items 4 and 7). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
cba3e329fc
commit
e313f32696
21 files changed
+532
-62
No files matched your search
@@ -81,6 +81,10 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -183,6 +187,7 @@ rather than repeated.
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -645,6 +650,32 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## EngineOverride
|
||||
|
||||
### `gpu_class`
|
||||
|
||||
`string`
|
||||
|
||||
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes picks nodes by id.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels picks nodes carrying all of these.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `engine_overrides`
|
||||
|
||||
`[]EngineOverride`
|
||||
|
||||
EngineOverrides give some nodes different engines from the rest: a
|
||||
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
engines over the cluster's, in order, later ones last; an empty version
|
||||
removes that engine from the nodes it picks.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user