Decide engines per node, and let a node join without one it cannot run

engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.

Refs #2 (items 4 and 7).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 12:07:25 +02:00
1 parent cba3e329fc
commit e313f32696
21 files changed
+532 -62

No files matched your search

+39 -7
View File
@@ -376,8 +376,10 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
vLLM is built for CUDA 13. A V100 cannot run it;
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
Volta (7.0);
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -394,8 +396,38 @@ engines:
```
A changed version reaches every node when it next connects. To use an engine
you installed yourself instead, set its version to `""` and name its path in
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
#### Different GPUs, different engines
A cluster of two GPU generations may need two builds of an engine, or one on
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
id, or by labels, every one given having to match — and lays engines over the
cluster's for them, in order:
```yaml
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
A node told nothing it can run of an engine joins without it: `llmgw doctor`
and `node add` warn, and the node serves what it does run. To keep an engine
off a node for good, whatever the control plane says, add the node with
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
than `--ignore-checks`, which passes over every other problem as well.
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
asks it which devices it sees, and a replica placed on GPUs is not ready until
its process holds memory on each of them. A build missing a CUDA library fails
there, naming the library, rather than serving on the CPU.
After changing a version, check it still takes the flags llmgw builds for it,
which drift between releases:
@@ -1216,9 +1248,9 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
install a vLLM built for your CUDA yourself.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the