Decide engines per node, and let a node join without one it cannot run
engine_overrides give nodes picked by GPU class, id or labels their own engine versions; node add --disable-engine keeps one off a node; what vLLM cannot do on a node is a warning rather than a refused node; the CUDA requirement follows the version; a hand-named binary is no longer replaced by an installed one. Refs #2 (items 4 and 7). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
cba3e329fc
commit
e313f32696
21 files changed
+532
-62
No files matched your search
+39
-7
@@ -376,8 +376,10 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
|
||||
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
|
||||
Volta (7.0);
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -394,8 +396,38 @@ engines:
|
||||
```
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
|
||||
id, or by labels, every one given having to match — and lays engines over the
|
||||
cluster's for them, in order:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
|
||||
A node told nothing it can run of an engine joins without it: `llmgw doctor`
|
||||
and `node add` warn, and the node serves what it does run. To keep an engine
|
||||
off a node for good, whatever the control plane says, add the node with
|
||||
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
|
||||
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
|
||||
than `--ignore-checks`, which passes over every other problem as well.
|
||||
|
||||
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
|
||||
asks it which devices it sees, and a replica placed on GPUs is not ready until
|
||||
its process holds memory on each of them. A build missing a CUDA library fails
|
||||
there, naming the library, rather than serving on the CPU.
|
||||
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
@@ -1216,9 +1248,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
Reference in new issue
Block a user