Give V100s the last vLLM built for them, not a build without their kernels
Read from vLLM's release pipeline: every x86 build from 0.20.0 has kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1 is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26. A V100 asked for a newer vLLM is refused, naming 0.19.1 for engine_overrides; the CUDA 12.9 fallback is for older drivers on Turing and newer. The cu129 wheel is found among the release's assets, since its manylinux tag differs between versions. Refs #2. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4f280a3546
commit
86e9a1cb83
7 files changed
+138
-61
No files matched your search
+16
-11
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
|
||||
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
|
||||
0.20 on. A node with an older driver gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
|
||||
Either way the install ends by computing something on the node's GPU, and a
|
||||
build that cannot fails to install, saying why;
|
||||
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
|
||||
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
|
||||
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
|
||||
it is installed, since llmgw's flags are checked against the current one;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -427,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
|
||||
engines: { vllm: "0.19.1" } # the last one that does
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
|
||||
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
|
||||
that version with `engine_overrides` (step 4) — its flags are not the ones
|
||||
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
Reference in new issue
Block a user