Give each GPU node a vLLM build it can run, chosen on the cluster page
A node that cannot run the CUDA 13 build installs the same version's CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch; the install computes on the GPU before it counts. The build can be chosen per node — auto, cu13, cu129, none — on the cluster page, with ctl engine-build, or over the admin API; the control plane keeps the choice and tells the node at once. Refs #2. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2735c5a318
commit
4f280a3546
32 files changed
+1004
-71
No files matched your search
+19
-3
@@ -377,9 +377,12 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
|
||||
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
|
||||
Volta (7.0);
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -400,6 +403,19 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
|
||||
*CUDA 12.9*, or *none* — or from the command line:
|
||||
|
||||
```
|
||||
llmgw ctl engine-build gpu07 vllm cu129
|
||||
```
|
||||
|
||||
The choice is kept by the control plane and the node is told at once; it
|
||||
installs the build in the background, and the column shows how that stands.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
|
||||
Reference in new issue
Block a user