Choose each engine's build from the node's hardware, for every GPU generation
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m52s

Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 14:35:58 +02:00
1 parent 43d7c4eac1
commit dc483d8c68
15 files changed
+590 -395

No files matched your search

+18 -18
View File
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
| Engine | On | How |
|---|---|---|
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
@@ -376,17 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
0.20 on. A node with an older driver gets **the same version built for CUDA
12.9** instead, from vLLM's GitHub release (checked against the digest
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
Either way the install ends by computing something on the node's GPU, and a
build that cannot fails to install, saying why;
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
it is installed, since llmgw's flags are checked against the current one;
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
kernels — and the cluster page shows it running that version; run `llmgw
ctl engine-check vllm` there once, since llmgw's flags are checked against
the current one. Pascal and older run no vLLM. Whatever is chosen, the
install ends by computing something on the GPU; a build that cannot is
passed over for the next;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -431,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
engines: { vllm: "0.19.1" } # the last one that does
- gpu_class: a10g # pin a class to a version you have tested
engines: { vllm: "0.29.0" }
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1273,10 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
that version with `engine_overrides` (step 4) — its flags are not the ones
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
no build from 0.20 on has Volta kernels, and the node picks the newest that
has. llmgw's flags are checked against the current version, so run `llmgw
ctl engine-check vllm` on a V100 node once.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the