Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 86e9a1cb83 Give V100s the last vLLM built for them, not a build without their kernels
Release / build (push) Successful in 1m13s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m47s
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:04:45 +02:00
7 changed files with 138 additions and 61 deletions

No files matched your search

+19 -12
View File
@@ -126,11 +126,19 @@ leaving the model with nothing. Now:
### Every GPU node gets a vLLM it can run, chosen on the cluster page
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
across the cluster, so the flags llmgw builds — checked against that version
— hold on every node.
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
llmgw builds — checked against that version — hold on every node.
**Volta (V100) is not covered by either build.** Read from vLLM's release
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
on such a node once it is installed.
- **The install computes on the GPU before it counts**: a small tensor
operation in the new environment. A build without kernels for the card
@@ -154,13 +162,12 @@ and `--ignore-checks` passed over every other problem with it.
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
or labels a different version of an engine, or none: vLLM 0.19.1, the last
built for Volta, on V100s beside the current one on newer cards.
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
had been CUDA 13's for every version, so the doctor's advice to set an older
version could not work. A refusal now says which version would.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
+16 -11
View File
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
run it — an older driver, a V100 — gets **the same version built for CUDA
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
0.20 on. A node with an older driver gets **the same version built for CUDA
12.9** instead, from vLLM's GitHub release (checked against the digest
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
(7.0) or newer. Either way the install ends by computing something on the
node's GPU, and a build that cannot fails to install, saying why;
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
Either way the install ends by computing something on the node's GPU, and a
build that cannot fails to install, saying why;
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
it is installed, since llmgw's flags are checked against the current one;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -427,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
engines: { vllm: "0.19.1" } # the last one that does
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
that version with `engine_overrides` (step 4) — its flags are not the ones
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+10 -5
View File
@@ -91,17 +91,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
// The engine from its GitHub release, pinned to the digest GitHub
// publishes for the file: the same trust as an archive, and uv
// refuses a download that does not match it.
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
if err != nil {
return err
}
sum, ok := sums[b.Wheel.Asset]
if !ok {
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
var asset, sum string
for name, s := range sums {
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
asset, sum = name, s
}
}
if asset == "" {
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
}
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
b.Wheel.Tag + "/" + b.Wheel.Asset
b.Wheel.Tag + "/" + asset
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
}
// A second index, for the torch the engine was built on. Every package
+24 -6
View File
@@ -143,7 +143,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
{"no GPU", ubuntu, Skipped, "has none"},
// Too old for CUDA 13 and for its CUDA 12.9 build both.
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
// vLLM 0.20 and later have no Volta kernels in any build; the
// refusal names the last version that has.
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -195,15 +198,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
}
}
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
// project's GitHub release, with the torch PyTorch publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
// 12.9 build, from the project's GitHub release, with the torch PyTorch
// publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, gh := pythonInstaller(t, volta, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Ready || got[0].Variant != "cu129" {
t.Fatalf("a V100 node: %+v", got[0])
t.Fatalf("a Turing node on driver 550: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
t.Errorf("not installed under its own build's name: %v", err)
@@ -268,3 +272,17 @@ func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
t.Errorf("a node with no GPU chosen cu129: %v", err)
}
}
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
b, err := chooseVLLM(voltaUntil, "", v100)
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
}
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
}
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
}
}
+60 -22
View File
@@ -66,9 +66,10 @@ type Build struct {
FreeGB int
}
// Wheel is a package published as a GitHub release asset.
// Wheel is a package published as a GitHub release asset: the asset whose
// name starts with Prefix and ends with Suffix.
type Wheel struct {
Project, Repo, Tag, Asset string
Project, Repo, Tag, Prefix, Suffix string
}
// Recipe knows how to install one engine.
@@ -140,30 +141,56 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
}, nil
}
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
const minDriverForCUDA13 = 580
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
// Volta, so a V100 cannot run these builds at all.
const minComputeForCUDA13 = 7.5
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
// version compatibility, and on Volta.
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
// compatibility; the oldest GPU one has kernels for is minComputeFor's
// answer, which is Volta only up to voltaUntil.
const (
minDriverForCUDA12 = 525
minComputeForCUDA12 = 7.0
)
// cuda13Since is the first version of each Python engine built for CUDA 13.
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
// cuda13Since is the first version of each Python engine whose PyPI build is
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
const voltaUntil = "0.19.1"
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
// on an architecture: what the project compiled, not what CUDA supports.
func minComputeFor(version, arch string) float64 {
if arch == "arm64" {
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
// since.
if versionBefore(version, "0.20") {
return 8.7
}
return 8.0
}
if versionBefore(version, "0.20") {
return minComputeForCUDA12
}
return minComputeForCUDA13
}
// cudaGate says whether a node can run this version of an engine, and which
// CUDA it is built for.
//
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
// nodes the older build (engine_overrides in the server's configuration)
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
// nodes the older version (engine_overrides in the server's configuration)
// rather than going without vLLM, or without the Volta nodes.
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
switch h.Arch {
@@ -178,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
if versionBefore(version, cuda13Since[engine]) {
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
}
if engine == "vLLM" {
minCC = minComputeFor(version, h.Arch)
}
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
hint := ""
if cuda == "13" {
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
}
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, hint)
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, minCC, cc, olderHint(engine, cc))
}
return variant, nil
}
// olderHint says which version runs on a GPU too old for the one asked for.
func olderHint(engine string, cc float64) string {
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
return "; the last vLLM built for it is " + voltaUntil +
": give these nodes that version with engine_overrides in the server's configuration"
}
return ""
}
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
@@ -262,19 +297,22 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
}
}
cu129 := func() (Build, error) {
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
return Build{}, err
}
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
b := pypi("cu129")
b.Requirements = nil
// Found in the release's assets rather than named: the manylinux tag
// in the file's name changes between versions (2_28, 2_31, 2_34).
b.Wheel = &Wheel{
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
}
b.ExtraIndex = pytorchCU129
return b, nil
}
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
modern := !versionBefore(version, cuda13Since["vLLM"])
switch variant {
case "":
@@ -301,7 +339,7 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
return pypi("cu13"), nil
case "cu129":
if !modern {
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
v, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
@@ -326,8 +364,8 @@ func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h disc
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
engine, version, cuda, minCC, cc)
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, olderHint(engine, cc))
}
return nil
}
+2 -1
View File
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
" none no vLLM on this node\n\n" +
"The node is told at once and installs it in the background; a build it\n" +
"cannot compute with on its GPU fails to install, saying why, and the\n" +
"cluster page shows it.\n\n" +
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
" llmgw ctl engine-build gpu07 vllm cu129",
Args: cobra.ExactArgs(3),
RunE: func(cmd *cobra.Command, args []string) error {
+7 -4
View File
@@ -154,11 +154,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
// What vLLM cannot do keeps vLLM off the node, not the node out of
// the cluster: it still serves llama.cpp.
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
// A V100 has no build of vLLM 0.20 or later, and is told which has.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},