diff --git a/CHANGELOG.md b/CHANGELOG.md
index de27a19d..5aaef403 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -130,31 +130,45 @@ leaving the model with nothing. Now:
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
-### Every GPU node gets a vLLM it can run, chosen on the cluster page
+### Every GPU node gets engine builds it can run, by itself
-The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
-where the driver (580+) allows, and otherwise the same version's CUDA 12.9
-build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
-CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
-llmgw builds — checked against that version — hold on every node.
+Which build of an engine a node installs follows its hardware — the CPU
+architecture, the driver, and the oldest GPU on it — from what each project
+publishes and how it was compiled. Each engine has a list of builds, best
+first; the node takes the first its hardware meets, and the first of those
+that installs and is seen to compute on the GPU. A build the table is wrong
+about costs its download, and the next one is tried.
-**Volta (V100) is not covered by either build.** Read from vLLM's release
-pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
-kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
-vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
-(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
-so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
-launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
-on such a node once it is installed.
+| | llama.cpp | vLLM |
+|---|---|---|
+| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
+| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
+| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
+| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
+| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
+| No NVIDIA GPU | the CPU build | none |
+Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
+from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
+compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
+carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
+starts at 7.5.
+
+- **A node running an older vLLM says so**: the cluster page's *vLLM build*
+ column shows the version installed, and the installation's detail says why
+ ("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
+ 0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
+ vLLM is not placed on a V100 by accident.
- **The install computes on the GPU before it counts**: a small tensor
- operation in the new environment. A build without kernels for the card
- imports and prints its version, and failed only on its first model; now it
- fails to install, saying so.
-- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
- CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
- `POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
- to the node at once.
+ operation for vLLM, the devices llama.cpp lists for its GPU builds.
+- **The build can still be chosen per node** on the cluster page (auto, CUDA
+ 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
+ `POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
+ at once.
+- **Whether an engine computes on its GPUs is read from every process
+ nvidia-smi lists**, not compute processes alone: the Vulkan build is a
+ graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
+ `CUDA_VISIBLE_DEVICES`.
### Engines per node, and nodes that do not run them all
@@ -169,12 +183,10 @@ and `--ignore-checks` passed over every other problem with it.
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
- or labels a different version of an engine, or none: vLLM 0.19.1, the last
- built for Volta, on V100s beside the current one on newer cards.
-- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
- PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
- had been CUDA 13's for every version, so the doctor's advice to set an older
- version could not work. A refusal now says which version would.
+ or labels a different version of an engine, or none — for exceptions; the
+ build each GPU generation needs is chosen without it (above).
+- **The requirements follow the version**, as the table above sets out; the
+ threshold had been CUDA 13's for every version.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
diff --git a/docs/install.md b/docs/install.md
index fed7dc55..631300c9 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
| Engine | On | How |
|---|---|---|
-| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
+| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
@@ -376,17 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
-- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
- **driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
- 0.20 on. A node with an older driver gets **the same version built for CUDA
- 12.9** instead, from vLLM's GitHub release (checked against the digest
- GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
- Either way the install ends by computing something on the node's GPU, and a
- build that cannot fails to install, saying why;
-- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
- every build from 0.20 on starts at 7.5. Give V100 nodes that version with
- `engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
- it is installed, since llmgw's flags are checked against the current one;
+- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
+ the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
+ CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
+ publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
+ (7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
+ **vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
+ kernels — and the cluster page shows it running that version; run `llmgw
+ ctl engine-check vllm` there once, since llmgw's flags are checked against
+ the current one. Pascal and older run no vLLM. Whatever is chosen, the
+ install ends by computing something on the GPU; a build that cannot is
+ passed over for the next;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -431,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- - gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
- engines: { vllm: "0.19.1" } # the last one that does
+ - gpu_class: a10g # pin a class to a version you have tested
+ engines: { vllm: "0.29.0" }
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1273,10 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
-- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
- Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
- that version with `engine_overrides` (step 4) — its flags are not the ones
- llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
+- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
+ no build from 0.20 on has Volta kernels, and the node picks the newest that
+ has. llmgw's flags are checked against the current version, so run `llmgw
+ ctl engine-check vllm` on a V100 node once.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
diff --git a/internal/agent/discovery/gpuprocs.go b/internal/agent/discovery/gpuprocs.go
index b99f2cf3..4c2b6684 100644
--- a/internal/agent/discovery/gpuprocs.go
+++ b/internal/agent/discovery/gpuprocs.go
@@ -2,6 +2,7 @@ package discovery
import (
"context"
+ "encoding/xml"
"fmt"
"os/exec"
"strconv"
@@ -10,10 +11,14 @@ import (
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
// nvidia-smi index: what a process actually computes on, as opposed to what it
-// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
+// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
// backend did not load runs on the CPU without a word, and this is where that
// shows — the process is on no GPU's list.
//
+// From the full query rather than --query-compute-apps, which lists compute
+// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
+// would have read as not on its GPU at all.
+//
// A node with no nvidia-smi returns nil and no error, as discovery does.
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
if binary == "" {
@@ -22,44 +27,38 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
}
binary = "nvidia-smi"
}
- gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
+ out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
- apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
- if err != nil {
- return nil, fmt.Errorf("running %s: %w", binary, err)
- }
- return parseGPUProcesses(string(gpus), string(apps)), nil
+ return parseGPUProcesses(out)
}
-// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
-// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
-func parseGPUProcesses(gpus, apps string) map[int][]int {
- index := map[string]int{}
- for _, line := range strings.Split(gpus, "\n") {
- i, uuid, ok := strings.Cut(line, ",")
- if !ok {
- continue
- }
- n, err := strconv.Atoi(strings.TrimSpace(i))
- if err != nil {
- continue
- }
- index[strings.TrimSpace(uuid)] = n
+// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
+// index order, and the processes on it.
+type smiLog struct {
+ GPUs []struct {
+ Processes []struct {
+ PID string `xml:"pid"`
+ Type string `xml:"type"`
+ } `xml:"processes>process_info"`
+ } `xml:"gpu"`
+}
+
+// parseGPUProcesses reads the report. GPUs are listed in index order, which
+// is nvidia-smi's own numbering — the one placement assigns by.
+func parseGPUProcesses(report []byte) (map[int][]int, error) {
+ var log smiLog
+ if err := xml.Unmarshal(report, &log); err != nil {
+ return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
}
out := map[int][]int{}
- for _, line := range strings.Split(apps, "\n") {
- uuid, p, ok := strings.Cut(line, ",")
- if !ok {
- continue
+ for i, g := range log.GPUs {
+ for _, p := range g.Processes {
+ if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
+ out[i] = append(out[i], pid)
+ }
}
- pid, err := strconv.Atoi(strings.TrimSpace(p))
- i, known := index[strings.TrimSpace(uuid)]
- if err != nil || !known {
- continue
- }
- out[i] = append(out[i], pid)
}
- return out
+ return out, nil
}
diff --git a/internal/agent/discovery/gpuprocs_test.go b/internal/agent/discovery/gpuprocs_test.go
index 723d2ade..4190d908 100644
--- a/internal/agent/discovery/gpuprocs_test.go
+++ b/internal/agent/discovery/gpuprocs_test.go
@@ -5,17 +5,34 @@ import (
"testing"
)
-func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
- gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
- // Two processes on GPU 1, one on GPU 2, and a GPU this list has never
- // heard of, which is dropped rather than guessed at.
- apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
- got := parseGPUProcesses(gpus, apps)
+func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
+ report := `
+
+ 580.65.06
+ Tesla V100-SXM2-32GB
+
+
+ Tesla V100-SXM2-32GB
+
+ 4100Cllama-server
+ 4300Gllama-server
+
+
+
+ 4200C+G
+
+`
+ got, err := parseGPUProcesses([]byte(report))
+ if err != nil {
+ t.Fatal(err)
+ }
+ // A Vulkan engine is a graphics process ("G") and is on its GPU all the
+ // same; a compute-only query would have missed it.
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
if !reflect.DeepEqual(got, want) {
t.Errorf("got %v, want %v", got, want)
}
- if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
- t.Errorf("no compute apps should be no GPU in use: %v", got)
+ if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
+ t.Error("a report that does not parse was read as no processes")
}
}
diff --git a/internal/agent/engines/llamacpp.go b/internal/agent/engines/llamacpp.go
index 9ca2253f..ad351286 100644
--- a/internal/agent/engines/llamacpp.go
+++ b/internal/agent/engines/llamacpp.go
@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
if s.APIKey != "" {
env["LLAMA_API_KEY"] = s.APIKey
}
+ // The Vulkan build — what a GPU older than the CUDA builds gets — does
+ // not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
+ // as the driver lists them, which on NVIDIA follows the same bus order
+ // as nvidia-smi; were it not to, the shard would not hold memory on its
+ // own GPUs, and starting it checks exactly that.
+ if len(s.GPUIndices) > 0 {
+ env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
+ }
return []Command{{
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
}}, nil
diff --git a/internal/agent/provision/install.go b/internal/agent/provision/install.go
index 9bb83afa..740759b5 100644
--- a/internal/agent/provision/install.go
+++ b/internal/agent/provision/install.go
@@ -143,7 +143,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
return st
}
- build, err := recipe.Choose(w.Version, w.Variant, in.Host)
+ builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
if err != nil {
st.State, st.Detail = Unsupported, err.Error()
var nh NotHere
@@ -152,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
}
return st
}
- st.Variant = build.Variant
- dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
- // Installed before: check it still runs, and trust nothing else about it.
- if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
+ // Installed before: whichever candidate is, best first, checked that it
+ // still runs and trusted for nothing else.
+ for _, build := range builds {
+ dir := in.dirFor(w, build)
+ if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
+ continue
+ }
+ st := st.as(w, build)
// The packages are checked too. Removing one is exactly the kind of
- // change an unrelated upgrade makes, and ldd below would catch it only
- // as a missing library with no hint of which package held it.
+ // change an unrelated upgrade makes, and ldd below would catch it
+ // only as a missing library with no hint of which package held it.
if err := in.packages(ctx, build.Packages, &st); err != nil {
st.State, st.Detail = Failed, err.Error()
return st
@@ -169,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
return in.ready(st, build, bin, version)
}
in.log().Warn("an installed engine no longer runs; installing it again",
- "engine", w.Engine, "version", w.Version, "err", err)
+ "engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
+ break
}
- if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
- st.State, st.Detail, st.Binary = Failed, err.Error(), ""
- return st
+ // Each candidate in turn until one installs and is seen to work: a build
+ // the table was wrong about — a driver too old for it after all, a GPU
+ // it has no kernels for — costs its download, and the next is tried.
+ var tried []string
+ for _, build := range builds {
+ st := st.as(w, build)
+ if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
+ tried = append(tried, build.Variant+": "+err.Error())
+ in.log().Warn("an engine build did not work here; trying the next",
+ "engine", w.Engine, "variant", build.Variant, "err", err)
+ continue
+ }
+ bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
+ if err != nil {
+ tried = append(tried, build.Variant+": "+err.Error())
+ continue
+ }
+ return in.ready(st, build, bin, version)
}
- bin, version, err := in.verify(ctx, dir, build)
- if err != nil {
- st.State, st.Detail = Failed, err.Error()
- return st
+ st.State, st.Binary = Failed, ""
+ st.Detail = strings.Join(tried, "; then ")
+ if len(builds) == 1 {
+ _, st.Detail, _ = strings.Cut(st.Detail, ": ")
}
- return in.ready(st, build, bin, version)
+ return st
+}
+
+// dirFor is where a build of an engine is installed: by the version it is
+// and its variant, so two builds of one version sit side by side.
+func (in *Installer) dirFor(w Want, b Build) string {
+ return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
+}
+
+// versionOr is the version a build is: its own, when it is not the one asked
+// for.
+func (b Build) versionOr(asked string) string {
+ if b.Version != "" {
+ return b.Version
+ }
+ return asked
+}
+
+// as is a status for one build of what was asked for.
+func (st Status) as(w Want, b Build) Status {
+ st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
+ return st
}
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
st.State, st.Binary = Ready, bin
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
+ if b.Note != "" {
+ st.Detail += "; " + b.Note
+ }
var missing []string
for _, f := range b.CPUFlags {
if !in.Host.HasCPUFlag(f) {
diff --git a/internal/agent/provision/install_test.go b/internal/agent/provision/install_test.go
index 07ee82c4..ca9daaec 100644
--- a/internal/agent/provision/install_test.go
+++ b/internal/agent/provision/install_test.go
@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
func TestTheBuildFollowsTheHardware(t *testing.T) {
gpu := ubuntu
gpu.NVIDIADriver = "570.86.15"
- b, err := chooseLlamaCpp("b1", "", gpu)
- if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
- t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
+ got, err := llamaCppCandidates("b1", "", gpu)
+ if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
+ t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
}
- if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
+ if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
}
+ for _, c := range []struct {
+ name string
+ host discovery.Host
+ want []string
+ }{
+ // Driver 580 and Turing: CUDA 13.3 first.
+ {"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
+ // A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
+ {"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
+ // An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
+ {"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
+ // A Pascal card: CUDA 12.8, not 13.3.
+ {"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
+ // arm64 with a GPU: never the CPU build.
+ {"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
+ } {
+ got, err := llamaCppCandidates("b1", "", c.host)
+ var names []string
+ for _, b := range got {
+ names = append(names, b.Variant)
+ }
+ if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
+ t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
+ }
+ }
slow := ubuntu
slow.CPUFlags = nil
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
// loader looks, and llama-server serving a GPU placement on the CPU.
- cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
+ cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
rel := newRelease(map[string][]byte{
"llama-b1-" + cuda: tarGz(t, map[string]string{
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
diff --git a/internal/agent/provision/python.go b/internal/agent/provision/python.go
index 0fc2e798..3cb74c43 100644
--- a/internal/agent/provision/python.go
+++ b/internal/agent/provision/python.go
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
defer cancel()
+ // The version this build is, which for a GPU older than every build of
+ // the one asked for is an older one: its publication date is the one
+ // the dependencies are resolved as of.
+ w.Version = b.versionOr(w.Version)
if err := in.packages(ctx, b.Packages, st); err != nil {
return err
diff --git a/internal/agent/provision/python_test.go b/internal/agent/provision/python_test.go
index 22b02c4f..6df8e235 100644
--- a/internal/agent/provision/python_test.go
+++ b/internal/agent/provision/python_test.go
@@ -25,8 +25,9 @@ type fakePython struct {
commands []string
failPip bool
// noKernel makes the GPU check fail as a torch without kernels for the
- // card does.
- noKernel bool
+ // card does: in every build, or in noKernelIn only ("-cu13.").
+ noKernel bool
+ noKernelIn string
}
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
@@ -48,7 +49,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
return []byte("0.30.0\n"), nil
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
- if f.noKernel {
+ if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
}
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
@@ -77,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
})
+ mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
+ _, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
+ })
pypi := httptest.NewServer(mux)
t.Cleanup(pypi.Close)
return &Installer{
@@ -143,10 +147,8 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
{"no GPU", ubuntu, Skipped, "has none"},
// Too old for CUDA 13 and for its CUDA 12.9 build both.
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
- {"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
- // vLLM 0.20 and later have no Volta kernels in any build; the
- // refusal names the last version that has.
- {"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
+ // No vLLM was ever built for Pascal.
+ {"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -240,11 +242,7 @@ func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
}
- volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
- if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
- t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
- }
- if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
+ if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
t.Error("a build that does not exist was accepted")
}
}
@@ -264,25 +262,51 @@ func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
// A machine vLLM is not published for is refused, not given the other
// build to fail with.
- if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
+ if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
!strings.Contains(err.Error(), "no Linux build") {
t.Errorf("an unknown architecture: %v", err)
}
- if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
+ if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
t.Errorf("a node with no GPU chosen cu129: %v", err)
}
+ // Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
+ got, err := vllmCandidates("0.30.0", "", gpuNode)
+ if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
+ t.Errorf("candidates on a current node: %+v, %v", got, err)
+ }
}
-func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
- v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
- b, err := chooseVLLM(voltaUntil, "", v100)
- if err != nil || b.Variant != "cu12" || b.Wheel != nil {
- t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
+func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
+ // 0.30.0 asked for; no build of it has Volta kernels. The node installs
+ // 0.19.1, the newest that has, and says so.
+ v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
+ sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
+ in, _ := pythonInstaller(t, v100, sys)
+ got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+ if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
+ !strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
+ t.Fatalf("a V100: %+v", got)
}
- if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
- t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
+ if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
+ t.Errorf("not installed as what it is: %v", err)
}
- if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
- t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
+ all := strings.Join(sys.commands, "\n")
+ if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
+ t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
+ }
+ // Chosen cu13, which 0.19.1 has none of: refused, saying why.
+ if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
+ t.Error("a CUDA 13 build was offered to a V100")
+ }
+}
+
+func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
+ // The table says CUDA 13 runs here; the card says otherwise. The next
+ // build is installed rather than the node going without.
+ sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
+ in, _ := pythonInstaller(t, gpuNode, sys)
+ got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
+ if got.State != Ready || got.Variant != "cu129" {
+ t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
}
}
diff --git a/internal/agent/provision/recipe.go b/internal/agent/provision/recipe.go
index 1a1c34a8..76c6c39f 100644
--- a/internal/agent/provision/recipe.go
+++ b/internal/agent/provision/recipe.go
@@ -10,7 +10,6 @@
package provision
import (
- "errors"
"fmt"
"slices"
"strings"
@@ -60,6 +59,13 @@ type Build struct {
// with no kernels for the card — a GPU generation its torch dropped —
// imports and prints its version, and fails on its first model.
GPUCheck bool
+ // Version is the engine version this build is, when it is not the one
+ // asked for: the newest built for this machine's GPUs, where the one
+ // asked for has no kernels for them. Empty is the version asked for.
+ Version string
+ // Note says why this build, when it is not the plain one: shown beside
+ // it wherever the installation is.
+ Note string
// FreeGB is the disk space the installation needs before it starts: a
// Python engine is several gigabytes, and one that fills the disk halfway
// leaves the node worse off than one that never began.
@@ -79,10 +85,12 @@ type Recipe struct {
Repo string
// PyPI is the project an engine installed from PyPI is published as.
PyPI string
- // Choose picks the build for a machine, or says why none fits. variant
- // is a build chosen for the node ("cu129"); empty leaves it to the
- // machine.
- Choose func(version, variant string, h discovery.Host) (Build, error)
+ // Candidates are the builds that may run on a machine, best first, or
+ // why none can. The installer takes the first that installs and is seen
+ // to work — computing on the GPU, where there is one — so a build this
+ // table is wrong about costs a download, not a node. variant is a build
+ // chosen for the node; empty leaves the choice to the machine.
+ Candidates func(version, variant string, h discovery.Host) ([]Build, error)
}
// Recipes are the engines this agent can install.
@@ -91,10 +99,10 @@ type Recipe struct {
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
// checked against its published checksum.
var Recipes = map[string]Recipe{
- "llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
- "vllm": {PyPI: "vllm", Choose: chooseVLLM},
- "sglang": {PyPI: "sglang", Choose: chooseSGLang},
- "uv": {Repo: "astral-sh/uv", Choose: chooseUV},
+ "llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
+ "vllm": {PyPI: "vllm", Candidates: vllmCandidates},
+ "sglang": {PyPI: "sglang", Candidates: sglangCandidates},
+ "uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
}
// Installable are the engines this agent can install, as it tells the
@@ -124,7 +132,15 @@ type NotHere struct{ Reason string }
func (e NotHere) Error() string { return e.Reason }
-func chooseUV(version, _ string, h discovery.Host) (Build, error) {
+func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
+ b, err := chooseUV(version, h)
+ if err != nil {
+ return nil, err
+ }
+ return []Build{b}, nil
+}
+
+func chooseUV(_ string, h discovery.Host) (Build, error) {
var arch string
switch h.Arch {
case "amd64":
@@ -141,58 +157,190 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
}, nil
}
-// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
-// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
-// "+cu130" wheel on GitHub), and SGLang's from 0.5.
-const minDriverForCUDA13 = 580
-
-// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
-// Volta, so a V100 cannot run these builds at all.
-const minComputeForCUDA13 = 7.5
-
-// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
-// compatibility; the oldest GPU one has kernels for is minComputeFor's
-// answer, which is Volta only up to voltaUntil.
+// What builds need of the NVIDIA side of a machine, read from what each
+// project publishes and how it was built:
+//
+// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
+// by CUDA's minor version compatibility — except code shipped only as
+// PTX, which the driver compiles and so needs a driver as new as the
+// CUDA it was compiled with.
+// - Which GPUs a build has kernels for is the build's own list, not what
+// its CUDA supports.
const (
- minDriverForCUDA12 = 525
- minComputeForCUDA12 = 7.0
+ minDriverForCUDA13 = 580
+ minDriverForCUDA12 = 525
)
-// cuda13Since is the first version of each Python engine whose PyPI build is
-// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
-// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
-var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
-
-// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
-// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
-// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
-const voltaUntil = "0.19.1"
-
-// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
-// on an architecture: what the project compiled, not what CUDA supports.
-func minComputeFor(version, arch string) float64 {
- if arch == "arm64" {
- // Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
- // since.
- if versionBefore(version, "0.20") {
- return 8.7
- }
- return 8.0
- }
- if versionBefore(version, "0.20") {
- return minComputeForCUDA12
- }
- return minComputeForCUDA13
+// gpuNeeds is what one build needs of a machine's GPUs and driver.
+type gpuNeeds struct {
+ driver int
+ cc float64
}
-// cudaGate says whether a node can run this version of an engine, and which
-// CUDA it is built for.
+// fits says why a machine cannot run a build, or nil.
+func (n gpuNeeds) fits(what string, h discovery.Host) error {
+ if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
+ return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
+ }
+ if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
+ return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
+ }
+ return nil
+}
+
+// candidate is a build and what it needs.
+type candidate struct {
+ b Build
+ needs gpuNeeds
+ what string
+}
+
+// pick keeps the candidates a machine can run, in order, and those a
+// variant chosen for the node names; with none left, it says why each went.
+func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
+ var out []Build
+ var why []string
+ for _, c := range cands {
+ if variant != "" && c.b.Variant != variant {
+ continue
+ }
+ if err := c.needs.fits(c.what, h); err != nil {
+ why = append(why, err.Error())
+ continue
+ }
+ out = append(out, c.b)
+ }
+ switch {
+ case len(out) > 0:
+ return out, nil
+ case len(why) == 0 && variant != "":
+ var names []string
+ for _, c := range cands {
+ names = append(names, c.b.Variant)
+ }
+ return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
+ }
+ return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
+}
+
+// --- vLLM ---
+
+// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
+// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
+// version, built for drivers CUDA 13 does not run on.
+var VLLMVariants = []string{"cu13", "cu129"}
+
+// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
+// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
+const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
+
+// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
+// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
+// "+cu129" beside a CUDA 13 one.
+const vllmCUDA13Since = "0.20"
+
+// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
+// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
+// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
+// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
+// 0.20 and 8.0 since.
+func vllmMinCompute(version, arch string) float64 {
+ old := versionBefore(version, vllmCUDA13Since)
+ switch {
+ case arch == "arm64" && old:
+ return 8.7
+ case arch == "arm64":
+ return 8.0
+ case old:
+ return 7.0
+ }
+ return 7.5
+}
+
+// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
+// on an architecture, for GPUs older than every build of the version asked
+// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
+// for a GPU no vLLM was built for.
+var vllmNewestFor = []struct {
+ arch string
+ from float64 // the oldest GPU it covers
+ version string
+}{
+ {"amd64", 7.0, "0.19.1"},
+}
+
+// vllmCandidates are the vLLM builds for a machine.
//
-// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
-// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
-// nodes the older version (engine_overrides in the server's configuration)
-// rather than going without vLLM, or without the Volta nodes.
-func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
+// The version asked for, in the build the machine runs: CUDA 13 where the
+// driver allows, the same version's CUDA 12.9 build where it does not. One
+// version across the cluster wherever the GPUs allow it, because the flags
+// llmgw builds are checked against it. Where they do not — GPUs older than
+// every build of that version — the newest version built for them, said so
+// beside the installation: vLLM on a V100 at all, rather than not.
+func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+ arch, err := gpuArch("vLLM", h)
+ if err != nil {
+ return nil, err
+ }
+ v, note := version, ""
+ if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
+ older := ""
+ for _, n := range vllmNewestFor {
+ if n.arch == h.Arch && cc >= n.from {
+ older = n.version
+ }
+ }
+ if older == "" {
+ return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
+ "the node serves llama.cpp models", cc, arch)
+ }
+ v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
+ }
+ pypi := func(variant string) Build {
+ b := Build{
+ Variant: variant,
+ Python: pythonForEngines,
+ Requirements: []string{"vllm==" + v},
+ Binary: "vllm",
+ Packages: compilerPackages,
+ FreeGB: 25,
+ GPUCheck: true,
+ Note: note,
+ }
+ if v != version {
+ b.Version = v
+ }
+ return b
+ }
+ minCC := vllmMinCompute(v, h.Arch)
+ if versionBefore(v, vllmCUDA13Since) {
+ // PyPI's own build is CUDA 12, and is the one there is; asked for
+ // cu129, this is it.
+ if variant == "cu129" {
+ variant = "cu12"
+ }
+ return pick("vLLM "+v, variant, []candidate{
+ {pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
+ }, h)
+ }
+ cu129 := pypi("cu129")
+ cu129.Requirements = nil
+ // Found in the release's assets rather than named: the manylinux tag
+ // in the file's name changes between versions (2_28, 2_31, 2_34).
+ cu129.Wheel = &Wheel{
+ Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
+ Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
+ }
+ cu129.ExtraIndex = pytorchCU129
+ return pick("vLLM "+v, variant, []candidate{
+ {pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
+ {cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
+ }, h)
+}
+
+// gpuArch says whether a Python GPU engine is published for this machine
+// and whether it has an NVIDIA GPU at all.
+func gpuArch(engine string, h discovery.Host) (string, error) {
switch h.Arch {
case "amd64", "arm64":
default:
@@ -201,33 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
if h.NVIDIADriver == "" {
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
}
- variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
- if versionBefore(version, cuda13Since[engine]) {
- variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
- }
- if engine == "vLLM" {
- minCC = minComputeFor(version, h.Arch)
- }
- if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
- return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
- engine, version, cuda, minDriver, h.NVIDIADriver)
- }
- if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
- return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
- engine, version, minCC, cc, olderHint(engine, cc))
- }
- return variant, nil
+ return h.Arch, nil
}
-// olderHint says which version runs on a GPU too old for the one asked for.
-func olderHint(engine string, cc float64) string {
- if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
- return "; the last vLLM built for it is " + voltaUntil +
- ": give these nodes that version with engine_overrides in the server's configuration"
+// --- SGLang ---
+
+// sglangCUDA13Since is the first SGLang built for CUDA 13.
+const sglangCUDA13Since = "0.5"
+
+func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
+ if _, err := gpuArch("SGLang", h); err != nil {
+ return nil, err
}
- return ""
+ b := Build{
+ Variant: "cu13",
+ Python: pythonForEngines,
+ Requirements: []string{"sglang==" + version},
+ // SGLang is launched as a module, so what the agent is given is the
+ // environment's interpreter, and the proof is importing it.
+ Binary: "python",
+ VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
+ Packages: compilerPackages,
+ FreeGB: 25,
+ GPUCheck: true,
+ }
+ needs := gpuNeeds{minDriverForCUDA13, 7.5}
+ if versionBefore(version, sglangCUDA13Since) {
+ b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
+ }
+ return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
}
+// Python engines need a C compiler at run time: Triton builds a small
+// launcher for its kernels the first time a model loads, and without gcc
+// that load fails long after the installation looked finished.
+var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
+
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
@@ -251,146 +408,6 @@ func versionBefore(a, b string) bool {
return false
}
-// Python engines need a C compiler at run time: Triton builds a small
-// launcher for its kernels the first time a model loads, and without gcc
-// that load fails long after the installation looked finished.
-var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
-
-// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
-// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
-// version, built for the drivers and GPUs CUDA 13 left behind.
-var VLLMVariants = []string{"cu13", "cu129"}
-
-// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
-// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
-const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
-
-// chooseVLLM picks the vLLM build for a machine.
-//
-// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
-// same version's CUDA 12.9 build where only that does: one version across
-// the cluster, which matters because the flags llmgw builds are checked
-// against one version, and an older vLLM on the older cards would be a
-// second set of flags nobody checked. A variant chosen for the node is that
-// build, if the node can run it.
-func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
- // What no build of it gets past: a machine it is not published for, or
- // one with no NVIDIA GPU. Only a CUDA requirement falls back to the
- // CUDA 12.9 build.
- switch h.Arch {
- case "amd64", "arm64":
- default:
- return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
- }
- if h.NVIDIADriver == "" {
- return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
- }
- pypi := func(v string) Build {
- return Build{
- Variant: v,
- Python: pythonForEngines,
- Requirements: []string{"vllm==" + version},
- Binary: "vllm",
- Packages: compilerPackages,
- FreeGB: 25,
- GPUCheck: true,
- }
- }
- cu129 := func() (Build, error) {
- if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
- return Build{}, err
- }
- arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
- b := pypi("cu129")
- b.Requirements = nil
- // Found in the release's assets rather than named: the manylinux tag
- // in the file's name changes between versions (2_28, 2_31, 2_34).
- b.Wheel = &Wheel{
- Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
- Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
- }
- b.ExtraIndex = pytorchCU129
- return b, nil
- }
- // From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
- modern := !versionBefore(version, cuda13Since["vLLM"])
- switch variant {
- case "":
- v, err := cudaGate("vLLM", version, h)
- if err == nil {
- return pypi(v), nil
- }
- var nh NotHere
- if errors.As(err, &nh) || !modern {
- return Build{}, err
- }
- b, err129 := cu129()
- if err129 != nil {
- return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
- }
- return b, nil
- case "cu13":
- if !modern {
- return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
- }
- if _, err := cudaGate("vLLM", version, h); err != nil {
- return Build{}, err
- }
- return pypi("cu13"), nil
- case "cu129":
- if !modern {
- // Before 0.20 PyPI's own build is CUDA 12, and is the one given.
- v, err := cudaGate("vLLM", version, h)
- if err != nil {
- return Build{}, err
- }
- return pypi(v), nil
- }
- if _, err := cudaGate("vLLM", version, h); err != nil {
- var nh NotHere
- if errors.As(err, &nh) {
- return Build{}, err
- }
- }
- return cu129()
- }
- return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
-}
-
-// cudaFits checks a node against a CUDA's driver and GPU requirements.
-func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
- if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
- return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
- engine, version, cuda, minDriver, h.NVIDIADriver)
- }
- if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
- return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
- engine, version, cuda, minCC, cc, olderHint(engine, cc))
- }
- return nil
-}
-
-func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
- if chosen != "" {
- return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
- }
- variant, err := cudaGate("SGLang", version, h)
- if err != nil {
- return Build{}, err
- }
- // SGLang is launched as a module, so what the agent is given is the
- // environment's interpreter, and the proof is importing it.
- return Build{
- Variant: variant,
- Python: pythonForEngines,
- Requirements: []string{"sglang==" + version},
- Binary: "python",
- VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
- Packages: compilerPackages,
- FreeGB: 25,
- }, nil
-}
-
// leadingInt is the number a version string starts with: 580 of "580.65.06".
func leadingInt(s string) int {
n := 0
@@ -403,17 +420,30 @@ func leadingInt(s string) int {
return n
}
-// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
-// carries a matching cudart archive, so the node needs a driver and nothing
-// else — no toolkit.
-const cudaForLlamaCpp = "12.8"
+// --- llama.cpp ---
-func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
- // The published Linux builds are built on Ubuntu, but nothing about them
- // is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
- // Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
- // name decides nothing here. The build is tried everywhere, and ldd says
- // whether this system can load it — in the words of the missing symbol.
+// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
+// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
+// carries as PTX, which a driver older than the CUDA it came from cannot
+// compile.
+var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
+
+// llamaCppCandidates are the llama.cpp builds for a machine.
+//
+// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
+// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
+// the GPUs it ships as PTX, 525 for those it ships as machine code), and
+// Vulkan, which runs on any card whose driver has Vulkan — the generations
+// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
+// a GPU placement served on the CPU, which is what installing checks for.
+// The CPU build where there is no NVIDIA GPU.
+//
+// The published Linux builds are built on Ubuntu, but nothing about them is
+// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
+// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
+// nothing here. The build is tried everywhere, and ldd says whether this
+// system can load it — in the words of the missing symbol.
+func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
var arch string
switch h.Arch {
case "amd64":
@@ -421,36 +451,53 @@ func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
case "arm64":
arch = "arm64"
default:
- return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
+ return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
}
-
- b := Build{
- Variant: "cpu",
- Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
- // Everything else the build links is in the archive or in any current
- // glibc system; OpenMP's runtime is the one a cloud image lacks, and
- // the server does not start without it.
- Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
- Binary: "llama-server",
- }
- if arch == "x64" {
- // The archive carries a backend per CPU generation and loads the best
- // one, down to SSE4.2. Below AVX2 it runs, several times slower.
- b.CPUFlags = []string{"avx2"}
- }
- // The CUDA build is published for x64 at this version; on arm64 it is a
- // newer CUDA that the driver may not support, so an arm64 GPU node runs
- // the CPU build until someone decides otherwise.
- if h.NVIDIADriver != "" && arch == "x64" {
- name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
- b.Variant = "cuda-" + cudaForLlamaCpp
- b.GPUBackend = "libggml-cuda.so"
- b.Assets = []string{
- fmt.Sprintf("llama-%s-%s", version, name),
- fmt.Sprintf("cudart-llama-%s-%s", version, name),
+ base := func(variant string, assets ...string) Build {
+ b := Build{
+ Variant: variant,
+ Assets: assets,
+ // Everything else the build links is in the archive or in any
+ // current glibc system; OpenMP's runtime is the one a cloud image
+ // lacks, and the server does not start without it.
+ Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
+ Binary: "llama-server",
}
+ if arch == "x64" {
+ // The archive carries a backend per CPU generation and loads the
+ // best one, down to SSE4.2. Below AVX2 it runs, several times
+ // slower.
+ b.CPUFlags = []string{"avx2"}
+ }
+ return b
}
- return b, nil
+ if h.NVIDIADriver == "" {
+ return pick("llama.cpp", variant, []candidate{
+ {base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
+ }, h)
+ }
+ cuda := func(cv string) Build {
+ name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
+ // The release carries the CUDA runtime beside the build, so the node
+ // needs a driver and nothing else — no toolkit.
+ b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
+ b.GPUBackend = "libggml-cuda.so"
+ return b
+ }
+ vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
+ vulkan.GPUBackend = "libggml-vulkan.so"
+ vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
+
+ cuda12Driver := 570
+ if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
+ cuda12Driver = minDriverForCUDA12
+ }
+ cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
+ if arch == "x64" {
+ cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
+ }
+ cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
+ return pick("llama.cpp", variant, cands, h)
}
func orUnknown(s string) string {
diff --git a/internal/cli/enginebuild.go b/internal/cli/enginebuild.go
index 423ea3bd..621a5275 100644
--- a/internal/cli/enginebuild.go
+++ b/internal/cli/enginebuild.go
@@ -22,8 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
" none no vLLM on this node\n\n" +
"The node is told at once and installs it in the background; a build it\n" +
"cannot compute with on its GPU fails to install, saying why, and the\n" +
- "cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
- "give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
+ "cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
+ "gets 0.19.1, the newest built for it, by itself.\n\n" +
" llmgw ctl engine-build gpu07 vllm cu129",
Args: cobra.ExactArgs(3),
RunE: func(cmd *cobra.Command, args []string) error {
diff --git a/internal/doctor/doctor.go b/internal/doctor/doctor.go
index d39eddf5..0de05897 100644
--- a/internal/doctor/doctor.go
+++ b/internal/doctor/doctor.go
@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
// the same decision the agent makes, so the report and the node agree.
func vllmFindings(f Facts, add func(Level, string, string, string)) {
v := f.VLLMVersion
- b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
+ builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
+ var b provision.Build
+ if err == nil {
+ // The first the agent would try; it moves to the next by itself if
+ // that one does not work here.
+ b = builds[0]
+ if b.Version != "" {
+ v = b.Version
+ }
+ }
var nh provision.NotHere
switch {
case errors.As(err, &nh):
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
// llama.cpp. Refusing it here kept older GPU generations out of a
// cluster for an engine nobody meant to run on them.
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
- "give this node a vLLM it can run with engine_overrides in the server's configuration, "+
- "or add it with --disable-engine vllm if it is meant for llama.cpp alone")
+ "add it with --disable-engine vllm if it is meant for llama.cpp alone")
return
case f.VLLMInstalled:
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
"free space there, or point engines_dir in agent.yaml at a larger disk")
return
}
- add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
- v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
+ note := ""
+ if b.Note != "" {
+ note = "; " + b.Note
+ }
+ add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
+ v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
}
func rank(l Level) int {
diff --git a/internal/doctor/doctor_test.go b/internal/doctor/doctor_test.go
index d54e541d..e77902b2 100644
--- a/internal/doctor/doctor_test.go
+++ b/internal/doctor/doctor_test.go
@@ -157,9 +157,12 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
- // A V100 has no build of vLLM 0.20 or later, and is told which has.
+ // A V100 has no build of vLLM 0.20 or later; it gets the newest
+ // that has Volta kernels, and doctor says which.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
- VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
+ VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
+ {"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
+ VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
diff --git a/internal/web/admin_nodeengine_test.go b/internal/web/admin_nodeengine_test.go
index 37b5008c..1942bf45 100644
--- a/internal/web/admin_nodeengine_test.go
+++ b/internal/web/admin_nodeengine_test.go
@@ -52,7 +52,7 @@ func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
t.Error("a node with no GPU was offered a vLLM build")
}
- if !strings.Contains(body, "CUDA 12.9 installing") {
+ if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
t.Error("what is installed is not shown beside the choice")
}
diff --git a/internal/web/templates/dashboard.html b/internal/web/templates/dashboard.html
index 0ca38e7d..195ca696 100644
--- a/internal/web/templates/dashboard.html
+++ b/internal/web/templates/dashboard.html
@@ -72,7 +72,7 @@
together. Only a GPU node has one; changing it is for an
administrator, on the page reached through a session. */}}
{{if $.AnyGPUs}}
{{if .GPUTotal}}{{$node := .}}
- {{range .Provision}}{{if eq .Engine "vllm"}}{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}{{end}}{{end}}
+ {{range .Provision}}{{if eq .Engine "vllm"}}{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}{{end}}{{end}}
{{if and $.Menu.CSRF .EngineBuilds}}
|