Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc483d8c68 |
No files matched your search
+39
-27
@@ -130,31 +130,45 @@ leaving the model with nothing. Now:
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
### Every GPU node gets engine builds it can run, by itself
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
|
||||
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
|
||||
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
|
||||
llmgw builds — checked against that version — hold on every node.
|
||||
Which build of an engine a node installs follows its hardware — the CPU
|
||||
architecture, the driver, and the oldest GPU on it — from what each project
|
||||
publishes and how it was compiled. Each engine has a list of builds, best
|
||||
first; the node takes the first its hardware meets, and the first of those
|
||||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||||
about costs its download, and the next one is tried.
|
||||
|
||||
**Volta (V100) is not covered by either build.** Read from vLLM's release
|
||||
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
|
||||
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
|
||||
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
|
||||
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
|
||||
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
|
||||
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
|
||||
on such a node once it is installed.
|
||||
| | llama.cpp | vLLM |
|
||||
|---|---|---|
|
||||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||||
| No NVIDIA GPU | the CPU build | none |
|
||||
|
||||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||||
starts at 7.5.
|
||||
|
||||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||||
column shows the version installed, and the installation's detail says why
|
||||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||||
vLLM is not placed on a V100 by accident.
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
imports and prints its version, and failed only on its first model; now it
|
||||
fails to install, saying so.
|
||||
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
|
||||
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
|
||||
to the node at once.
|
||||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||||
at once.
|
||||
- **Whether an engine computes on its GPUs is read from every process
|
||||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
@@ -169,12 +183,10 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: vLLM 0.19.1, the last
|
||||
built for Volta, on V100s beside the current one on newer cards.
|
||||
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
|
||||
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
|
||||
had been CUDA 13's for every version, so the doctor's advice to set an older
|
||||
version could not work. A refusal now says which version would.
|
||||
or labels a different version of an engine, or none — for exceptions; the
|
||||
build each GPU generation needs is chosen without it (above).
|
||||
- **The requirements follow the version**, as the table above sets out; the
|
||||
threshold had been CUDA 13's for every version.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
+18
-18
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
@@ -376,17 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
|
||||
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
|
||||
0.20 on. A node with an older driver gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
|
||||
Either way the install ends by computing something on the node's GPU, and a
|
||||
build that cannot fails to install, saying why;
|
||||
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
|
||||
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
|
||||
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
|
||||
it is installed, since llmgw's flags are checked against the current one;
|
||||
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
|
||||
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
|
||||
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
|
||||
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
|
||||
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
|
||||
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
|
||||
kernels — and the cluster page shows it running that version; run `llmgw
|
||||
ctl engine-check vllm` there once, since llmgw's flags are checked against
|
||||
the current one. Pascal and older run no vLLM. Whatever is chosen, the
|
||||
install ends by computing something on the GPU; a build that cannot is
|
||||
passed over for the next;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -431,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
|
||||
engines: { vllm: "0.19.1" } # the last one that does
|
||||
- gpu_class: a10g # pin a class to a version you have tested
|
||||
engines: { vllm: "0.29.0" }
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1273,10 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
|
||||
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
|
||||
that version with `engine_overrides` (step 4) — its flags are not the ones
|
||||
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
has. llmgw's flags are checked against the current version, so run `llmgw
|
||||
ctl engine-check vllm` on a V100 node once.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -2,6 +2,7 @@ package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
@@ -10,10 +11,14 @@ import (
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// From the full query rather than --query-compute-apps, which lists compute
|
||||
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
|
||||
// would have read as not on its GPU at all.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
@@ -22,44 +27,38 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(string(gpus), string(apps)), nil
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
|
||||
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
|
||||
func parseGPUProcesses(gpus, apps string) map[int][]int {
|
||||
index := map[string]int{}
|
||||
for _, line := range strings.Split(gpus, "\n") {
|
||||
i, uuid, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.Atoi(strings.TrimSpace(i))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
index[strings.TrimSpace(uuid)] = n
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for _, line := range strings.Split(apps, "\n") {
|
||||
uuid, p, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
for i, g := range log.GPUs {
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(p))
|
||||
i, known := index[strings.TrimSpace(uuid)]
|
||||
if err != nil || !known {
|
||||
continue
|
||||
}
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
return out
|
||||
return out, nil
|
||||
}
|
||||
@@ -5,17 +5,34 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
|
||||
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
|
||||
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
|
||||
// heard of, which is dropped rather than guessed at.
|
||||
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
|
||||
got := parseGPUProcesses(gpus, apps)
|
||||
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>580.65.06</driver_version>
|
||||
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes>
|
||||
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
|
||||
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:B3:00.0"><processes>
|
||||
<process_info><pid>4200</pid><type>C+G</type></process_info>
|
||||
</processes></gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUProcesses([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
|
||||
// same; a compute-only query would have missed it.
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
|
||||
t.Errorf("no compute apps should be no GPU in use: %v", got)
|
||||
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLAMA_API_KEY"] = s.APIKey
|
||||
}
|
||||
// The Vulkan build — what a GPU older than the CUDA builds gets — does
|
||||
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
|
||||
}}, nil
|
||||
|
||||
@@ -143,7 +143,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
|
||||
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -152,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
st.Variant = build.Variant
|
||||
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
|
||||
|
||||
// Installed before: check it still runs, and trust nothing else about it.
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
|
||||
// Installed before: whichever candidate is, best first, checked that it
|
||||
// still runs and trusted for nothing else.
|
||||
for _, build := range builds {
|
||||
dir := in.dirFor(w, build)
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
|
||||
continue
|
||||
}
|
||||
st := st.as(w, build)
|
||||
// The packages are checked too. Removing one is exactly the kind of
|
||||
// change an unrelated upgrade makes, and ldd below would catch it only
|
||||
// as a missing library with no hint of which package held it.
|
||||
// change an unrelated upgrade makes, and ldd below would catch it
|
||||
// only as a missing library with no hint of which package held it.
|
||||
if err := in.packages(ctx, build.Packages, &st); err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
@@ -169,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
in.log().Warn("an installed engine no longer runs; installing it again",
|
||||
"engine", w.Engine, "version", w.Version, "err", err)
|
||||
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
|
||||
break
|
||||
}
|
||||
|
||||
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
|
||||
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
|
||||
return st
|
||||
// Each candidate in turn until one installs and is seen to work: a build
|
||||
// the table was wrong about — a driver too old for it after all, a GPU
|
||||
// it has no kernels for — costs its download, and the next is tried.
|
||||
var tried []string
|
||||
for _, build := range builds {
|
||||
st := st.as(w, build)
|
||||
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
in.log().Warn("an engine build did not work here; trying the next",
|
||||
"engine", w.Engine, "variant", build.Variant, "err", err)
|
||||
continue
|
||||
}
|
||||
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
|
||||
if err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
bin, version, err := in.verify(ctx, dir, build)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
st.State, st.Binary = Failed, ""
|
||||
st.Detail = strings.Join(tried, "; then ")
|
||||
if len(builds) == 1 {
|
||||
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
return st
|
||||
}
|
||||
|
||||
// dirFor is where a build of an engine is installed: by the version it is
|
||||
// and its variant, so two builds of one version sit side by side.
|
||||
func (in *Installer) dirFor(w Want, b Build) string {
|
||||
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
|
||||
}
|
||||
|
||||
// versionOr is the version a build is: its own, when it is not the one asked
|
||||
// for.
|
||||
func (b Build) versionOr(asked string) string {
|
||||
if b.Version != "" {
|
||||
return b.Version
|
||||
}
|
||||
return asked
|
||||
}
|
||||
|
||||
// as is a status for one build of what was asked for.
|
||||
func (st Status) as(w Want, b Build) Status {
|
||||
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
|
||||
return st
|
||||
}
|
||||
|
||||
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
|
||||
st.State, st.Binary = Ready, bin
|
||||
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
|
||||
if b.Note != "" {
|
||||
st.Detail += "; " + b.Note
|
||||
}
|
||||
var missing []string
|
||||
for _, f := range b.CPUFlags {
|
||||
if !in.Host.HasCPUFlag(f) {
|
||||
|
||||
@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", "", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
got, err := llamaCppCandidates("b1", "", gpu)
|
||||
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
|
||||
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
}{
|
||||
// Driver 580 and Turing: CUDA 13.3 first.
|
||||
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
|
||||
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
|
||||
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
|
||||
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
|
||||
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
|
||||
// A Pascal card: CUDA 12.8, not 13.3.
|
||||
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
|
||||
// arm64 with a GPU: never the CPU build.
|
||||
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
|
||||
} {
|
||||
got, err := llamaCppCandidates("b1", "", c.host)
|
||||
var names []string
|
||||
for _, b := range got {
|
||||
names = append(names, b.Variant)
|
||||
}
|
||||
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
|
||||
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
slow := ubuntu
|
||||
slow.CPUFlags = nil
|
||||
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
|
||||
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
|
||||
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
// The version this build is, which for a GPU older than every build of
|
||||
// the one asked for is an older one: its publication date is the one
|
||||
// the dependencies are resolved as of.
|
||||
w.Version = b.versionOr(w.Version)
|
||||
|
||||
if err := in.packages(ctx, b.Packages, st); err != nil {
|
||||
return err
|
||||
|
||||
@@ -25,8 +25,9 @@ type fakePython struct {
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does.
|
||||
noKernel bool
|
||||
// card does: in every build, or in noKernelIn only ("-cu13.").
|
||||
noKernel bool
|
||||
noKernelIn string
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -48,7 +49,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel {
|
||||
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
@@ -77,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
|
||||
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
|
||||
})
|
||||
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
|
||||
})
|
||||
pypi := httptest.NewServer(mux)
|
||||
t.Cleanup(pypi.Close)
|
||||
return &Installer{
|
||||
@@ -143,10 +147,8 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
|
||||
// vLLM 0.20 and later have no Volta kernels in any build; the
|
||||
// refusal names the last version that has.
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
|
||||
// No vLLM was ever built for Pascal.
|
||||
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -240,11 +242,7 @@ func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
|
||||
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
|
||||
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
@@ -264,25 +262,51 @@ func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
|
||||
got, err := vllmCandidates("0.30.0", "", gpuNode)
|
||||
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
|
||||
t.Errorf("candidates on a current node: %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
|
||||
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
b, err := chooseVLLM(voltaUntil, "", v100)
|
||||
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
|
||||
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
|
||||
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
|
||||
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
|
||||
// 0.19.1, the newest that has, and says so.
|
||||
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, v100, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
|
||||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
|
||||
t.Fatalf("a V100: %+v", got)
|
||||
}
|
||||
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
|
||||
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
|
||||
t.Errorf("not installed as what it is: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
|
||||
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
|
||||
}
|
||||
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
|
||||
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
|
||||
t.Error("a CUDA 13 build was offered to a V100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
|
||||
// The table says CUDA 13 runs here; the card says otherwise. The next
|
||||
// build is installed rather than the node going without.
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Variant != "cu129" {
|
||||
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
|
||||
}
|
||||
}
|
||||
+302
-255
@@ -10,7 +10,6 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -60,6 +59,13 @@ type Build struct {
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// Version is the engine version this build is, when it is not the one
|
||||
// asked for: the newest built for this machine's GPUs, where the one
|
||||
// asked for has no kernels for them. Empty is the version asked for.
|
||||
Version string
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
@@ -79,10 +85,12 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits. variant
|
||||
// is a build chosen for the node ("cu129"); empty leaves it to the
|
||||
// machine.
|
||||
Choose func(version, variant string, h discovery.Host) (Build, error)
|
||||
// Candidates are the builds that may run on a machine, best first, or
|
||||
// why none can. The installer takes the first that installs and is seen
|
||||
// to work — computing on the GPU, where there is one — so a build this
|
||||
// table is wrong about costs a download, not a node. variant is a build
|
||||
// chosen for the node; empty leaves the choice to the machine.
|
||||
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -91,10 +99,10 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
|
||||
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
|
||||
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
|
||||
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -124,7 +132,15 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
|
||||
b, err := chooseUV(version, h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []Build{b}, nil
|
||||
}
|
||||
|
||||
func chooseUV(_ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -141,58 +157,190 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
|
||||
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
|
||||
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
|
||||
// compatibility; the oldest GPU one has kernels for is minComputeFor's
|
||||
// answer, which is Volta only up to voltaUntil.
|
||||
// What builds need of the NVIDIA side of a machine, read from what each
|
||||
// project publishes and how it was built:
|
||||
//
|
||||
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
|
||||
// by CUDA's minor version compatibility — except code shipped only as
|
||||
// PTX, which the driver compiles and so needs a driver as new as the
|
||||
// CUDA it was compiled with.
|
||||
// - Which GPUs a build has kernels for is the build's own list, not what
|
||||
// its CUDA supports.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
minDriverForCUDA13 = 580
|
||||
minDriverForCUDA12 = 525
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine whose PyPI build is
|
||||
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
|
||||
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
|
||||
|
||||
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
|
||||
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
|
||||
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
|
||||
const voltaUntil = "0.19.1"
|
||||
|
||||
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
|
||||
// on an architecture: what the project compiled, not what CUDA supports.
|
||||
func minComputeFor(version, arch string) float64 {
|
||||
if arch == "arm64" {
|
||||
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
|
||||
// since.
|
||||
if versionBefore(version, "0.20") {
|
||||
return 8.7
|
||||
}
|
||||
return 8.0
|
||||
}
|
||||
if versionBefore(version, "0.20") {
|
||||
return minComputeForCUDA12
|
||||
}
|
||||
return minComputeForCUDA13
|
||||
// gpuNeeds is what one build needs of a machine's GPUs and driver.
|
||||
type gpuNeeds struct {
|
||||
driver int
|
||||
cc float64
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
// fits says why a machine cannot run a build, or nil.
|
||||
func (n gpuNeeds) fits(what string, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
|
||||
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
|
||||
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// candidate is a build and what it needs.
|
||||
type candidate struct {
|
||||
b Build
|
||||
needs gpuNeeds
|
||||
what string
|
||||
}
|
||||
|
||||
// pick keeps the candidates a machine can run, in order, and those a
|
||||
// variant chosen for the node names; with none left, it says why each went.
|
||||
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
|
||||
var out []Build
|
||||
var why []string
|
||||
for _, c := range cands {
|
||||
if variant != "" && c.b.Variant != variant {
|
||||
continue
|
||||
}
|
||||
if err := c.needs.fits(c.what, h); err != nil {
|
||||
why = append(why, err.Error())
|
||||
continue
|
||||
}
|
||||
out = append(out, c.b)
|
||||
}
|
||||
switch {
|
||||
case len(out) > 0:
|
||||
return out, nil
|
||||
case len(why) == 0 && variant != "":
|
||||
var names []string
|
||||
for _, c := range cands {
|
||||
names = append(names, c.b.Variant)
|
||||
}
|
||||
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
|
||||
}
|
||||
|
||||
// --- vLLM ---
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for drivers CUDA 13 does not run on.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
const vllmCUDA13Since = "0.20"
|
||||
|
||||
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
|
||||
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
|
||||
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
|
||||
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
|
||||
// 0.20 and 8.0 since.
|
||||
func vllmMinCompute(version, arch string) float64 {
|
||||
old := versionBefore(version, vllmCUDA13Since)
|
||||
switch {
|
||||
case arch == "arm64" && old:
|
||||
return 8.7
|
||||
case arch == "arm64":
|
||||
return 8.0
|
||||
case old:
|
||||
return 7.0
|
||||
}
|
||||
return 7.5
|
||||
}
|
||||
|
||||
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
|
||||
// on an architecture, for GPUs older than every build of the version asked
|
||||
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
|
||||
// for a GPU no vLLM was built for.
|
||||
var vllmNewestFor = []struct {
|
||||
arch string
|
||||
from float64 // the oldest GPU it covers
|
||||
version string
|
||||
}{
|
||||
{"amd64", 7.0, "0.19.1"},
|
||||
}
|
||||
|
||||
// vllmCandidates are the vLLM builds for a machine.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
|
||||
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older version (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
// The version asked for, in the build the machine runs: CUDA 13 where the
|
||||
// driver allows, the same version's CUDA 12.9 build where it does not. One
|
||||
// version across the cluster wherever the GPUs allow it, because the flags
|
||||
// llmgw builds are checked against it. Where they do not — GPUs older than
|
||||
// every build of that version — the newest version built for them, said so
|
||||
// beside the installation: vLLM on a V100 at all, rather than not.
|
||||
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
arch, err := gpuArch("vLLM", h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, note := version, ""
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
|
||||
older := ""
|
||||
for _, n := range vllmNewestFor {
|
||||
if n.arch == h.Arch && cc >= n.from {
|
||||
older = n.version
|
||||
}
|
||||
}
|
||||
if older == "" {
|
||||
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
|
||||
"the node serves llama.cpp models", cc, arch)
|
||||
}
|
||||
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
|
||||
}
|
||||
pypi := func(variant string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + v},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
Note: note,
|
||||
}
|
||||
if v != version {
|
||||
b.Version = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
minCC := vllmMinCompute(v, h.Arch)
|
||||
if versionBefore(v, vllmCUDA13Since) {
|
||||
// PyPI's own build is CUDA 12, and is the one there is; asked for
|
||||
// cu129, this is it.
|
||||
if variant == "cu129" {
|
||||
variant = "cu12"
|
||||
}
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
|
||||
}, h)
|
||||
}
|
||||
cu129 := pypi("cu129")
|
||||
cu129.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
cu129.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// gpuArch says whether a Python GPU engine is published for this machine
|
||||
// and whether it has an NVIDIA GPU at all.
|
||||
func gpuArch(engine string, h discovery.Host) (string, error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
@@ -201,33 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if h.NVIDIADriver == "" {
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if engine == "vLLM" {
|
||||
minCC = minComputeFor(version, h.Arch)
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return variant, nil
|
||||
return h.Arch, nil
|
||||
}
|
||||
|
||||
// olderHint says which version runs on a GPU too old for the one asked for.
|
||||
func olderHint(engine string, cc float64) string {
|
||||
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
|
||||
return "; the last vLLM built for it is " + voltaUntil +
|
||||
": give these nodes that version with engine_overrides in the server's configuration"
|
||||
// --- SGLang ---
|
||||
|
||||
// sglangCUDA13Since is the first SGLang built for CUDA 13.
|
||||
const sglangCUDA13Since = "0.5"
|
||||
|
||||
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("SGLang", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ""
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
needs := gpuNeeds{minDriverForCUDA13, 7.5}
|
||||
if versionBefore(version, sglangCUDA13Since) {
|
||||
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
|
||||
}
|
||||
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -251,146 +408,6 @@ func versionBefore(a, b string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for the drivers and GPUs CUDA 13 left behind.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// chooseVLLM picks the vLLM build for a machine.
|
||||
//
|
||||
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
|
||||
// same version's CUDA 12.9 build where only that does: one version across
|
||||
// the cluster, which matters because the flags llmgw builds are checked
|
||||
// against one version, and an older vLLM on the older cards would be a
|
||||
// second set of flags nobody checked. A variant chosen for the node is that
|
||||
// build, if the node can run it.
|
||||
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
// What no build of it gets past: a machine it is not published for, or
|
||||
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
|
||||
// CUDA 12.9 build.
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
pypi := func(v string) Build {
|
||||
return Build{
|
||||
Variant: v,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err == nil {
|
||||
return pypi(v), nil
|
||||
}
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) || !modern {
|
||||
return Build{}, err
|
||||
}
|
||||
b, err129 := cu129()
|
||||
if err129 != nil {
|
||||
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
|
||||
}
|
||||
return b, nil
|
||||
case "cu13":
|
||||
if !modern {
|
||||
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi(v), nil
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) {
|
||||
return Build{}, err
|
||||
}
|
||||
}
|
||||
return cu129()
|
||||
}
|
||||
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
|
||||
}
|
||||
|
||||
// cudaFits checks a node against a CUDA's driver and GPU requirements.
|
||||
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
|
||||
if chosen != "" {
|
||||
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
|
||||
}
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// leadingInt is the number a version string starts with: 580 of "580.65.06".
|
||||
func leadingInt(s string) int {
|
||||
n := 0
|
||||
@@ -403,17 +420,30 @@ func leadingInt(s string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
|
||||
// carries a matching cudart archive, so the node needs a driver and nothing
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
// --- llama.cpp ---
|
||||
|
||||
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
// name decides nothing here. The build is tried everywhere, and ldd says
|
||||
// whether this system can load it — in the words of the missing symbol.
|
||||
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
|
||||
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
|
||||
// carries as PTX, which a driver older than the CUDA it came from cannot
|
||||
// compile.
|
||||
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
|
||||
|
||||
// llamaCppCandidates are the llama.cpp builds for a machine.
|
||||
//
|
||||
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
|
||||
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
|
||||
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
|
||||
// Vulkan, which runs on any card whose driver has Vulkan — the generations
|
||||
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
|
||||
// a GPU placement served on the CPU, which is what installing checks for.
|
||||
// The CPU build where there is no NVIDIA GPU.
|
||||
//
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them is
|
||||
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
|
||||
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
|
||||
// nothing here. The build is tried everywhere, and ldd says whether this
|
||||
// system can load it — in the words of the missing symbol.
|
||||
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -421,36 +451,53 @@ func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
case "arm64":
|
||||
arch = "arm64"
|
||||
default:
|
||||
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
|
||||
b := Build{
|
||||
Variant: "cpu",
|
||||
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
|
||||
// Everything else the build links is in the archive or in any current
|
||||
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
|
||||
// the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the best
|
||||
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
// The CUDA build is published for x64 at this version; on arm64 it is a
|
||||
// newer CUDA that the driver may not support, so an arm64 GPU node runs
|
||||
// the CPU build until someone decides otherwise.
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
base := func(variant string, assets ...string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Assets: assets,
|
||||
// Everything else the build links is in the archive or in any
|
||||
// current glibc system; OpenMP's runtime is the one a cloud image
|
||||
// lacks, and the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the
|
||||
// best one, down to SSE4.2. Below AVX2 it runs, several times
|
||||
// slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
return b
|
||||
}
|
||||
return b, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return pick("llama.cpp", variant, []candidate{
|
||||
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
|
||||
}, h)
|
||||
}
|
||||
cuda := func(cv string) Build {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
|
||||
// The release carries the CUDA runtime beside the build, so the node
|
||||
// needs a driver and nothing else — no toolkit.
|
||||
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
return b
|
||||
}
|
||||
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
|
||||
vulkan.GPUBackend = "libggml-vulkan.so"
|
||||
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
|
||||
|
||||
cuda12Driver := 570
|
||||
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
|
||||
cuda12Driver = minDriverForCUDA12
|
||||
}
|
||||
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
|
||||
if arch == "x64" {
|
||||
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
|
||||
}
|
||||
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
|
||||
return pick("llama.cpp", variant, cands, h)
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
|
||||
@@ -22,8 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
|
||||
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
|
||||
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
|
||||
"gets 0.19.1, the newest built for it, by itself.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
|
||||
@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
|
||||
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
|
||||
var b provision.Build
|
||||
if err == nil {
|
||||
// The first the agent would try; it moves to the next by itself if
|
||||
// that one does not work here.
|
||||
b = builds[0]
|
||||
if b.Version != "" {
|
||||
v = b.Version
|
||||
}
|
||||
}
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
|
||||
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
|
||||
note := ""
|
||||
if b.Note != "" {
|
||||
note = "; " + b.Note
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
|
||||
}
|
||||
|
||||
func rank(l Level) int {
|
||||
|
||||
@@ -157,9 +157,12 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later, and is told which has.
|
||||
// A V100 has no build of vLLM 0.20 or later; it gets the newest
|
||||
// that has Volta kernels, and doctor says which.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
|
||||
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
|
||||
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
|
||||
@@ -52,7 +52,7 @@ func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "CUDA 12.9 installing") {
|
||||
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
|
||||
Reference in new issue
Block a user