Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc483d8c68 | ||
|
|
43d7c4eac1 | ||
|
|
86e9a1cb83 |
No files matched your search
+53
-26
@@ -34,6 +34,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
@@ -68,8 +73,10 @@ file of HTML.
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page is refused as a source**, saying to mirror
|
||||
it or to give the repository id for vLLM to fetch itself.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
@@ -123,23 +130,45 @@ leaving the model with nothing. Now:
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
### Every GPU node gets engine builds it can run, by itself
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
|
||||
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
|
||||
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
|
||||
across the cluster, so the flags llmgw builds — checked against that version
|
||||
— hold on every node.
|
||||
Which build of an engine a node installs follows its hardware — the CPU
|
||||
architecture, the driver, and the oldest GPU on it — from what each project
|
||||
publishes and how it was compiled. Each engine has a list of builds, best
|
||||
first; the node takes the first its hardware meets, and the first of those
|
||||
that installs and is seen to compute on the GPU. A build the table is wrong
|
||||
about costs its download, and the next one is tried.
|
||||
|
||||
| | llama.cpp | vLLM |
|
||||
|---|---|---|
|
||||
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
|
||||
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
|
||||
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
|
||||
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
|
||||
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
|
||||
| No NVIDIA GPU | the CPU build | none |
|
||||
|
||||
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
|
||||
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
|
||||
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
|
||||
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
|
||||
starts at 7.5.
|
||||
|
||||
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
|
||||
column shows the version installed, and the installation's detail says why
|
||||
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
|
||||
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
|
||||
vLLM is not placed on a V100 by accident.
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
imports and prints its version, and failed only on its first model; now it
|
||||
fails to install, saying so.
|
||||
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
|
||||
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
|
||||
to the node at once.
|
||||
operation for vLLM, the devices llama.cpp lists for its GPU builds.
|
||||
- **The build can still be chosen per node** on the cluster page (auto, CUDA
|
||||
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
|
||||
at once.
|
||||
- **Whether an engine computes on its GPUs is read from every process
|
||||
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
|
||||
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
|
||||
`CUDA_VISIBLE_DEVICES`.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
@@ -154,13 +183,10 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
or labels a different version of an engine, or none — for exceptions; the
|
||||
build each GPU generation needs is chosen without it (above).
|
||||
- **The requirements follow the version**, as the table above sets out; the
|
||||
threshold had been CUDA 13's for every version.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
@@ -194,9 +220,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||||
`nodes` above one.
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
|
||||
+18
-13
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
|
||||
| Engine | On | How |
|
||||
|---|---|---|
|
||||
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
|
||||
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
|
||||
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
|
||||
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
|
||||
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
|
||||
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
|
||||
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
|
||||
kernels — and the cluster page shows it running that version; run `llmgw
|
||||
ctl engine-check vllm` there once, since llmgw's flags are checked against
|
||||
the current one. Pascal and older run no vLLM. Whatever is chosen, the
|
||||
install ends by computing something on the GPU; a build that cannot is
|
||||
passed over for the next;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -427,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: a10g # pin a class to a version you have tested
|
||||
engines: { vllm: "0.29.0" }
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
|
||||
no build from 0.20 on has Volta kernels, and the node picks the newest that
|
||||
has. llmgw's flags are checked against the current version, so run `llmgw
|
||||
ctl engine-check vllm` on a V100 node once.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -314,6 +314,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strconv"
|
||||
@@ -10,10 +11,14 @@ import (
|
||||
|
||||
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
|
||||
// nvidia-smi index: what a process actually computes on, as opposed to what it
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
|
||||
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
|
||||
// backend did not load runs on the CPU without a word, and this is where that
|
||||
// shows — the process is on no GPU's list.
|
||||
//
|
||||
// From the full query rather than --query-compute-apps, which lists compute
|
||||
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
|
||||
// would have read as not on its GPU at all.
|
||||
//
|
||||
// A node with no nvidia-smi returns nil and no error, as discovery does.
|
||||
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
if binary == "" {
|
||||
@@ -22,44 +27,38 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
}
|
||||
binary = "nvidia-smi"
|
||||
}
|
||||
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseGPUProcesses(string(gpus), string(apps)), nil
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
|
||||
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
|
||||
func parseGPUProcesses(gpus, apps string) map[int][]int {
|
||||
index := map[string]int{}
|
||||
for _, line := range strings.Split(gpus, "\n") {
|
||||
i, uuid, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
n, err := strconv.Atoi(strings.TrimSpace(i))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
index[strings.TrimSpace(uuid)] = n
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
var log smiLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for _, line := range strings.Split(apps, "\n") {
|
||||
uuid, p, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
for i, g := range log.GPUs {
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
}
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(p))
|
||||
i, known := index[strings.TrimSpace(uuid)]
|
||||
if err != nil || !known {
|
||||
continue
|
||||
}
|
||||
out[i] = append(out[i], pid)
|
||||
}
|
||||
return out
|
||||
return out, nil
|
||||
}
|
||||
@@ -5,17 +5,34 @@ import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
|
||||
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
|
||||
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
|
||||
// heard of, which is dropped rather than guessed at.
|
||||
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
|
||||
got := parseGPUProcesses(gpus, apps)
|
||||
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
|
||||
report := `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>580.65.06</driver_version>
|
||||
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes></processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
|
||||
<processes>
|
||||
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
|
||||
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:B3:00.0"><processes>
|
||||
<process_info><pid>4200</pid><type>C+G</type></process_info>
|
||||
</processes></gpu>
|
||||
</nvidia_smi_log>`
|
||||
got, err := parseGPUProcesses([]byte(report))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
|
||||
// same; a compute-only query would have missed it.
|
||||
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %v, want %v", got, want)
|
||||
}
|
||||
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
|
||||
t.Errorf("no compute apps should be no GPU in use: %v", got)
|
||||
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
|
||||
t.Error("a report that does not parse was read as no processes")
|
||||
}
|
||||
}
|
||||
@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLAMA_API_KEY"] = s.APIKey
|
||||
}
|
||||
// The Vulkan build — what a GPU older than the CUDA builds gets — does
|
||||
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
|
||||
}}, nil
|
||||
|
||||
@@ -143,7 +143,7 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
|
||||
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
@@ -152,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
}
|
||||
return st
|
||||
}
|
||||
st.Variant = build.Variant
|
||||
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
|
||||
|
||||
// Installed before: check it still runs, and trust nothing else about it.
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
|
||||
// Installed before: whichever candidate is, best first, checked that it
|
||||
// still runs and trusted for nothing else.
|
||||
for _, build := range builds {
|
||||
dir := in.dirFor(w, build)
|
||||
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
|
||||
continue
|
||||
}
|
||||
st := st.as(w, build)
|
||||
// The packages are checked too. Removing one is exactly the kind of
|
||||
// change an unrelated upgrade makes, and ldd below would catch it only
|
||||
// as a missing library with no hint of which package held it.
|
||||
// change an unrelated upgrade makes, and ldd below would catch it
|
||||
// only as a missing library with no hint of which package held it.
|
||||
if err := in.packages(ctx, build.Packages, &st); err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
@@ -169,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
in.log().Warn("an installed engine no longer runs; installing it again",
|
||||
"engine", w.Engine, "version", w.Version, "err", err)
|
||||
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
|
||||
break
|
||||
}
|
||||
|
||||
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
|
||||
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
|
||||
return st
|
||||
// Each candidate in turn until one installs and is seen to work: a build
|
||||
// the table was wrong about — a driver too old for it after all, a GPU
|
||||
// it has no kernels for — costs its download, and the next is tried.
|
||||
var tried []string
|
||||
for _, build := range builds {
|
||||
st := st.as(w, build)
|
||||
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
in.log().Warn("an engine build did not work here; trying the next",
|
||||
"engine", w.Engine, "variant", build.Variant, "err", err)
|
||||
continue
|
||||
}
|
||||
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
|
||||
if err != nil {
|
||||
tried = append(tried, build.Variant+": "+err.Error())
|
||||
continue
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
}
|
||||
bin, version, err := in.verify(ctx, dir, build)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Failed, err.Error()
|
||||
return st
|
||||
st.State, st.Binary = Failed, ""
|
||||
st.Detail = strings.Join(tried, "; then ")
|
||||
if len(builds) == 1 {
|
||||
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
|
||||
}
|
||||
return in.ready(st, build, bin, version)
|
||||
return st
|
||||
}
|
||||
|
||||
// dirFor is where a build of an engine is installed: by the version it is
|
||||
// and its variant, so two builds of one version sit side by side.
|
||||
func (in *Installer) dirFor(w Want, b Build) string {
|
||||
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
|
||||
}
|
||||
|
||||
// versionOr is the version a build is: its own, when it is not the one asked
|
||||
// for.
|
||||
func (b Build) versionOr(asked string) string {
|
||||
if b.Version != "" {
|
||||
return b.Version
|
||||
}
|
||||
return asked
|
||||
}
|
||||
|
||||
// as is a status for one build of what was asked for.
|
||||
func (st Status) as(w Want, b Build) Status {
|
||||
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
|
||||
return st
|
||||
}
|
||||
|
||||
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
|
||||
st.State, st.Binary = Ready, bin
|
||||
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
|
||||
if b.Note != "" {
|
||||
st.Detail += "; " + b.Note
|
||||
}
|
||||
var missing []string
|
||||
for _, f := range b.CPUFlags {
|
||||
if !in.Host.HasCPUFlag(f) {
|
||||
|
||||
@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", "", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
got, err := llamaCppCandidates("b1", "", gpu)
|
||||
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
|
||||
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
host discovery.Host
|
||||
want []string
|
||||
}{
|
||||
// Driver 580 and Turing: CUDA 13.3 first.
|
||||
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
|
||||
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
|
||||
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
|
||||
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
|
||||
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
|
||||
// A Pascal card: CUDA 12.8, not 13.3.
|
||||
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
|
||||
// arm64 with a GPU: never the CPU build.
|
||||
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
|
||||
} {
|
||||
got, err := llamaCppCandidates("b1", "", c.host)
|
||||
var names []string
|
||||
for _, b := range got {
|
||||
names = append(names, b.Variant)
|
||||
}
|
||||
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
|
||||
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
|
||||
}
|
||||
}
|
||||
|
||||
slow := ubuntu
|
||||
slow.CPUFlags = nil
|
||||
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
|
||||
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
|
||||
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
|
||||
// loader looks, and llama-server serving a GPU placement on the CPU.
|
||||
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
|
||||
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
|
||||
rel := newRelease(map[string][]byte{
|
||||
"llama-b1-" + cuda: tarGz(t, map[string]string{
|
||||
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
|
||||
|
||||
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
|
||||
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
|
||||
defer cancel()
|
||||
// The version this build is, which for a GPU older than every build of
|
||||
// the one asked for is an older one: its publication date is the one
|
||||
// the dependencies are resolved as of.
|
||||
w.Version = b.versionOr(w.Version)
|
||||
|
||||
if err := in.packages(ctx, b.Packages, st); err != nil {
|
||||
return err
|
||||
@@ -91,17 +95,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, ok := sums[b.Wheel.Asset]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + b.Wheel.Asset
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
|
||||
@@ -25,8 +25,9 @@ type fakePython struct {
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does.
|
||||
noKernel bool
|
||||
// card does: in every build, or in noKernelIn only ("-cu13.").
|
||||
noKernel bool
|
||||
noKernelIn string
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -48,7 +49,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel {
|
||||
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
@@ -77,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
|
||||
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
|
||||
})
|
||||
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
|
||||
})
|
||||
pypi := httptest.NewServer(mux)
|
||||
t.Cleanup(pypi.Close)
|
||||
return &Installer{
|
||||
@@ -143,7 +147,8 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
|
||||
// No vLLM was ever built for Pascal.
|
||||
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -195,15 +200,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
|
||||
// project's GitHub release, with the torch PyTorch publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a V100 node: %+v", got[0])
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
@@ -236,11 +242,7 @@ func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
|
||||
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
|
||||
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
@@ -260,11 +262,51 @@ func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
|
||||
got, err := vllmCandidates("0.30.0", "", gpuNode)
|
||||
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
|
||||
t.Errorf("candidates on a current node: %+v, %v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
|
||||
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
|
||||
// 0.19.1, the newest that has, and says so.
|
||||
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, v100, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
|
||||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
|
||||
t.Fatalf("a V100: %+v", got)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
|
||||
t.Errorf("not installed as what it is: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
|
||||
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
|
||||
}
|
||||
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
|
||||
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
|
||||
t.Error("a CUDA 13 build was offered to a V100")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
|
||||
// The table says CUDA 13 runs here; the card says otherwise. The next
|
||||
// build is installed rather than the node going without.
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
|
||||
if got.State != Ready || got.Variant != "cu129" {
|
||||
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
|
||||
}
|
||||
}
|
||||
+309
-224
@@ -10,7 +10,6 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -60,15 +59,23 @@ type Build struct {
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// Version is the engine version this build is, when it is not the one
|
||||
// asked for: the newest built for this machine's GPUs, where the one
|
||||
// asked for has no kernels for them. Empty is the version asked for.
|
||||
Version string
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset.
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Asset string
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
@@ -78,10 +85,12 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits. variant
|
||||
// is a build chosen for the node ("cu129"); empty leaves it to the
|
||||
// machine.
|
||||
Choose func(version, variant string, h discovery.Host) (Build, error)
|
||||
// Candidates are the builds that may run on a machine, best first, or
|
||||
// why none can. The installer takes the first that installs and is seen
|
||||
// to work — computing on the GPU, where there is one — so a build this
|
||||
// table is wrong about costs a download, not a node. variant is a build
|
||||
// chosen for the node; empty leaves the choice to the machine.
|
||||
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -90,10 +99,10 @@ type Recipe struct {
|
||||
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
|
||||
// checked against its published checksum.
|
||||
var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
|
||||
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
|
||||
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
|
||||
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
// Installable are the engines this agent can install, as it tells the
|
||||
@@ -123,7 +132,15 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
|
||||
b, err := chooseUV(version, h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return []Build{b}, nil
|
||||
}
|
||||
|
||||
func chooseUV(_ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -140,32 +157,190 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
// What builds need of the NVIDIA side of a machine, read from what each
|
||||
// project publishes and how it was built:
|
||||
//
|
||||
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
|
||||
// by CUDA's minor version compatibility — except code shipped only as
|
||||
// PTX, which the driver compiles and so needs a driver as new as the
|
||||
// CUDA it was compiled with.
|
||||
// - Which GPUs a build has kernels for is the build's own list, not what
|
||||
// its CUDA supports.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
minDriverForCUDA13 = 580
|
||||
minDriverForCUDA12 = 525
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
// gpuNeeds is what one build needs of a machine's GPUs and driver.
|
||||
type gpuNeeds struct {
|
||||
driver int
|
||||
cc float64
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
// fits says why a machine cannot run a build, or nil.
|
||||
func (n gpuNeeds) fits(what string, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
|
||||
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
|
||||
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// candidate is a build and what it needs.
|
||||
type candidate struct {
|
||||
b Build
|
||||
needs gpuNeeds
|
||||
what string
|
||||
}
|
||||
|
||||
// pick keeps the candidates a machine can run, in order, and those a
|
||||
// variant chosen for the node names; with none left, it says why each went.
|
||||
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
|
||||
var out []Build
|
||||
var why []string
|
||||
for _, c := range cands {
|
||||
if variant != "" && c.b.Variant != variant {
|
||||
continue
|
||||
}
|
||||
if err := c.needs.fits(c.what, h); err != nil {
|
||||
why = append(why, err.Error())
|
||||
continue
|
||||
}
|
||||
out = append(out, c.b)
|
||||
}
|
||||
switch {
|
||||
case len(out) > 0:
|
||||
return out, nil
|
||||
case len(why) == 0 && variant != "":
|
||||
var names []string
|
||||
for _, c := range cands {
|
||||
names = append(names, c.b.Variant)
|
||||
}
|
||||
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
|
||||
}
|
||||
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
|
||||
}
|
||||
|
||||
// --- vLLM ---
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for drivers CUDA 13 does not run on.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
|
||||
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
|
||||
// "+cu129" beside a CUDA 13 one.
|
||||
const vllmCUDA13Since = "0.20"
|
||||
|
||||
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
|
||||
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
|
||||
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
|
||||
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
|
||||
// 0.20 and 8.0 since.
|
||||
func vllmMinCompute(version, arch string) float64 {
|
||||
old := versionBefore(version, vllmCUDA13Since)
|
||||
switch {
|
||||
case arch == "arm64" && old:
|
||||
return 8.7
|
||||
case arch == "arm64":
|
||||
return 8.0
|
||||
case old:
|
||||
return 7.0
|
||||
}
|
||||
return 7.5
|
||||
}
|
||||
|
||||
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
|
||||
// on an architecture, for GPUs older than every build of the version asked
|
||||
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
|
||||
// for a GPU no vLLM was built for.
|
||||
var vllmNewestFor = []struct {
|
||||
arch string
|
||||
from float64 // the oldest GPU it covers
|
||||
version string
|
||||
}{
|
||||
{"amd64", 7.0, "0.19.1"},
|
||||
}
|
||||
|
||||
// vllmCandidates are the vLLM builds for a machine.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
// The version asked for, in the build the machine runs: CUDA 13 where the
|
||||
// driver allows, the same version's CUDA 12.9 build where it does not. One
|
||||
// version across the cluster wherever the GPUs allow it, because the flags
|
||||
// llmgw builds are checked against it. Where they do not — GPUs older than
|
||||
// every build of that version — the newest version built for them, said so
|
||||
// beside the installation: vLLM on a V100 at all, rather than not.
|
||||
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
arch, err := gpuArch("vLLM", h)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
v, note := version, ""
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
|
||||
older := ""
|
||||
for _, n := range vllmNewestFor {
|
||||
if n.arch == h.Arch && cc >= n.from {
|
||||
older = n.version
|
||||
}
|
||||
}
|
||||
if older == "" {
|
||||
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
|
||||
"the node serves llama.cpp models", cc, arch)
|
||||
}
|
||||
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
|
||||
}
|
||||
pypi := func(variant string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + v},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
Note: note,
|
||||
}
|
||||
if v != version {
|
||||
b.Version = v
|
||||
}
|
||||
return b
|
||||
}
|
||||
minCC := vllmMinCompute(v, h.Arch)
|
||||
if versionBefore(v, vllmCUDA13Since) {
|
||||
// PyPI's own build is CUDA 12, and is the one there is; asked for
|
||||
// cu129, this is it.
|
||||
if variant == "cu129" {
|
||||
variant = "cu12"
|
||||
}
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
|
||||
}, h)
|
||||
}
|
||||
cu129 := pypi("cu129")
|
||||
cu129.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
cu129.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
|
||||
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
|
||||
}
|
||||
cu129.ExtraIndex = pytorchCU129
|
||||
return pick("vLLM "+v, variant, []candidate{
|
||||
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
|
||||
}, h)
|
||||
}
|
||||
|
||||
// gpuArch says whether a Python GPU engine is published for this machine
|
||||
// and whether it has an NVIDIA GPU at all.
|
||||
func gpuArch(engine string, h discovery.Host) (string, error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
@@ -174,25 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if h.NVIDIADriver == "" {
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
}
|
||||
return variant, nil
|
||||
return h.Arch, nil
|
||||
}
|
||||
|
||||
// --- SGLang ---
|
||||
|
||||
// sglangCUDA13Since is the first SGLang built for CUDA 13.
|
||||
const sglangCUDA13Since = "0.5"
|
||||
|
||||
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("SGLang", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b := Build{
|
||||
Variant: "cu13",
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
needs := gpuNeeds{minDriverForCUDA13, 7.5}
|
||||
if versionBefore(version, sglangCUDA13Since) {
|
||||
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
|
||||
}
|
||||
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -216,143 +408,6 @@ func versionBefore(a, b string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
// launcher for its kernels the first time a model loads, and without gcc
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for the drivers and GPUs CUDA 13 left behind.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// chooseVLLM picks the vLLM build for a machine.
|
||||
//
|
||||
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
|
||||
// same version's CUDA 12.9 build where only that does: one version across
|
||||
// the cluster, which matters because the flags llmgw builds are checked
|
||||
// against one version, and an older vLLM on the older cards would be a
|
||||
// second set of flags nobody checked. A variant chosen for the node is that
|
||||
// build, if the node can run it.
|
||||
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
// What no build of it gets past: a machine it is not published for, or
|
||||
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
|
||||
// CUDA 12.9 build.
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
pypi := func(v string) Build {
|
||||
return Build{
|
||||
Variant: v,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err == nil {
|
||||
return pypi(v), nil
|
||||
}
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) || !modern {
|
||||
return Build{}, err
|
||||
}
|
||||
b, err129 := cu129()
|
||||
if err129 != nil {
|
||||
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
|
||||
}
|
||||
return b, nil
|
||||
case "cu13":
|
||||
if !modern {
|
||||
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi(v), nil
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) {
|
||||
return Build{}, err
|
||||
}
|
||||
}
|
||||
return cu129()
|
||||
}
|
||||
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
|
||||
}
|
||||
|
||||
// cudaFits checks a node against a CUDA's driver and GPU requirements.
|
||||
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
|
||||
engine, version, cuda, minCC, cc)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
|
||||
if chosen != "" {
|
||||
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
|
||||
}
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// leadingInt is the number a version string starts with: 580 of "580.65.06".
|
||||
func leadingInt(s string) int {
|
||||
n := 0
|
||||
@@ -365,17 +420,30 @@ func leadingInt(s string) int {
|
||||
return n
|
||||
}
|
||||
|
||||
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
|
||||
// carries a matching cudart archive, so the node needs a driver and nothing
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
// --- llama.cpp ---
|
||||
|
||||
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
// name decides nothing here. The build is tried everywhere, and ldd says
|
||||
// whether this system can load it — in the words of the missing symbol.
|
||||
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
|
||||
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
|
||||
// carries as PTX, which a driver older than the CUDA it came from cannot
|
||||
// compile.
|
||||
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
|
||||
|
||||
// llamaCppCandidates are the llama.cpp builds for a machine.
|
||||
//
|
||||
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
|
||||
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
|
||||
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
|
||||
// Vulkan, which runs on any card whose driver has Vulkan — the generations
|
||||
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
|
||||
// a GPU placement served on the CPU, which is what installing checks for.
|
||||
// The CPU build where there is no NVIDIA GPU.
|
||||
//
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them is
|
||||
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
|
||||
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
|
||||
// nothing here. The build is tried everywhere, and ldd says whether this
|
||||
// system can load it — in the words of the missing symbol.
|
||||
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -383,36 +451,53 @@ func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
case "arm64":
|
||||
arch = "arm64"
|
||||
default:
|
||||
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
|
||||
b := Build{
|
||||
Variant: "cpu",
|
||||
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
|
||||
// Everything else the build links is in the archive or in any current
|
||||
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
|
||||
// the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the best
|
||||
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
// The CUDA build is published for x64 at this version; on arm64 it is a
|
||||
// newer CUDA that the driver may not support, so an arm64 GPU node runs
|
||||
// the CPU build until someone decides otherwise.
|
||||
if h.NVIDIADriver != "" && arch == "x64" {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
|
||||
b.Variant = "cuda-" + cudaForLlamaCpp
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
b.Assets = []string{
|
||||
fmt.Sprintf("llama-%s-%s", version, name),
|
||||
fmt.Sprintf("cudart-llama-%s-%s", version, name),
|
||||
base := func(variant string, assets ...string) Build {
|
||||
b := Build{
|
||||
Variant: variant,
|
||||
Assets: assets,
|
||||
// Everything else the build links is in the archive or in any
|
||||
// current glibc system; OpenMP's runtime is the one a cloud image
|
||||
// lacks, and the server does not start without it.
|
||||
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
|
||||
Binary: "llama-server",
|
||||
}
|
||||
if arch == "x64" {
|
||||
// The archive carries a backend per CPU generation and loads the
|
||||
// best one, down to SSE4.2. Below AVX2 it runs, several times
|
||||
// slower.
|
||||
b.CPUFlags = []string{"avx2"}
|
||||
}
|
||||
return b
|
||||
}
|
||||
return b, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return pick("llama.cpp", variant, []candidate{
|
||||
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
|
||||
}, h)
|
||||
}
|
||||
cuda := func(cv string) Build {
|
||||
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
|
||||
// The release carries the CUDA runtime beside the build, so the node
|
||||
// needs a driver and nothing else — no toolkit.
|
||||
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
|
||||
b.GPUBackend = "libggml-cuda.so"
|
||||
return b
|
||||
}
|
||||
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
|
||||
vulkan.GPUBackend = "libggml-vulkan.so"
|
||||
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
|
||||
|
||||
cuda12Driver := 570
|
||||
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
|
||||
cuda12Driver = minDriverForCUDA12
|
||||
}
|
||||
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
|
||||
if arch == "x64" {
|
||||
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
|
||||
}
|
||||
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
|
||||
return pick("llama.cpp", variant, cands, h)
|
||||
}
|
||||
|
||||
func orUnknown(s string) string {
|
||||
|
||||
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it.\n\n" +
|
||||
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
|
||||
"gets 0.19.1, the newest built for it, by itself.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -335,9 +352,11 @@ func (m Model) validate() error {
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
|
||||
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
@@ -398,8 +417,9 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
|
||||
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp is a width llama.cpp has no way to use; refused, saying what to
|
||||
// write instead.
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
_, err := New([]Model{model("x", p)})
|
||||
if err == nil {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
|
||||
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsNotASource(t *testing.T) {
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
_, err := New([]Model{m})
|
||||
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
|
||||
t.Errorf("%s: %v", src, err)
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
m := model("file")
|
||||
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{m}); err != nil {
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
|
||||
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
|
||||
var b provision.Build
|
||||
if err == nil {
|
||||
// The first the agent would try; it moves to the next by itself if
|
||||
// that one does not work here.
|
||||
b = builds[0]
|
||||
if b.Version != "" {
|
||||
v = b.Version
|
||||
}
|
||||
}
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
|
||||
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
|
||||
note := ""
|
||||
if b.Note != "" {
|
||||
note = "; " + b.Note
|
||||
}
|
||||
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
|
||||
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
|
||||
}
|
||||
|
||||
func rank(l Level) int {
|
||||
|
||||
@@ -154,11 +154,17 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later; it gets the newest
|
||||
// that has Volta kernels, and doctor says which.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
|
||||
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
|
||||
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
|
||||
@@ -52,7 +52,7 @@ func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "CUDA 12.9 installing") {
|
||||
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
|
||||
Reference in new issue
Block a user