Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
43d7c4eac1 | ||
|
|
86e9a1cb83 |
No files matched your search
+32
-17
@@ -34,6 +34,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
@@ -68,8 +73,10 @@ file of HTML.
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page is refused as a source**, saying to mirror
|
||||
it or to give the repository id for vLLM to fetch itself.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
@@ -126,11 +133,19 @@ leaving the model with nothing. Now:
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
|
||||
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
|
||||
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
|
||||
across the cluster, so the flags llmgw builds — checked against that version
|
||||
— hold on every node.
|
||||
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
|
||||
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
|
||||
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
|
||||
llmgw builds — checked against that version — hold on every node.
|
||||
|
||||
**Volta (V100) is not covered by either build.** Read from vLLM's release
|
||||
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
|
||||
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
|
||||
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
|
||||
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
|
||||
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
|
||||
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
|
||||
on such a node once it is installed.
|
||||
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
@@ -154,13 +169,12 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
or labels a different version of an engine, or none: vLLM 0.19.1, the last
|
||||
built for Volta, on V100s beside the current one on newer cards.
|
||||
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
|
||||
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
|
||||
had been CUDA 13's for every version, so the doctor's advice to set an older
|
||||
version could not work. A refusal now says which version would.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
@@ -194,9 +208,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||||
`nodes` above one.
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
|
||||
+16
-11
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
|
||||
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
|
||||
0.20 on. A node with an older driver gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
|
||||
Either way the install ends by computing something on the node's GPU, and a
|
||||
build that cannot fails to install, saying why;
|
||||
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
|
||||
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
|
||||
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
|
||||
it is installed, since llmgw's flags are checked against the current one;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -427,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
|
||||
engines: { vllm: "0.19.1" } # the last one that does
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
|
||||
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
|
||||
that version with `engine_overrides` (step 4) — its flags are not the ones
|
||||
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -314,6 +314,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
|
||||
@@ -91,17 +91,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, ok := sums[b.Wheel.Asset]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + b.Wheel.Asset
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
|
||||
@@ -143,7 +143,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
|
||||
// vLLM 0.20 and later have no Volta kernels in any build; the
|
||||
// refusal names the last version that has.
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -195,15 +198,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
|
||||
// project's GitHub release, with the torch PyTorch publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a V100 node: %+v", got[0])
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
@@ -268,3 +272,17 @@ func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
|
||||
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
b, err := chooseVLLM(voltaUntil, "", v100)
|
||||
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
|
||||
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
|
||||
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -66,9 +66,10 @@ type Build struct {
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset.
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Asset string
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
@@ -140,30 +141,56 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
|
||||
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
|
||||
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
|
||||
// compatibility; the oldest GPU one has kernels for is minComputeFor's
|
||||
// answer, which is Volta only up to voltaUntil.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
// cuda13Since is the first version of each Python engine whose PyPI build is
|
||||
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
|
||||
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
|
||||
|
||||
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
|
||||
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
|
||||
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
|
||||
const voltaUntil = "0.19.1"
|
||||
|
||||
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
|
||||
// on an architecture: what the project compiled, not what CUDA supports.
|
||||
func minComputeFor(version, arch string) float64 {
|
||||
if arch == "arm64" {
|
||||
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
|
||||
// since.
|
||||
if versionBefore(version, "0.20") {
|
||||
return 8.7
|
||||
}
|
||||
return 8.0
|
||||
}
|
||||
if versionBefore(version, "0.20") {
|
||||
return minComputeForCUDA12
|
||||
}
|
||||
return minComputeForCUDA13
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
|
||||
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older version (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
switch h.Arch {
|
||||
@@ -178,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if engine == "vLLM" {
|
||||
minCC = minComputeFor(version, h.Arch)
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return variant, nil
|
||||
}
|
||||
|
||||
// olderHint says which version runs on a GPU too old for the one asked for.
|
||||
func olderHint(engine string, cc float64) string {
|
||||
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
|
||||
return "; the last vLLM built for it is " + voltaUntil +
|
||||
": give these nodes that version with engine_overrides in the server's configuration"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -262,19 +297,22 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
|
||||
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
@@ -301,7 +339,7 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
|
||||
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
@@ -326,8 +364,8 @@ func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h disc
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
|
||||
engine, version, cuda, minCC, cc)
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it.\n\n" +
|
||||
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
|
||||
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -335,9 +352,11 @@ func (m Model) validate() error {
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
|
||||
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
@@ -398,8 +417,9 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
|
||||
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp is a width llama.cpp has no way to use; refused, saying what to
|
||||
// write instead.
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
_, err := New([]Model{model("x", p)})
|
||||
if err == nil {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
|
||||
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsNotASource(t *testing.T) {
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
_, err := New([]Model{m})
|
||||
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
|
||||
t.Errorf("%s: %v", src, err)
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
m := model("file")
|
||||
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{m}); err != nil {
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -154,11 +154,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later, and is told which has.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
|
||||
Reference in new issue
Block a user