Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 43d7c4eac1 Let the upgrade repair what the field's catalog already holds
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
tp on a llama.cpp GPU profile is read as gpus_per_node rather than
refused, and a Hugging Face repository page as a vLLM source as the
repository id, so models written that way are placed after the upgrade
instead of dropped from the catalog. Engines adopted after an upgrade
are checked for computing on their GPUs too.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:20:11 +02:00
Ilker ManapandClaude Opus 5.5 86e9a1cb83 Give V100s the last vLLM built for them, not a build without their kernels
Release / build (push) Successful in 1m13s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m47s
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:04:45 +02:00
10 changed files with 221 additions and 86 deletions

No files matched your search

+32 -17
View File
@@ -34,6 +34,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
own `ldd` could not say it: the backend is opened at run time.
- **After the upgrade**, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
- **Starting** a shard on GPUs is ready only when its process holds memory on
each of them (`nvidia-smi --query-compute-apps`, its process group
included). One that does not is stopped and reported failed: "placed on GPU
@@ -68,8 +73,10 @@ file of HTML.
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
- **`llmgw mirror add` takes several URLs** into one version.
- **A Hugging Face repository page is refused as a source**, saying to mirror
it or to give the repository id for vLLM to fetch itself.
- **A Hugging Face repository page given as a source** is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
such a model, as before.
@@ -126,11 +133,19 @@ leaving the model with nothing. Now:
### Every GPU node gets a vLLM it can run, chosen on the cluster page
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
across the cluster, so the flags llmgw builds — checked against that version
— hold on every node.
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
llmgw builds — checked against that version — hold on every node.
**Volta (V100) is not covered by either build.** Read from vLLM's release
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
on such a node once it is installed.
- **The install computes on the GPU before it counts**: a small tensor
operation in the new environment. A build without kernels for the card
@@ -154,13 +169,12 @@ and `--ignore-checks` passed over every other problem with it.
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
or labels a different version of an engine, or none: vLLM 0.19.1, the last
built for Volta, on V100s beside the current one on newer cards.
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
had been CUDA 13's for every version, so the doctor's advice to set an older
version could not work. A refusal now says which version would.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
@@ -194,9 +208,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
(`--split-mode layer`, in proportion to each card's free memory, which for
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write `gpus_per_node` instead; so is
`nodes` above one.
placed — the replica stayed unplaced or ran on one card. It is now read as
`gpus_per_node`, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
`nodes` above one, is refused.
## v0.10.6 — 2026-10-01
+16 -11
View File
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
run it — an older driver, a V100 — gets **the same version built for CUDA
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
0.20 on. A node with an older driver gets **the same version built for CUDA
12.9** instead, from vLLM's GitHub release (checked against the digest
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
(7.0) or newer. Either way the install ends by computing something on the
node's GPU, and a build that cannot fails to install, saying why;
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
Either way the install ends by computing something on the node's GPU, and a
build that cannot fails to install, saying why;
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
it is installed, since llmgw's flags are checked against the current one;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -427,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
engines: { vllm: "0.19.1" } # the last one that does
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
that version with `engine_overrides` (step 4) — its flags are not the ones
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+11
View File
@@ -314,6 +314,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
return
}
// An engine started by an agent older than the GPU check may be one
// serving a GPU placement on the CPU. Stopped, so its replacement starts
// through the check and fails there in words, if the build still sees
// no GPU — rather than going on unnoticed after the upgrade that could
// have caught it.
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
_ = a.sup.Stop(ctx, o.ShardID)
return
}
a.log.Info("adopted engine is serving",
"shard", o.ShardID, "replica", o.ReplicaID)
}
+10 -5
View File
@@ -91,17 +91,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
// The engine from its GitHub release, pinned to the digest GitHub
// publishes for the file: the same trust as an archive, and uv
// refuses a download that does not match it.
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
if err != nil {
return err
}
sum, ok := sums[b.Wheel.Asset]
if !ok {
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
var asset, sum string
for name, s := range sums {
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
asset, sum = name, s
}
}
if asset == "" {
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
}
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
b.Wheel.Tag + "/" + b.Wheel.Asset
b.Wheel.Tag + "/" + asset
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
}
// A second index, for the torch the engine was built on. Every package
+24 -6
View File
@@ -143,7 +143,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
{"no GPU", ubuntu, Skipped, "has none"},
// Too old for CUDA 13 and for its CUDA 12.9 build both.
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
// vLLM 0.20 and later have no Volta kernels in any build; the
// refusal names the last version that has.
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -195,15 +198,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
}
}
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
// project's GitHub release, with the torch PyTorch publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
// 12.9 build, from the project's GitHub release, with the torch PyTorch
// publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, gh := pythonInstaller(t, volta, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Ready || got[0].Variant != "cu129" {
t.Fatalf("a V100 node: %+v", got[0])
t.Fatalf("a Turing node on driver 550: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
t.Errorf("not installed under its own build's name: %v", err)
@@ -268,3 +272,17 @@ func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
t.Errorf("a node with no GPU chosen cu129: %v", err)
}
}
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
b, err := chooseVLLM(voltaUntil, "", v100)
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
}
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
}
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
}
}
+60 -22
View File
@@ -66,9 +66,10 @@ type Build struct {
FreeGB int
}
// Wheel is a package published as a GitHub release asset.
// Wheel is a package published as a GitHub release asset: the asset whose
// name starts with Prefix and ends with Suffix.
type Wheel struct {
Project, Repo, Tag, Asset string
Project, Repo, Tag, Prefix, Suffix string
}
// Recipe knows how to install one engine.
@@ -140,30 +141,56 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
}, nil
}
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
const minDriverForCUDA13 = 580
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
// Volta, so a V100 cannot run these builds at all.
const minComputeForCUDA13 = 7.5
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
// version compatibility, and on Volta.
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
// compatibility; the oldest GPU one has kernels for is minComputeFor's
// answer, which is Volta only up to voltaUntil.
const (
minDriverForCUDA12 = 525
minComputeForCUDA12 = 7.0
)
// cuda13Since is the first version of each Python engine built for CUDA 13.
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
// cuda13Since is the first version of each Python engine whose PyPI build is
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
const voltaUntil = "0.19.1"
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
// on an architecture: what the project compiled, not what CUDA supports.
func minComputeFor(version, arch string) float64 {
if arch == "arm64" {
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
// since.
if versionBefore(version, "0.20") {
return 8.7
}
return 8.0
}
if versionBefore(version, "0.20") {
return minComputeForCUDA12
}
return minComputeForCUDA13
}
// cudaGate says whether a node can run this version of an engine, and which
// CUDA it is built for.
//
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
// nodes the older build (engine_overrides in the server's configuration)
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
// nodes the older version (engine_overrides in the server's configuration)
// rather than going without vLLM, or without the Volta nodes.
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
switch h.Arch {
@@ -178,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
if versionBefore(version, cuda13Since[engine]) {
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
}
if engine == "vLLM" {
minCC = minComputeFor(version, h.Arch)
}
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
hint := ""
if cuda == "13" {
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
}
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, hint)
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, minCC, cc, olderHint(engine, cc))
}
return variant, nil
}
// olderHint says which version runs on a GPU too old for the one asked for.
func olderHint(engine string, cc float64) string {
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
return "; the last vLLM built for it is " + voltaUntil +
": give these nodes that version with engine_overrides in the server's configuration"
}
return ""
}
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
@@ -262,19 +297,22 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
}
}
cu129 := func() (Build, error) {
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
return Build{}, err
}
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
b := pypi("cu129")
b.Requirements = nil
// Found in the release's assets rather than named: the manylinux tag
// in the file's name changes between versions (2_28, 2_31, 2_34).
b.Wheel = &Wheel{
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
}
b.ExtraIndex = pytorchCU129
return b, nil
}
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
modern := !versionBefore(version, cuda13Since["vLLM"])
switch variant {
case "":
@@ -301,7 +339,7 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
return pypi("cu13"), nil
case "cu129":
if !modern {
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
v, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
@@ -326,8 +364,8 @@ func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h disc
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
engine, version, cuda, minCC, cc)
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, olderHint(engine, cc))
}
return nil
}
+2 -1
View File
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
" none no vLLM on this node\n\n" +
"The node is told at once and installs it in the background; a build it\n" +
"cannot compute with on its GPU fails to install, saying why, and the\n" +
"cluster page shows it.\n\n" +
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
" llmgw ctl engine-build gpu07 vllm cu129",
Args: cobra.ExactArgs(3),
RunE: func(cmd *cobra.Command, args []string) error {
+25 -5
View File
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
if p.Nodes <= 0 {
p.Nodes = 1
}
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
// anything for it, is read as the width it was meant as: N cards of
// one node. Refusing it would have dropped a catalog's model on the
// upgrade that made the width work. Only both given, disagreeing, is
// refused below, as ambiguous.
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
}
}
switch {
case p.CPUOnly():
// Nothing to divide up: this profile asks for no cards at all.
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
p.Weight = 1
}
}
// A Hugging Face repository page for a model vLLM or SGLang runs is the
// repository it names: the engine downloads a repository id itself,
// where the page was fetched as one file of HTML and failed at start.
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
m.Source = repo
}
return m
}
@@ -335,9 +352,11 @@ func (m Model) validate() error {
}
if repo, ok := repositoryPage(m.Source); ok {
// A page, not weights: a node fetches the HTML as one file and the
// engine fails at start, on a path that looks right.
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
// engine fails at start, on a path that looks right. For vLLM it was
// read as the repository id when the model was made; only a model
// llama.cpp runs is left here, which needs a file.
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
m.Name, m.Source, repo)
}
if err := validateAdapters(m); err != nil {
@@ -398,8 +417,9 @@ func (m Model) validate() error {
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
m.Name, p.Name, p.Nodes)
case p.TP > 1 || p.PP > 1:
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
}
continue
}
+34 -15
View File
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
t.Errorf("placement was not asked for four GPUs: %+v", pl)
}
// tp is a width llama.cpp has no way to use; refused, saying what to
// write instead.
// tp, written before gpus_per_node meant anything for llama.cpp, is read
// as the same width rather than refused: refusing dropped the model on
// the upgrade that made the width work.
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
c, err = New([]Model{old})
if err != nil {
t.Fatalf("tp 4 on llama.cpp: %v", err)
}
got, _ := c.Resolve("old")
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
t.Errorf("tp 4 should place on four GPUs: %+v", p)
}
// Both given and disagreeing is ambiguous, and refused; so is more than
// one node.
for _, p := range []Profile{
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
} {
_, err := New([]Model{model("x", p)})
if err == nil {
if _, err := New([]Model{model("x", p)}); err == nil {
t.Errorf("%s: accepted", p.Name)
}
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
}
}
}
func TestARepositoryPageIsNotASource(t *testing.T) {
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
// The field: an AWQ model's page given as a vLLM source, fetched as one
// file of HTML. vLLM downloads a repository id itself.
for _, src := range []string{
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
} {
m := model("awq")
m.Source = src
_, err := New([]Model{m})
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
t.Errorf("%s: %v", src, err)
c, err := New([]Model{m})
if err != nil {
t.Fatalf("%s: %v", src, err)
}
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
t.Errorf("%s became %q", src, got.Source)
}
}
m := model("file")
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
if _, err := New([]Model{m}); err != nil {
// llama.cpp needs a file, which a page is not.
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
g.Source = "https://huggingface.co/Qwen/X-GGUF"
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
t.Errorf("a page for llama.cpp: %v", err)
}
f := model("file")
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
if _, err := New([]Model{f}); err != nil {
t.Errorf("a file under /resolve/ is a source: %v", err)
}
}
+7 -4
View File
@@ -154,11 +154,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
// What vLLM cannot do keeps vLLM off the node, not the node out of
// the cluster: it still serves llama.cpp.
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
// A V100 has no build of vLLM 0.20 or later, and is told which has.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},