Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
86e9a1cb83 |
No files matched your search
+19
-12
@@ -126,11 +126,19 @@ leaving the model with nothing. Now:
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) and the GPUs (Turing+) allow, and otherwise the same
|
||||
version's CUDA 12.9 build from vLLM's GitHub release, pinned to GitHub's
|
||||
digest, with PyTorch's CUDA 12.9 torch (driver 525+, Volta+). One version
|
||||
across the cluster, so the flags llmgw builds — checked against that version
|
||||
— hold on every node.
|
||||
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
|
||||
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
|
||||
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
|
||||
llmgw builds — checked against that version — hold on every node.
|
||||
|
||||
**Volta (V100) is not covered by either build.** Read from vLLM's release
|
||||
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
|
||||
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
|
||||
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
|
||||
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
|
||||
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
|
||||
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
|
||||
on such a node once it is installed.
|
||||
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
@@ -154,13 +162,12 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
or labels a different version of an engine, or none: vLLM 0.19.1, the last
|
||||
built for Volta, on V100s beside the current one on newer cards.
|
||||
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
|
||||
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
|
||||
had been CUDA 13's for every version, so the doctor's advice to set an older
|
||||
version could not work. A refusal now says which version would.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
+16
-11
@@ -376,13 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for the CUDA 13 build of vLLM 0.26 and later, from PyPI. A node that cannot
|
||||
run it — an older driver, a V100 — gets **the same version built for CUDA
|
||||
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
|
||||
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
|
||||
0.20 on. A node with an older driver gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer, Volta
|
||||
(7.0) or newer. Either way the install ends by computing something on the
|
||||
node's GPU, and a build that cannot fails to install, saying why;
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
|
||||
Either way the install ends by computing something on the node's GPU, and a
|
||||
build that cannot fails to install, saying why;
|
||||
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
|
||||
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
|
||||
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
|
||||
it is installed, since llmgw's flags are checked against the current one;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -427,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
|
||||
engines: { vllm: "0.19.1" } # the last one that does
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1269,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
|
||||
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
|
||||
that version with `engine_overrides` (step 4) — its flags are not the ones
|
||||
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -91,17 +91,22 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s's digest", b.Wheel.Asset)
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sum, ok := sums[b.Wheel.Asset]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s %s publishes no %s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Asset)
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + b.Wheel.Asset
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
|
||||
@@ -143,7 +143,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.0 or newer"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
|
||||
// vLLM 0.20 and later have no Volta kernels in any build; the
|
||||
// refusal names the last version that has.
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -195,15 +198,16 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderGPUGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A V100: CUDA 13 dropped it. The same vLLM's CUDA 12.9 build, from the
|
||||
// project's GitHub release, with the torch PyTorch publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a V100 node: %+v", got[0])
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
@@ -268,3 +272,17 @@ func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
|
||||
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
b, err := chooseVLLM(voltaUntil, "", v100)
|
||||
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
|
||||
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
|
||||
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -66,9 +66,10 @@ type Build struct {
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset.
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Asset string
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
@@ -140,30 +141,56 @@ func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
|
||||
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
|
||||
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
|
||||
// compatibility; the oldest GPU one has kernels for is minComputeFor's
|
||||
// answer, which is Volta only up to voltaUntil.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
// cuda13Since is the first version of each Python engine whose PyPI build is
|
||||
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
|
||||
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
|
||||
|
||||
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
|
||||
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
|
||||
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
|
||||
const voltaUntil = "0.19.1"
|
||||
|
||||
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
|
||||
// on an architecture: what the project compiled, not what CUDA supports.
|
||||
func minComputeFor(version, arch string) float64 {
|
||||
if arch == "arm64" {
|
||||
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
|
||||
// since.
|
||||
if versionBefore(version, "0.20") {
|
||||
return 8.7
|
||||
}
|
||||
return 8.0
|
||||
}
|
||||
if versionBefore(version, "0.20") {
|
||||
return minComputeForCUDA12
|
||||
}
|
||||
return minComputeForCUDA13
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
|
||||
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older version (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
switch h.Arch {
|
||||
@@ -178,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if engine == "vLLM" {
|
||||
minCC = minComputeFor(version, h.Arch)
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return variant, nil
|
||||
}
|
||||
|
||||
// olderHint says which version runs on a GPU too old for the one asked for.
|
||||
func olderHint(engine string, cc float64) string {
|
||||
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
|
||||
return "; the last vLLM built for it is " + voltaUntil +
|
||||
": give these nodes that version with engine_overrides in the server's configuration"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -262,19 +297,22 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeForCUDA12, h); err != nil {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Asset: fmt.Sprintf("vllm-%s+cu129-cp38-abi3-manylinux_2_28_%s.whl", version, arch),
|
||||
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
@@ -301,7 +339,7 @@ func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.26 PyPI's own build is CUDA 12, and is the one given.
|
||||
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
@@ -326,8 +364,8 @@ func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h disc
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f",
|
||||
engine, version, cuda, minCC, cc)
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -22,7 +22,8 @@ func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it.\n\n" +
|
||||
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
|
||||
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
|
||||
@@ -154,11 +154,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// What CUDA 13 cannot run, its CUDA 12.9 build of the same version can.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later, and is told which has.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
|
||||
Reference in new issue
Block a user