Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
43d7c4eac1 | ||
|
|
86e9a1cb83 | ||
|
|
4f280a3546 |
No files matched your search
+45
-12
@@ -34,6 +34,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
|
||||
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
|
||||
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
|
||||
own `ldd` could not say it: the backend is opened at run time.
|
||||
- **After the upgrade**, engines the agent finds already running are held to
|
||||
the same check when it adopts them: one serving a GPU placement on the CPU
|
||||
is stopped, and its replacement starts through the check. The installed
|
||||
CUDA build is checked again within ten minutes, and installed afresh if it
|
||||
sees no GPU.
|
||||
- **Starting** a shard on GPUs is ready only when its process holds memory on
|
||||
each of them (`nvidia-smi --query-compute-apps`, its process group
|
||||
included). One that does not is stopped and reported failed: "placed on GPU
|
||||
@@ -68,8 +73,10 @@ file of HTML.
|
||||
it and the manifest carries it; a node checks every file of a set against
|
||||
it before using the set, and fetches one that differs again.
|
||||
- **`llmgw mirror add` takes several URLs** into one version.
|
||||
- **A Hugging Face repository page is refused as a source**, saying to mirror
|
||||
it or to give the repository id for vLLM to fetch itself.
|
||||
- **A Hugging Face repository page given as a source** is read as the
|
||||
repository id for a model vLLM or SGLang runs, which the engine downloads
|
||||
itself (the node then needs to reach Hugging Face), and refused for one
|
||||
llama.cpp runs, which needs a file.
|
||||
|
||||
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
|
||||
such a model, as before.
|
||||
@@ -123,6 +130,32 @@ leaving the model with nothing. Now:
|
||||
stopping — is waited for, as before.
|
||||
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
|
||||
|
||||
### Every GPU node gets a vLLM it can run, chosen on the cluster page
|
||||
|
||||
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
|
||||
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
|
||||
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
|
||||
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
|
||||
llmgw builds — checked against that version — hold on every node.
|
||||
|
||||
**Volta (V100) is not covered by either build.** Read from vLLM's release
|
||||
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
|
||||
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
|
||||
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
|
||||
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
|
||||
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
|
||||
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
|
||||
on such a node once it is installed.
|
||||
|
||||
- **The install computes on the GPU before it counts**: a small tensor
|
||||
operation in the new environment. A build without kernels for the card
|
||||
imports and prints its version, and failed only on its first model; now it
|
||||
fails to install, saying so.
|
||||
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
|
||||
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
|
||||
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
|
||||
to the node at once.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||||
@@ -136,13 +169,12 @@ and `--ignore-checks` passed over every other problem with it.
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
or labels a different version of an engine, or none: vLLM 0.19.1, the last
|
||||
built for Volta, on V100s beside the current one on newer cards.
|
||||
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
|
||||
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
|
||||
had been CUDA 13's for every version, so the doctor's advice to set an older
|
||||
version could not work. A refusal now says which version would.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
@@ -176,9 +208,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
|
||||
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
|
||||
(`--split-mode layer`, in proportion to each card's free memory, which for
|
||||
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
|
||||
placed — the replica stayed unplaced or ran on one card — and is now refused
|
||||
when the model is added, saying to write `gpus_per_node` instead; so is
|
||||
`nodes` above one.
|
||||
placed — the replica stayed unplaced or ran on one card. It is now read as
|
||||
`gpus_per_node`, so a profile written that way is placed on that many cards
|
||||
after the upgrade, rather than dropped; only both given and disagreeing, or
|
||||
`nodes` above one, is refused.
|
||||
|
||||
## v0.10.6 — 2026-10-01
|
||||
|
||||
|
||||
@@ -2370,9 +2370,12 @@ func (x *Provision) GetEngines() []*EngineWant {
|
||||
|
||||
// EngineWant names one engine at one version.
|
||||
type EngineWant struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
|
||||
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -2421,6 +2424,13 @@ func (x *EngineWant) GetVersion() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *EngineWant) GetVariant() string {
|
||||
if x != nil {
|
||||
return x.Variant
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
type ProvisionStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
@@ -4068,11 +4078,12 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
|
||||
"\tProvision\x12.\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
|
||||
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
|
||||
"\n" +
|
||||
"EngineWant\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
|
||||
"\x0fProvisionStatus\x12\x16\n" +
|
||||
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
|
||||
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
|
||||
|
||||
@@ -465,6 +465,9 @@ message Provision {
|
||||
message EngineWant {
|
||||
string engine = 1; // "llamacpp"
|
||||
string version = 2; // "b10991"
|
||||
// variant is the build to install, when one is chosen for the node rather
|
||||
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
|
||||
string variant = 3;
|
||||
}
|
||||
|
||||
// ProvisionStatus is where one engine's installation stands on a node.
|
||||
|
||||
+30
-9
@@ -376,10 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
|
||||
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
|
||||
Volta (7.0);
|
||||
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
|
||||
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
|
||||
0.20 on. A node with an older driver gets **the same version built for CUDA
|
||||
12.9** instead, from vLLM's GitHub release (checked against the digest
|
||||
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
|
||||
Either way the install ends by computing something on the node's GPU, and a
|
||||
build that cannot fails to install, saying why;
|
||||
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
|
||||
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
|
||||
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
|
||||
it is installed, since llmgw's flags are checked against the current one;
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -400,6 +407,19 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
|
||||
*CUDA 12.9*, or *none* — or from the command line:
|
||||
|
||||
```
|
||||
llmgw ctl engine-build gpu07 vllm cu129
|
||||
```
|
||||
|
||||
The choice is kept by the control plane and the node is told at once; it
|
||||
installs the build in the background, and the column shows how that stands.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
@@ -411,8 +431,8 @@ cluster's for them, in order:
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
|
||||
engines: { vllm: "0.19.1" } # the last one that does
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
@@ -1253,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
|
||||
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
|
||||
that version with `engine_overrides` (step 4) — its flags are not the ones
|
||||
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -314,6 +314,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
return
|
||||
}
|
||||
// An engine started by an agent older than the GPU check may be one
|
||||
// serving a GPU placement on the CPU. Stopped, so its replacement starts
|
||||
// through the check and fails there in words, if the build still sees
|
||||
// no GPU — rather than going on unnoticed after the upgrade that could
|
||||
// have caught it.
|
||||
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
|
||||
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
|
||||
_ = a.sup.Stop(ctx, o.ShardID)
|
||||
return
|
||||
}
|
||||
a.log.Info("adopted engine is serving",
|
||||
"shard", o.ShardID, "replica", o.ReplicaID)
|
||||
}
|
||||
|
||||
@@ -88,7 +88,7 @@ func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
// the installed build is used first — and quietly replace theirs.
|
||||
continue
|
||||
}
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
a.provWants = wants
|
||||
|
||||
@@ -27,6 +27,9 @@ import (
|
||||
type Want struct {
|
||||
Engine string
|
||||
Version string
|
||||
// Variant is a build chosen for this node ("cu129"); empty leaves it to
|
||||
// the node's hardware.
|
||||
Variant string
|
||||
}
|
||||
|
||||
// The states an engine's installation can be in.
|
||||
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
|
||||
"; install it by hand and name it in the agent's configuration"
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, in.Host)
|
||||
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
|
||||
st.State = Unsupported
|
||||
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
|
||||
return st
|
||||
}
|
||||
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
|
||||
if err != nil {
|
||||
st.State, st.Detail = Unsupported, err.Error()
|
||||
var nh NotHere
|
||||
|
||||
@@ -255,11 +255,11 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
|
||||
func TestTheBuildFollowsTheHardware(t *testing.T) {
|
||||
gpu := ubuntu
|
||||
gpu.NVIDIADriver = "570.86.15"
|
||||
b, err := chooseLlamaCpp("b1", gpu)
|
||||
b, err := chooseLlamaCpp("b1", "", gpu)
|
||||
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
|
||||
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
|
||||
}
|
||||
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
|
||||
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
|
||||
}
|
||||
|
||||
|
||||
@@ -86,28 +86,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
|
||||
reqs := b.Requirements
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
// refuses a download that does not match it.
|
||||
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var asset, sum string
|
||||
for name, s := range sums {
|
||||
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
|
||||
asset, sum = name, s
|
||||
}
|
||||
}
|
||||
if asset == "" {
|
||||
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
|
||||
}
|
||||
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
|
||||
b.Wheel.Tag + "/" + asset
|
||||
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
|
||||
}
|
||||
// A second index, for the torch the engine was built on. Every package
|
||||
// is taken from whichever of the two has the version resolved — both are
|
||||
// the publishers' own — and each file is still pinned by its hash.
|
||||
var index []string
|
||||
if b.ExtraIndex != "" {
|
||||
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
|
||||
}
|
||||
|
||||
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
|
||||
lock := filepath.Join(tmp, lockName)
|
||||
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
|
||||
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
|
||||
args = append(args, "-")
|
||||
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
|
||||
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
|
||||
strings.Join(b.Requirements, " "))
|
||||
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
|
||||
strings.Join(reqs, " "))
|
||||
install := append([]string{"pip", "install", "--python", python,
|
||||
"--require-hashes", "--no-deps", "-r", lock}, index...)
|
||||
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
|
||||
}
|
||||
|
||||
in.progress(st, "checking that %s runs", w.Engine)
|
||||
if _, _, err := in.verify(ctx, tmp, b); err != nil {
|
||||
return err
|
||||
}
|
||||
if b.GPUCheck && in.Host.NVIDIADriver != "" {
|
||||
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
|
||||
if err := in.gpuCheck(ctx, python, w, b); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
m, _ := json.MarshalIndent(markerFile{
|
||||
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
|
||||
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
|
||||
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
|
||||
}, "", " ")
|
||||
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
|
||||
@@ -203,3 +242,26 @@ func tail(b []byte) string {
|
||||
}
|
||||
return strings.Join(lines, " / ")
|
||||
}
|
||||
|
||||
// gpuProbe is what the environment is asked to run on the GPU: one small
|
||||
// operation, which needs the torch to have kernels for the card.
|
||||
const gpuProbe = `import torch
|
||||
assert torch.cuda.is_available(), "torch sees no CUDA device"
|
||||
x = torch.ones(64, device="cuda")
|
||||
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
|
||||
|
||||
// gpuCheck runs gpuProbe in a freshly installed environment.
|
||||
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
|
||||
defer cancel()
|
||||
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
|
||||
if err != nil {
|
||||
hint := ""
|
||||
if strings.Contains(string(out), "no kernel image") {
|
||||
hint = "; this build has no kernels for this GPU, so try the other build for this node"
|
||||
}
|
||||
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
|
||||
w.Engine, w.Version, b.Variant, tail(out), hint)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package provision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
@@ -23,6 +24,9 @@ type fakePython struct {
|
||||
pmu sync.Mutex
|
||||
commands []string
|
||||
failPip bool
|
||||
// noKernel makes the GPU check fail as a torch without kernels for the
|
||||
// card does.
|
||||
noKernel bool
|
||||
}
|
||||
|
||||
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
|
||||
@@ -43,6 +47,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
|
||||
if f.noKernel {
|
||||
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
|
||||
}
|
||||
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
|
||||
}
|
||||
return f.fakeSystem.run(ctx, env, name, args...)
|
||||
}
|
||||
@@ -59,6 +68,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
|
||||
uv := newRelease(map[string][]byte{
|
||||
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
|
||||
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
|
||||
// The fake GitHub serves every repository's release from this one.
|
||||
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
|
||||
})
|
||||
gh := uv.serve(t)
|
||||
mux := http.NewServeMux()
|
||||
@@ -130,8 +141,12 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"no GPU", ubuntu, Skipped, "has none"},
|
||||
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
|
||||
// Too old for CUDA 13 and for its CUDA 12.9 build both.
|
||||
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
|
||||
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
|
||||
// vLLM 0.20 and later have no Volta kernels in any build; the
|
||||
// refusal names the last version that has.
|
||||
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
t.Run(c.name, func(t *testing.T) {
|
||||
@@ -182,3 +197,92 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
|
||||
t.Errorf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
|
||||
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
|
||||
// 12.9 build, from the project's GitHub release, with the torch PyTorch
|
||||
// publishes for 12.9.
|
||||
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, gh := pythonInstaller(t, volta, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("a Turing node on driver 550: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
|
||||
t.Errorf("not installed under its own build's name: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
|
||||
for _, want := range []string{
|
||||
// Pinned to the digest GitHub publishes for the wheel.
|
||||
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
|
||||
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
|
||||
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
|
||||
"--require-hashes --no-deps -r",
|
||||
// Computing on the card, not only importing.
|
||||
"/bin/python -c import torch",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
|
||||
t.Errorf("installed without the index its torch comes from: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
|
||||
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
|
||||
}
|
||||
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
|
||||
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
|
||||
t.Error("a build that does not exist was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
|
||||
in, _ := pythonInstaller(t, gpuNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
|
||||
t.Fatalf("a build without kernels for the card: %+v", got[0])
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
|
||||
t.Error("marked installed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
|
||||
// A machine vLLM is not published for is refused, not given the other
|
||||
// build to fail with.
|
||||
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
|
||||
!strings.Contains(err.Error(), "no Linux build") {
|
||||
t.Errorf("an unknown architecture: %v", err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
|
||||
t.Errorf("a node with no GPU chosen cu129: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
|
||||
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
|
||||
b, err := chooseVLLM(voltaUntil, "", v100)
|
||||
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
|
||||
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
|
||||
}
|
||||
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
|
||||
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
|
||||
}
|
||||
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
|
||||
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -10,6 +10,7 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -48,12 +49,29 @@ type Build struct {
|
||||
// build must list a device with it, or it is not installed — a llama.cpp
|
||||
// whose CUDA backend cannot load runs on the CPU without complaint.
|
||||
GPUBackend string
|
||||
// Wheel, when set, is the engine itself as a wheel from a GitHub
|
||||
// release rather than from PyPI, checked against the digest GitHub
|
||||
// publishes for it. ExtraIndex is a package index besides PyPI that the
|
||||
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
|
||||
Wheel *Wheel
|
||||
ExtraIndex string
|
||||
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
|
||||
// compute something on one before the build counts as installed. A build
|
||||
// with no kernels for the card — a GPU generation its torch dropped —
|
||||
// imports and prints its version, and fails on its first model.
|
||||
GPUCheck bool
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
Project, Repo, Tag, Prefix, Suffix string
|
||||
}
|
||||
|
||||
// Recipe knows how to install one engine.
|
||||
type Recipe struct {
|
||||
// Repo is the GitHub repository whose releases carry the builds; empty
|
||||
@@ -61,8 +79,10 @@ type Recipe struct {
|
||||
Repo string
|
||||
// PyPI is the project an engine installed from PyPI is published as.
|
||||
PyPI string
|
||||
// Choose picks the build for a machine, or says why none fits.
|
||||
Choose func(version string, h discovery.Host) (Build, error)
|
||||
// Choose picks the build for a machine, or says why none fits. variant
|
||||
// is a build chosen for the node ("cu129"); empty leaves it to the
|
||||
// machine.
|
||||
Choose func(version, variant string, h discovery.Host) (Build, error)
|
||||
}
|
||||
|
||||
// Recipes are the engines this agent can install.
|
||||
@@ -104,7 +124,7 @@ type NotHere struct{ Reason string }
|
||||
|
||||
func (e NotHere) Error() string { return e.Reason }
|
||||
|
||||
func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
|
||||
var arch string
|
||||
switch h.Arch {
|
||||
case "amd64":
|
||||
@@ -121,30 +141,56 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
|
||||
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
|
||||
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
|
||||
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
|
||||
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
|
||||
const minDriverForCUDA13 = 580
|
||||
|
||||
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
|
||||
// compatibility; the oldest GPU one has kernels for is minComputeFor's
|
||||
// answer, which is Volta only up to voltaUntil.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
// cuda13Since is the first version of each Python engine whose PyPI build is
|
||||
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
|
||||
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
|
||||
|
||||
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
|
||||
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
|
||||
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
|
||||
const voltaUntil = "0.19.1"
|
||||
|
||||
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
|
||||
// on an architecture: what the project compiled, not what CUDA supports.
|
||||
func minComputeFor(version, arch string) float64 {
|
||||
if arch == "arm64" {
|
||||
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
|
||||
// since.
|
||||
if versionBefore(version, "0.20") {
|
||||
return 8.7
|
||||
}
|
||||
return 8.0
|
||||
}
|
||||
if versionBefore(version, "0.20") {
|
||||
return minComputeForCUDA12
|
||||
}
|
||||
return minComputeForCUDA13
|
||||
}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
|
||||
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older version (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
switch h.Arch {
|
||||
@@ -159,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if engine == "vLLM" {
|
||||
minCC = minComputeFor(version, h.Arch)
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return variant, nil
|
||||
}
|
||||
|
||||
// olderHint says which version runs on a GPU too old for the one asked for.
|
||||
func olderHint(engine string, cc float64) string {
|
||||
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
|
||||
return "; the last vLLM built for it is " + voltaUntil +
|
||||
": give these nodes that version with engine_overrides in the server's configuration"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
@@ -202,22 +256,124 @@ func versionBefore(a, b string) bool {
|
||||
// that load fails long after the installation looked finished.
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
variant, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
|
||||
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
|
||||
// version, built for the drivers and GPUs CUDA 13 left behind.
|
||||
var VLLMVariants = []string{"cu13", "cu129"}
|
||||
|
||||
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
|
||||
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
|
||||
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
|
||||
|
||||
// chooseVLLM picks the vLLM build for a machine.
|
||||
//
|
||||
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
|
||||
// same version's CUDA 12.9 build where only that does: one version across
|
||||
// the cluster, which matters because the flags llmgw builds are checked
|
||||
// against one version, and an older vLLM on the older cards would be a
|
||||
// second set of flags nobody checked. A variant chosen for the node is that
|
||||
// build, if the node can run it.
|
||||
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
|
||||
// What no build of it gets past: a machine it is not published for, or
|
||||
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
|
||||
// CUDA 12.9 build.
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
|
||||
}
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
}, nil
|
||||
if h.NVIDIADriver == "" {
|
||||
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
pypi := func(v string) Build {
|
||||
return Build{
|
||||
Variant: v,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 25,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := func() (Build, error) {
|
||||
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
|
||||
b := pypi("cu129")
|
||||
b.Requirements = nil
|
||||
// Found in the release's assets rather than named: the manylinux tag
|
||||
// in the file's name changes between versions (2_28, 2_31, 2_34).
|
||||
b.Wheel = &Wheel{
|
||||
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
|
||||
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
|
||||
}
|
||||
b.ExtraIndex = pytorchCU129
|
||||
return b, nil
|
||||
}
|
||||
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
|
||||
modern := !versionBefore(version, cuda13Since["vLLM"])
|
||||
switch variant {
|
||||
case "":
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err == nil {
|
||||
return pypi(v), nil
|
||||
}
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) || !modern {
|
||||
return Build{}, err
|
||||
}
|
||||
b, err129 := cu129()
|
||||
if err129 != nil {
|
||||
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
|
||||
}
|
||||
return b, nil
|
||||
case "cu13":
|
||||
if !modern {
|
||||
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi("cu13"), nil
|
||||
case "cu129":
|
||||
if !modern {
|
||||
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
|
||||
v, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return pypi(v), nil
|
||||
}
|
||||
if _, err := cudaGate("vLLM", version, h); err != nil {
|
||||
var nh NotHere
|
||||
if errors.As(err, &nh) {
|
||||
return Build{}, err
|
||||
}
|
||||
}
|
||||
return cu129()
|
||||
}
|
||||
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
|
||||
}
|
||||
|
||||
func chooseSGLang(version string, h discovery.Host) (Build, error) {
|
||||
// cudaFits checks a node against a CUDA's driver and GPU requirements.
|
||||
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, olderHint(engine, cc))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
|
||||
if chosen != "" {
|
||||
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
|
||||
}
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
@@ -252,7 +408,7 @@ func leadingInt(s string) int {
|
||||
// else — no toolkit.
|
||||
const cudaForLlamaCpp = "12.8"
|
||||
|
||||
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
|
||||
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
|
||||
// The published Linux builds are built on Ubuntu, but nothing about them
|
||||
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
|
||||
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
|
||||
|
||||
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(),
|
||||
newWeightsCmd(client, &asJSON),
|
||||
newCapacityCmd(client, &asJSON),
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "engine-build <node> <engine> <build>",
|
||||
Short: "Choose which build of an engine a GPU node installs",
|
||||
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
|
||||
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
|
||||
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
|
||||
" auto what the hardware runs (the default)\n" +
|
||||
" cu13 the CUDA 13 build, from PyPI\n" +
|
||||
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
|
||||
" none no vLLM on this node\n\n" +
|
||||
"The node is told at once and installs it in the background; a build it\n" +
|
||||
"cannot compute with on its GPU fails to install, saying why, and the\n" +
|
||||
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
|
||||
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
|
||||
" llmgw ctl engine-build gpu07 vllm cu129",
|
||||
Args: cobra.ExactArgs(3),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
|
||||
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
|
||||
args[0], args[1], args[2])
|
||||
return nil
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -495,6 +495,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
// And what each model was measured to serve, for scaling and for
|
||||
// choosing between profiles.
|
||||
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
|
||||
// The engine builds chosen per node on the cluster page.
|
||||
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
|
||||
log.Info("competing to be the control plane that places work",
|
||||
"owner", lead.Owner())
|
||||
}
|
||||
@@ -504,6 +506,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
|
||||
})
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
|
||||
if ch := ctl.EngineChoices(); ch != nil {
|
||||
svc.SetEngineChoices(ch.For)
|
||||
ch.OnChange(svc.Reprovision)
|
||||
}
|
||||
// A standby must refuse agents as well as refusing to place. Commands
|
||||
// travel down the stream the agent opened, so a standby holding one is a
|
||||
// node the leader cannot command — and the leader's placements would fail
|
||||
|
||||
@@ -9,6 +9,7 @@ package adminapi
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
@@ -111,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
|
||||
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
|
||||
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
|
||||
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
|
||||
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
|
||||
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
|
||||
@@ -156,6 +158,10 @@ type NodeView struct {
|
||||
// Provision is where installing each wanted engine stands. An engine is in
|
||||
// Engines only once its installation here is ready.
|
||||
Provision []inventory.EngineInstall `json:"provision,omitempty"`
|
||||
// EngineBuilds are the builds chosen for this node, by engine: "auto"
|
||||
// (its hardware decides), a build ("cu129"), or "none". Only on a node
|
||||
// with GPUs, the only kind with a choice to make.
|
||||
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
|
||||
// OS is the operating system and architecture, as the agent read them.
|
||||
OS string `json:"os,omitempty"`
|
||||
// Warnings are facts about the machine that make it slow or unusable
|
||||
@@ -473,6 +479,13 @@ func (a *API) Nodes() []NodeView {
|
||||
}
|
||||
slices.Sort(v.Engines)
|
||||
v.Provision = n.Provision
|
||||
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
|
||||
chosen := ch.For(n.ID)
|
||||
v.EngineBuilds = map[string]string{"vllm": "auto"}
|
||||
for e, b := range chosen {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
@@ -1532,3 +1545,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
|
||||
}
|
||||
|
||||
// NodeEngineRequest chooses one engine's build for a node.
|
||||
type NodeEngineRequest struct {
|
||||
Engine string `json:"engine"`
|
||||
// Build is "auto", a build such as "cu129", or "none".
|
||||
Build string `json:"build"`
|
||||
}
|
||||
|
||||
// setNodeEngine chooses an engine's build for a node, and has the node
|
||||
// install it.
|
||||
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, ok := a.ctl.Inventory().Get(id); !ok {
|
||||
writeError(w, http.StatusNotFound, "no node "+id)
|
||||
return
|
||||
}
|
||||
var req NodeEngineRequest
|
||||
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, req)
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
|
||||
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
|
||||
ch := a.ctl.EngineChoices()
|
||||
if ch == nil {
|
||||
return errors.New("this control plane has no store to keep the choice in")
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(node); !ok {
|
||||
return errors.New("no node " + node)
|
||||
}
|
||||
return ch.Set(ctx, node, engine, build, by)
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// choiceStore keeps engine choices in memory.
|
||||
type choiceStore struct{ m map[string]map[string]string }
|
||||
|
||||
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
|
||||
return c.m, nil
|
||||
}
|
||||
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
|
||||
if c.m[node] == nil {
|
||||
c.m[node] = map[string]string{}
|
||||
}
|
||||
c.m[node][engine] = variant
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
|
||||
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
|
||||
var told []string
|
||||
ch.OnChange(func(node string) error { told = append(told, node); return nil })
|
||||
ctl.SetEngineChoices(ch)
|
||||
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
|
||||
t.Fatalf("status %d", code)
|
||||
}
|
||||
if len(told) != 1 || told[0] != "gpu07" {
|
||||
t.Errorf("the node was not told: %v", told)
|
||||
}
|
||||
var nodes []NodeView
|
||||
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
|
||||
for _, n := range nodes {
|
||||
switch n.ID {
|
||||
case "gpu07":
|
||||
if n.EngineBuilds["vllm"] != "cu129" {
|
||||
t.Errorf("gpu07 shows %v", n.EngineBuilds)
|
||||
}
|
||||
case "cpu01":
|
||||
if n.EngineBuilds != nil {
|
||||
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
|
||||
t.Errorf("%s: status %d", bad, code)
|
||||
}
|
||||
}
|
||||
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
|
||||
ch.For("gpu07")["vllm"] != "" {
|
||||
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
|
||||
}
|
||||
}
|
||||
@@ -43,14 +43,40 @@ func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" {
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("the Volta node: %v", got)
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" {
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
|
||||
t.Errorf("the Hopper node: %v", got)
|
||||
}
|
||||
// A node that disabled vLLM does not offer it, and is not asked for it.
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" {
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
|
||||
t.Errorf("asked a node for an engine it does not install: %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
|
||||
s.SetEngineChoices(func(node string) map[string]string {
|
||||
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
|
||||
})
|
||||
can := []string{"llamacpp", "vllm"}
|
||||
variant := func(node string) (string, bool) {
|
||||
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
|
||||
if e.GetEngine() == "vllm" {
|
||||
return e.GetVariant(), true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
if v, ok := variant("gpu07"); !ok || v != "cu129" {
|
||||
t.Errorf("gpu07: %q, %v", v, ok)
|
||||
}
|
||||
if _, ok := variant("gpu08"); ok {
|
||||
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
|
||||
}
|
||||
if v, ok := variant("gpu09"); !ok || v != "" {
|
||||
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
|
||||
}
|
||||
}
|
||||
@@ -56,6 +56,10 @@ type Service struct {
|
||||
// enginesFor is what a node is told to install, set once at start: the
|
||||
// cluster's engines, with any override that picks the node laid over.
|
||||
enginesFor func(node inventory.Node) map[string]string
|
||||
// choices are the engine builds chosen per node, and installableBy what
|
||||
// each connected node said it can install, for telling it again.
|
||||
choices func(nodeID string) map[string]string
|
||||
installableBy map[string][]string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -78,6 +82,29 @@ func (s *Service) SetEngines(e map[string]string) {
|
||||
// of two GPU generations needs two vLLM builds, or vLLM on only one.
|
||||
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
|
||||
|
||||
// SetEngineChoices sets where the builds chosen for each node come from: by
|
||||
// engine, a build ("cu129") or "none". An engine not named is left to the
|
||||
// node's hardware.
|
||||
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
|
||||
|
||||
// Reprovision tells a connected node again what to install, after a choice
|
||||
// for it changed; one not connected is told when it next connects.
|
||||
func (s *Service) Reprovision(nodeID string) error {
|
||||
if s.enginesFor == nil {
|
||||
return nil
|
||||
}
|
||||
s.mu.RLock()
|
||||
can, ok := s.installableBy[nodeID]
|
||||
s.mu.RUnlock()
|
||||
node, known := s.inv.Get(nodeID)
|
||||
if !ok || !known {
|
||||
return nil
|
||||
}
|
||||
return s.send(nodeID, &llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
|
||||
})
|
||||
}
|
||||
|
||||
// conn is one agent's open stream.
|
||||
type conn struct {
|
||||
nodeID string
|
||||
@@ -188,7 +215,14 @@ func (s *Service) Connect(
|
||||
// node rebuilt from nothing is told the same as one that has been here a
|
||||
// month, and a changed version reaches every node on its next connect.
|
||||
if s.enginesFor != nil {
|
||||
p := s.provisionFor(node, reg)
|
||||
can := installable(reg)
|
||||
s.mu.Lock()
|
||||
if s.installableBy == nil {
|
||||
s.installableBy = map[string][]string{}
|
||||
}
|
||||
s.installableBy[nodeID] = can
|
||||
s.mu.Unlock()
|
||||
p := s.provisionFor(node, can)
|
||||
if err := stream.Send(&llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
|
||||
}); err != nil {
|
||||
@@ -856,15 +890,18 @@ func rate(before, now uint64, secs float64) float64 {
|
||||
|
||||
// provisionFor is what a node is told to install: the engines decided for
|
||||
// it, those it can install.
|
||||
func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision {
|
||||
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
|
||||
p := &llmgwv1.Provision{}
|
||||
can := installable(reg)
|
||||
wants := s.enginesFor(node)
|
||||
var chosen map[string]string
|
||||
if s.choices != nil {
|
||||
chosen = s.choices(node.ID)
|
||||
}
|
||||
for _, e := range slices.Sorted(maps.Keys(wants)) {
|
||||
if !slices.Contains(can, e) {
|
||||
if !slices.Contains(can, e) || chosen[e] == "none" {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]})
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
|
||||
if p.Nodes <= 0 {
|
||||
p.Nodes = 1
|
||||
}
|
||||
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
|
||||
// anything for it, is read as the width it was meant as: N cards of
|
||||
// one node. Refusing it would have dropped a catalog's model on the
|
||||
// upgrade that made the width work. Only both given, disagreeing, is
|
||||
// refused below, as ambiguous.
|
||||
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
|
||||
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
|
||||
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case p.CPUOnly():
|
||||
// Nothing to divide up: this profile asks for no cards at all.
|
||||
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
|
||||
p.Weight = 1
|
||||
}
|
||||
}
|
||||
// A Hugging Face repository page for a model vLLM or SGLang runs is the
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -335,9 +352,11 @@ func (m Model) validate() error {
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
|
||||
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
@@ -398,8 +417,9 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
|
||||
m.Name, p.Name, p.Nodes)
|
||||
case p.TP > 1 || p.PP > 1:
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
|
||||
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
|
||||
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
|
||||
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
|
||||
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
|
||||
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
|
||||
t.Errorf("placement was not asked for four GPUs: %+v", pl)
|
||||
}
|
||||
// tp is a width llama.cpp has no way to use; refused, saying what to
|
||||
// write instead.
|
||||
// tp, written before gpus_per_node meant anything for llama.cpp, is read
|
||||
// as the same width rather than refused: refusing dropped the model on
|
||||
// the upgrade that made the width work.
|
||||
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
|
||||
c, err = New([]Model{old})
|
||||
if err != nil {
|
||||
t.Fatalf("tp 4 on llama.cpp: %v", err)
|
||||
}
|
||||
got, _ := c.Resolve("old")
|
||||
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
|
||||
t.Errorf("tp 4 should place on four GPUs: %+v", p)
|
||||
}
|
||||
// Both given and disagreeing is ambiguous, and refused; so is more than
|
||||
// one node.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
|
||||
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
|
||||
} {
|
||||
_, err := New([]Model{model("x", p)})
|
||||
if err == nil {
|
||||
if _, err := New([]Model{model("x", p)}); err == nil {
|
||||
t.Errorf("%s: accepted", p.Name)
|
||||
}
|
||||
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
|
||||
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryPageIsNotASource(t *testing.T) {
|
||||
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
|
||||
// The field: an AWQ model's page given as a vLLM source, fetched as one
|
||||
// file of HTML. vLLM downloads a repository id itself.
|
||||
for _, src := range []string{
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
|
||||
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
|
||||
} {
|
||||
m := model("awq")
|
||||
m.Source = src
|
||||
_, err := New([]Model{m})
|
||||
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
|
||||
t.Errorf("%s: %v", src, err)
|
||||
c, err := New([]Model{m})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", src, err)
|
||||
}
|
||||
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
|
||||
t.Errorf("%s became %q", src, got.Source)
|
||||
}
|
||||
}
|
||||
m := model("file")
|
||||
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{m}); err != nil {
|
||||
// llama.cpp needs a file, which a page is not.
|
||||
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
|
||||
g.Source = "https://huggingface.co/Qwen/X-GGUF"
|
||||
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
|
||||
t.Errorf("a page for llama.cpp: %v", err)
|
||||
}
|
||||
f := model("file")
|
||||
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
|
||||
if _, err := New([]Model{f}); err != nil {
|
||||
t.Errorf("a file under /resolve/ is a source: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -75,15 +75,18 @@ func DefaultConfig() Config {
|
||||
|
||||
// Controller owns the control plane's state.
|
||||
type Controller struct {
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
// engineChoices are the engine builds chosen per node; nil without a
|
||||
// store to keep them in.
|
||||
engineChoices *EngineChoices
|
||||
cfg Config
|
||||
log *slog.Logger
|
||||
inv *inventory.Store
|
||||
cat *catalog.Live
|
||||
rec *reconciler.Reconciler
|
||||
lead *Leadership
|
||||
disp reconciler.Dispatcher
|
||||
sink EndpointSink
|
||||
usage UsageSource
|
||||
|
||||
mu sync.Mutex
|
||||
last string // fingerprint of the last published list
|
||||
@@ -408,3 +411,9 @@ func EngineKeys(secret []byte) func(replicaID string) string {
|
||||
|
||||
// QuotaUse is every capacity quota beside what its scope holds.
|
||||
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
|
||||
|
||||
// SetEngineChoices gives the controller the engine builds chosen per node.
|
||||
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
|
||||
|
||||
// EngineChoices are the engine builds chosen per node, or nil.
|
||||
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
|
||||
@@ -0,0 +1,108 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
)
|
||||
|
||||
// EngineChoiceStore is where the engine builds chosen per node are kept.
|
||||
type EngineChoiceStore interface {
|
||||
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
|
||||
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
|
||||
}
|
||||
|
||||
// EngineChoices are the engine builds chosen for nodes, rather than left to
|
||||
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
|
||||
// also install on, or no vLLM at all. Kept in the store, held here for the
|
||||
// agent service to ask on every registration, and pushed to a node the
|
||||
// moment its choice changes.
|
||||
type EngineChoices struct {
|
||||
st EngineChoiceStore
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.RWMutex
|
||||
byNode map[string]map[string]string
|
||||
notify func(node string) error
|
||||
}
|
||||
|
||||
// NewEngineChoices reads what was chosen before.
|
||||
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
|
||||
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
|
||||
if got, err := st.NodeEngines(ctx); err != nil {
|
||||
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
|
||||
} else {
|
||||
c.byNode = got
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// OnChange is called with a node whose choice changed, to tell it.
|
||||
func (c *EngineChoices) OnChange(f func(node string) error) {
|
||||
c.mu.Lock()
|
||||
c.notify = f
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// For is what was chosen for one node, by engine: a build, or "none".
|
||||
func (c *EngineChoices) For(node string) map[string]string {
|
||||
c.mu.RLock()
|
||||
defer c.mu.RUnlock()
|
||||
return maps.Clone(c.byNode[node])
|
||||
}
|
||||
|
||||
// EngineBuilds are what can be chosen for each engine that has a choice:
|
||||
// "auto" is the node's hardware deciding, "none" no engine at all.
|
||||
func EngineBuilds(engine string) []string {
|
||||
switch engine {
|
||||
case "vllm":
|
||||
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
|
||||
case "sglang":
|
||||
return []string{"auto", "none"}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Set chooses an engine's build for a node, keeps it, and tells the node.
|
||||
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
|
||||
builds := EngineBuilds(engine)
|
||||
if builds == nil {
|
||||
return fmt.Errorf("%s has no builds to choose between", engine)
|
||||
}
|
||||
if !slices.Contains(builds, build) {
|
||||
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
|
||||
}
|
||||
variant := build
|
||||
if build == "auto" {
|
||||
variant = ""
|
||||
}
|
||||
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
|
||||
return err
|
||||
}
|
||||
c.mu.Lock()
|
||||
if variant == "" {
|
||||
delete(c.byNode[node], engine)
|
||||
} else {
|
||||
if c.byNode[node] == nil {
|
||||
c.byNode[node] = map[string]string{}
|
||||
}
|
||||
c.byNode[node][engine] = variant
|
||||
}
|
||||
notify := c.notify
|
||||
c.mu.Unlock()
|
||||
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
|
||||
if notify != nil {
|
||||
if err := notify(node); err != nil {
|
||||
// Kept and applied on its next connection; said, since a node that
|
||||
// is not connected is the usual reason.
|
||||
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -581,7 +581,7 @@ func Evaluate(f Facts) []Finding {
|
||||
// the same decision the agent makes, so the report and the node agree.
|
||||
func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
v := f.VLLMVersion
|
||||
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
|
||||
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
|
||||
var nh provision.NotHere
|
||||
switch {
|
||||
case errors.As(err, &nh):
|
||||
|
||||
@@ -153,11 +153,15 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"},
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"},
|
||||
Warn, "before 0.26 is built for CUDA 12"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
|
||||
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
|
||||
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
|
||||
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
|
||||
// A V100 has no build of vLLM 0.20 or later, and is told which has.
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
|
||||
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
|
||||
@@ -196,6 +196,11 @@ func (r *Router) Handler() http.Handler {
|
||||
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
|
||||
pages.SetCapacity(c)
|
||||
}
|
||||
// The engine build each GPU node installs, chosen on the cluster
|
||||
// page.
|
||||
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
|
||||
pages.RegisterNodeEngines(mux, e)
|
||||
}
|
||||
pages.RegisterAdminQuotas(mux, r.store, r.acct)
|
||||
// And the report of what that tree actually used.
|
||||
pages.RegisterAdminUsage(mux, r.store, r.acct)
|
||||
|
||||
@@ -45,6 +45,7 @@ var copyOrder = []string{
|
||||
// derived from this, and a new secret would lock the router out of all
|
||||
// of them at the moment of the move.
|
||||
"cluster_secrets",
|
||||
"node_engines",
|
||||
}
|
||||
|
||||
// notCopied are tables deliberately left behind.
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,11 @@
|
||||
-- The build of an engine chosen for a node, where it is not left to the
|
||||
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
|
||||
-- or no vLLM at all. A node with no row here installs what its hardware runs.
|
||||
CREATE TABLE node_engines (
|
||||
node_id TEXT NOT NULL,
|
||||
engine TEXT NOT NULL,
|
||||
variant TEXT NOT NULL,
|
||||
updated_at BIGINT NOT NULL,
|
||||
updated_by TEXT NOT NULL,
|
||||
PRIMARY KEY (node_id, engine)
|
||||
);
|
||||
@@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NodeEngines are the engine builds chosen per node: node id, then engine,
|
||||
// then the build ("cu129") or "none". A node or engine not here is left to
|
||||
// the node's hardware.
|
||||
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
|
||||
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
out := map[string]map[string]string{}
|
||||
for rows.Next() {
|
||||
var node, engine, variant string
|
||||
if err := rows.Scan(&node, &engine, &variant); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if out[node] == nil {
|
||||
out[node] = map[string]string{}
|
||||
}
|
||||
out[node][engine] = variant
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SetNodeEngine chooses an engine's build for a node; an empty variant
|
||||
// leaves it to the node's hardware again.
|
||||
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
|
||||
if variant == "" {
|
||||
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
|
||||
return err
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
|
||||
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
|
||||
node, engine, variant, time.Now().UnixMilli(), by)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package store
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := db.NodeEngines(ctx)
|
||||
if err != nil || got["gpu07"]["vllm"] != "none" {
|
||||
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
|
||||
}
|
||||
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
|
||||
t.Errorf("left to the node again, still %v", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/url"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
|
||||
// such as "cu129", or "none".
|
||||
type NodeEngineAdmin interface {
|
||||
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
|
||||
}
|
||||
|
||||
// RegisterNodeEngines mounts the choice of engine builds on the cluster
|
||||
// page: for an administrator, with the session's CSRF token, and answered by
|
||||
// going back to the page with what happened.
|
||||
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
|
||||
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
|
||||
sess, _, _ := p.session(r)
|
||||
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
|
||||
http.Error(w, "bad request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
|
||||
q := url.Values{}
|
||||
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
|
||||
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
|
||||
q.Set("problem", node+": "+err.Error())
|
||||
} else {
|
||||
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
|
||||
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
|
||||
"; the node is told now, and installs it in the background")
|
||||
}
|
||||
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
|
||||
}))
|
||||
}
|
||||
|
||||
func buildWords(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "in the build its hardware runs"
|
||||
case "none":
|
||||
return "no longer"
|
||||
}
|
||||
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
|
||||
type buildFleet struct{ fakeFleet }
|
||||
|
||||
func (buildFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{
|
||||
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
|
||||
EngineBuilds: map[string]string{"vllm": "cu129"},
|
||||
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
|
||||
{ID: "cpu01", State: "ready"},
|
||||
}
|
||||
}
|
||||
|
||||
type recordedChoice struct{ node, engine, build, by string }
|
||||
|
||||
type fakeEngineAdmin struct{ got []recordedChoice }
|
||||
|
||||
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
|
||||
f.got = append(f.got, recordedChoice{node, engine, build, by})
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
|
||||
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
|
||||
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
|
||||
fe := &fakeEngineAdmin{}
|
||||
p.RegisterNodeEngines(mux, fe)
|
||||
|
||||
body := mirrorGet(t, mux, "/admin").Body.String()
|
||||
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
|
||||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
|
||||
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
|
||||
}
|
||||
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
|
||||
t.Error("a node with no GPU was offered a vLLM build")
|
||||
}
|
||||
if !strings.Contains(body, "CUDA 12.9 installing") {
|
||||
t.Error("what is installed is not shown beside the choice")
|
||||
}
|
||||
|
||||
// The read-only page on the admin listener has no session, so no form.
|
||||
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
|
||||
if strings.Contains(ro, "/engines\"") {
|
||||
t.Error("the admin listener's page offers a change it cannot sign")
|
||||
}
|
||||
|
||||
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
|
||||
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
|
||||
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
|
||||
}
|
||||
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
|
||||
t.Errorf("redirected to %s, saying nothing", loc)
|
||||
}
|
||||
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
|
||||
t.Errorf("a forged form was accepted: %d", w.Code)
|
||||
}
|
||||
}
|
||||
@@ -19,13 +19,16 @@ type DashboardData struct {
|
||||
// of a GPU table that would only say there are none.
|
||||
CPU cpuCapacity
|
||||
// Menu is the shared top menu; Nav below is what the cluster page adds.
|
||||
Menu Nav
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
Menu Nav
|
||||
// Notice and Problem are what the last change made from this page did,
|
||||
// carried over the redirect back to it.
|
||||
Notice, Problem string
|
||||
Nodes []adminapi.NodeView
|
||||
Models []adminapi.ModelView
|
||||
Replicas []adminapi.ReplicaView
|
||||
Unplaced []adminapi.ExplainView
|
||||
Usage []store.KeyUsage
|
||||
Summary adminapi.Summary
|
||||
// Fabrics is the physical fabrics and what disagrees about them, from a
|
||||
// source that can say.
|
||||
Fabrics *fabrics.Report
|
||||
@@ -170,6 +173,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
|
||||
Now: time.Now().UTC().Format(time.RFC3339),
|
||||
AuthEnabled: usage != nil,
|
||||
}
|
||||
if d.Menu.Admin {
|
||||
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
|
||||
}
|
||||
if src != nil {
|
||||
d.Nodes = src.Nodes()
|
||||
d.CPU = cpuCapacityOf(d.Nodes)
|
||||
|
||||
@@ -50,8 +50,9 @@
|
||||
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
|
||||
|
||||
<h2>Nodes</h2>
|
||||
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
|
||||
{{if .Nodes}}
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
|
||||
{{range .Nodes}}<tr>
|
||||
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
|
||||
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
|
||||
@@ -66,6 +67,23 @@
|
||||
was never asked about its disk must not read as a full one. */}}
|
||||
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
|
||||
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
|
||||
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
|
||||
What is installed beside it, so a choice and its outcome are read
|
||||
together. Only a GPU node has one; changing it is for an
|
||||
administrator, on the page reached through a session. */}}
|
||||
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
|
||||
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
|
||||
{{if and $.Menu.CSRF .EngineBuilds}}
|
||||
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
|
||||
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
|
||||
<input type="hidden" name="engine" value="vllm">
|
||||
<select name="build" title="the vLLM build this node installs">
|
||||
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
|
||||
</select>
|
||||
<noscript><button type="submit">Save</button></noscript>
|
||||
</form>
|
||||
{{end}}
|
||||
{{else}}—{{end}}</td>{{end}}
|
||||
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
|
||||
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
|
||||
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
|
||||
@@ -75,6 +93,13 @@
|
||||
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
|
||||
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
|
||||
{{end}}</table>
|
||||
<script>
|
||||
// A build chosen is saved as it is chosen, with the page reloaded on the
|
||||
// answer: one select, no separate button to forget.
|
||||
document.querySelectorAll("form.build-choice select").forEach(function (s) {
|
||||
s.addEventListener("change", function () { s.form.submit(); });
|
||||
});
|
||||
</script>
|
||||
{{else}}<p class="empty">No agents have registered.</p>{{end}}
|
||||
|
||||
<h2>Models</h2>
|
||||
|
||||
+20
-1
@@ -26,6 +26,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -118,7 +119,25 @@ var funcs = template.FuncMap{
|
||||
},
|
||||
// shortNode is a node's name without what every node's name starts with.
|
||||
"shortNode": strings.TrimPrefix,
|
||||
"age": age,
|
||||
// engineBuilds are the builds an engine can be given on a node, and
|
||||
// buildLabel says one in words.
|
||||
"engineBuilds": control.EngineBuilds,
|
||||
"buildLabel": func(b string) string {
|
||||
switch b {
|
||||
case "auto":
|
||||
return "auto: what the hardware runs"
|
||||
case "cu13":
|
||||
return "CUDA 13"
|
||||
case "cu129":
|
||||
return "CUDA 12.9"
|
||||
case "cu12":
|
||||
return "CUDA 12"
|
||||
case "none":
|
||||
return "none"
|
||||
}
|
||||
return b
|
||||
},
|
||||
"age": age,
|
||||
// pctbar is a percentage as the minibar template takes it: whole, and
|
||||
// held to 0..100 so a measurement a little over does not draw past its box.
|
||||
"pctbar": func(v float64) struct{ P int } {
|
||||
|
||||
Reference in new issue
Block a user