Compare commits

...
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 86e9a1cb83 Give V100s the last vLLM built for them, not a build without their kernels
Release / build (push) Successful in 1m13s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m47s
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:04:45 +02:00
Ilker ManapandClaude Opus 5.5 4f280a3546 Give each GPU node a vLLM build it can run, chosen on the cluster page
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m55s
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:39:14 +02:00
32 changed files with 1111 additions and 101 deletions

No files matched your search

+32 -7
View File
@@ -123,6 +123,32 @@ leaving the model with nothing. Now:
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
### Every GPU node gets a vLLM it can run, chosen on the cluster page
The vLLM a node installs is the build its hardware runs: CUDA 13 from PyPI
where the driver (580+) allows, and otherwise the same version's CUDA 12.9
build from vLLM's GitHub release, pinned to GitHub's digest, with PyTorch's
CUDA 12.9 torch (driver 525+). One version across the cluster, so the flags
llmgw builds — checked against that version — hold on every node.
**Volta (V100) is not covered by either build.** Read from vLLM's release
pipeline: every x86 build from 0.20.0 on — CUDA 13 and CUDA 12.9 alike — has
kernels for compute capability 7.5 and up (aarch64: 8.0 and up). The last
vLLM built for 7.0 is 0.19.1, whose PyPI wheel is CUDA 12 on PyPI's torch 2.10
(also built for 7.0). A V100 node asked for a newer vLLM is refused, saying
so; give such nodes 0.19.1 with `engine_overrides` by `gpu_class`. llmgw's
launch flags are checked against 0.30.0, so run `llmgw ctl engine-check vllm`
on such a node once it is installed.
- **The install computes on the GPU before it counts**: a small tensor
operation in the new environment. A build without kernels for the card
imports and prints its version, and failed only on its first model; now it
fails to install, saying so.
- **The build is chosen per node** on the cluster page (*vLLM build*: auto,
CUDA 13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
`POST /api/v1/nodes/{id}/engines`. The choice is kept in the store and sent
to the node at once.
### Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly: `node add` refused a
@@ -136,13 +162,12 @@ and `--ignore-checks` passed over every other problem with it.
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
or labels a different version of an engine, or none: vLLM 0.19.1, the last
built for Volta, on V100s beside the current one on newer cards.
- **The requirements follow the version.** vLLM before 0.20 is CUDA 12 on
PyPI (driver 525+), and only up to 0.19.1 has Volta kernels; the threshold
had been CUDA 13's for every version, so the doctor's advice to set an older
version could not work. A refusal now says which version would.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
+16 -5
View File
@@ -2370,9 +2370,12 @@ func (x *Provision) GetEngines() []*EngineWant {
// EngineWant names one engine at one version.
type EngineWant struct {
state protoimpl.MessageState `protogen:"open.v1"`
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
state protoimpl.MessageState `protogen:"open.v1"`
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
// variant is the build to install, when one is chosen for the node rather
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -2421,6 +2424,13 @@ func (x *EngineWant) GetVersion() string {
return ""
}
func (x *EngineWant) GetVariant() string {
if x != nil {
return x.Variant
}
return ""
}
// ProvisionStatus is where one engine's installation stands on a node.
type ProvisionStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
@@ -4068,11 +4078,12 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
"\tProvision\x12.\n" +
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
"\n" +
"EngineWant\x12\x16\n" +
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
"\x0fProvisionStatus\x12\x16\n" +
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
+3
View File
@@ -465,6 +465,9 @@ message Provision {
message EngineWant {
string engine = 1; // "llamacpp"
string version = 2; // "b10991"
// variant is the build to install, when one is chosen for the node rather
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
string variant = 3;
}
// ProvisionStatus is where one engine's installation stands on a node.
+30 -9
View File
@@ -376,10 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
Volta (7.0);
- **a Turing GPU or newer** (compute capability 7.5; 8.0 on aarch64), and
**driver 580 or newer** for the CUDA 13 build that PyPI carries from vLLM
0.20 on. A node with an older driver gets **the same version built for CUDA
12.9** instead, from vLLM's GitHub release (checked against the digest
GitHub publishes) with PyTorch's CUDA 12.9 torch: driver 525 or newer.
Either way the install ends by computing something on the node's GPU, and a
build that cannot fails to install, saying why;
- **a V100 (Volta, 7.0) needs vLLM 0.19.1**, the last version built for it:
every build from 0.20 on starts at 7.5. Give V100 nodes that version with
`engine_overrides` (below), and run `llmgw ctl engine-check vllm` there once
it is installed, since llmgw's flags are checked against the current one;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -400,6 +407,19 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
*CUDA 12.9*, or *none* — or from the command line:
```
llmgw ctl engine-build gpu07 vllm cu129
```
The choice is kept by the control plane and the node is told at once; it
installs the build in the background, and the column shows how that stands.
#### Different GPUs, different engines
A cluster of two GPU generations may need two builds of an engine, or one on
@@ -411,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- gpu_class: v100-32g # Volta: no vLLM build from 0.20 on has its kernels
engines: { vllm: "0.19.1" } # the last one that does
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1253,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **No vLLM from 0.20 on runs on a V100.** Every build of it has kernels for
Turing (7.5) and newer; 0.19.1 is the last built for Volta. Give V100 nodes
that version with `engine_overrides` (step 4) — its flags are not the ones
llmgw was checked against, so run `llmgw ctl engine-check vllm` there.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+1 -1
View File
@@ -88,7 +88,7 @@ func (a *Agent) setProvision(p *llmgwv1.Provision) {
// the installed build is used first — and quietly replace theirs.
continue
}
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
}
a.mu.Lock()
a.provWants = wants
+9 -1
View File
@@ -27,6 +27,9 @@ import (
type Want struct {
Engine string
Version string
// Variant is a build chosen for this node ("cu129"); empty leaves it to
// the node's hardware.
Variant string
}
// The states an engine's installation can be in.
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
"; install it by hand and name it in the agent's configuration"
return st
}
build, err := recipe.Choose(w.Version, in.Host)
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
st.State = Unsupported
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
return st
}
build, err := recipe.Choose(w.Version, w.Variant, in.Host)
if err != nil {
st.State, st.Detail = Unsupported, err.Error()
var nh NotHere
+2 -2
View File
@@ -255,11 +255,11 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
func TestTheBuildFollowsTheHardware(t *testing.T) {
gpu := ubuntu
gpu.NVIDIADriver = "570.86.15"
b, err := chooseLlamaCpp("b1", gpu)
b, err := chooseLlamaCpp("b1", "", gpu)
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
}
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
if _, err := chooseLlamaCpp("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
}
+72 -10
View File
@@ -86,28 +86,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
}
python := filepath.Join(tmp, "bin", "python")
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
reqs := b.Requirements
if b.Wheel != nil {
// The engine from its GitHub release, pinned to the digest GitHub
// publishes for the file: the same trust as an archive, and uv
// refuses a download that does not match it.
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
if err != nil {
return err
}
var asset, sum string
for name, s := range sums {
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
asset, sum = name, s
}
}
if asset == "" {
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
}
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
b.Wheel.Tag + "/" + asset
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
}
// A second index, for the torch the engine was built on. Every package
// is taken from whichever of the two has the version resolved — both are
// the publishers' own — and each file is still pinned by its hash.
var index []string
if b.ExtraIndex != "" {
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
}
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
lock := filepath.Join(tmp, lockName)
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
args = append(args, "-")
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
strings.Join(b.Requirements, " "))
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock); err != nil {
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
strings.Join(reqs, " "))
install := append([]string{"pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock}, index...)
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
in.progress(st, "checking that %s runs", w.Engine)
if _, _, err := in.verify(ctx, tmp, b); err != nil {
return err
}
if b.GPUCheck && in.Host.NVIDIADriver != "" {
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
if err := in.gpuCheck(ctx, python, w, b); err != nil {
return err
}
}
m, _ := json.MarshalIndent(markerFile{
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
}, "", " ")
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
@@ -203,3 +242,26 @@ func tail(b []byte) string {
}
return strings.Join(lines, " / ")
}
// gpuProbe is what the environment is asked to run on the GPU: one small
// operation, which needs the torch to have kernels for the card.
const gpuProbe = `import torch
assert torch.cuda.is_available(), "torch sees no CUDA device"
x = torch.ones(64, device="cuda")
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
// gpuCheck runs gpuProbe in a freshly installed environment.
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
defer cancel()
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
if err != nil {
hint := ""
if strings.Contains(string(out), "no kernel image") {
hint = "; this build has no kernels for this GPU, so try the other build for this node"
}
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
w.Engine, w.Version, b.Variant, tail(out), hint)
}
return nil
}
+106 -2
View File
@@ -2,6 +2,7 @@ package provision
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os"
@@ -23,6 +24,9 @@ type fakePython struct {
pmu sync.Mutex
commands []string
failPip bool
// noKernel makes the GPU check fail as a torch without kernels for the
// card does.
noKernel bool
}
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
@@ -43,6 +47,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
return []byte("0.30.0\n"), nil
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
if f.noKernel {
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
}
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
}
return f.fakeSystem.run(ctx, env, name, args...)
}
@@ -59,6 +68,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
uv := newRelease(map[string][]byte{
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
// The fake GitHub serves every repository's release from this one.
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
})
gh := uv.serve(t)
mux := http.NewServeMux()
@@ -130,8 +141,12 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
says string
}{
{"no GPU", ubuntu, Skipped, "has none"},
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
// Too old for CUDA 13 and for its CUDA 12.9 build both.
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
{"Pascal", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1}, Unsupported, "7.5 and newer"},
// vLLM 0.20 and later have no Volta kernels in any build; the
// refusal names the last version that has.
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, Unsupported, "the last vLLM built for it is 0.19.1"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -182,3 +197,92 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
t.Errorf("got %s %q", got[0].State, got[0].Detail)
}
}
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
// 12.9 build, from the project's GitHub release, with the torch PyTorch
// publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, gh := pythonInstaller(t, volta, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Ready || got[0].Variant != "cu129" {
t.Fatalf("a Turing node on driver 550: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
t.Errorf("not installed under its own build's name: %v", err)
}
all := strings.Join(sys.commands, "\n")
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
for _, want := range []string{
// Pinned to the digest GitHub publishes for the wheel.
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
"--require-hashes --no-deps -r",
// Computing on the card, not only importing.
"/bin/python -c import torch",
} {
if !strings.Contains(all, want) {
t.Errorf("no %q in:\n%s", want, all)
}
}
for _, c := range sys.commands {
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
t.Errorf("installed without the index its torch comes from: %s", c)
}
}
}
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, _ := pythonInstaller(t, gpuNode, sys)
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
}
volta := discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}
if _, err := chooseVLLM("0.30.0", "cu13", volta); err == nil || !strings.Contains(err.Error(), "7.5") {
t.Errorf("cu13 chosen for a V100 should be refused, saying why: %v", err)
}
if _, err := chooseVLLM("0.30.0", "cu200", gpuNode); err == nil {
t.Error("a build that does not exist was accepted")
}
}
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
in, _ := pythonInstaller(t, gpuNode, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
t.Fatalf("a build without kernels for the card: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
t.Error("marked installed")
}
}
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
// A machine vLLM is not published for is refused, not given the other
// build to fail with.
if _, err := chooseVLLM("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
!strings.Contains(err.Error(), "no Linux build") {
t.Errorf("an unknown architecture: %v", err)
}
if _, err := chooseVLLM("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
t.Errorf("a node with no GPU chosen cu129: %v", err)
}
}
func TestAV100GetsTheLastVLLMBuiltForIt(t *testing.T) {
v100 := discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
b, err := chooseVLLM(voltaUntil, "", v100)
if err != nil || b.Variant != "cu12" || b.Wheel != nil {
t.Fatalf("vLLM %s on a V100: %+v, %v; want PyPI's CUDA 12 build", voltaUntil, b, err)
}
if _, err := chooseVLLM("0.20.0", "", v100); err == nil {
t.Error("vLLM 0.20, with no Volta kernels in any build, was offered to a V100")
}
if _, err := chooseVLLM("0.30.0", "cu129", v100); err == nil || !strings.Contains(err.Error(), "7.5") {
t.Errorf("cu129 chosen for a V100 should be refused: %v", err)
}
}
+188 -32
View File
@@ -10,6 +10,7 @@
package provision
import (
"errors"
"fmt"
"slices"
"strings"
@@ -48,12 +49,29 @@ type Build struct {
// build must list a device with it, or it is not installed — a llama.cpp
// whose CUDA backend cannot load runs on the CPU without complaint.
GPUBackend string
// Wheel, when set, is the engine itself as a wheel from a GitHub
// release rather than from PyPI, checked against the digest GitHub
// publishes for it. ExtraIndex is a package index besides PyPI that the
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
Wheel *Wheel
ExtraIndex string
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
// compute something on one before the build counts as installed. A build
// with no kernels for the card — a GPU generation its torch dropped —
// imports and prints its version, and fails on its first model.
GPUCheck bool
// FreeGB is the disk space the installation needs before it starts: a
// Python engine is several gigabytes, and one that fills the disk halfway
// leaves the node worse off than one that never began.
FreeGB int
}
// Wheel is a package published as a GitHub release asset: the asset whose
// name starts with Prefix and ends with Suffix.
type Wheel struct {
Project, Repo, Tag, Prefix, Suffix string
}
// Recipe knows how to install one engine.
type Recipe struct {
// Repo is the GitHub repository whose releases carry the builds; empty
@@ -61,8 +79,10 @@ type Recipe struct {
Repo string
// PyPI is the project an engine installed from PyPI is published as.
PyPI string
// Choose picks the build for a machine, or says why none fits.
Choose func(version string, h discovery.Host) (Build, error)
// Choose picks the build for a machine, or says why none fits. variant
// is a build chosen for the node ("cu129"); empty leaves it to the
// machine.
Choose func(version, variant string, h discovery.Host) (Build, error)
}
// Recipes are the engines this agent can install.
@@ -104,7 +124,7 @@ type NotHere struct{ Reason string }
func (e NotHere) Error() string { return e.Reason }
func chooseUV(version string, h discovery.Host) (Build, error) {
func chooseUV(version, _ string, h discovery.Host) (Build, error) {
var arch string
switch h.Arch {
case "amd64":
@@ -121,30 +141,56 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
}, nil
}
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. vLLM's
// PyPI build is CUDA 13 from 0.20 (before it, CUDA 12, with CUDA 13 as the
// "+cu130" wheel on GitHub), and SGLang's from 0.5.
const minDriverForCUDA13 = 580
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
// Volta, so a V100 cannot run these builds at all.
const minComputeForCUDA13 = 7.5
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
// version compatibility, and on Volta.
// The CUDA 12 builds run on any driver from 525, by CUDA's minor version
// compatibility; the oldest GPU one has kernels for is minComputeFor's
// answer, which is Volta only up to voltaUntil.
const (
minDriverForCUDA12 = 525
minComputeForCUDA12 = 7.0
)
// cuda13Since is the first version of each Python engine built for CUDA 13.
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
// cuda13Since is the first version of each Python engine whose PyPI build is
// CUDA 13, read from vLLM's releases: 0.19.1 publishes "+cu130" beside a
// CUDA 12 default, 0.20.0 "+cu129" beside a CUDA 13 one.
var cuda13Since = map[string]string{"vLLM": "0.20", "SGLang": "0.5"}
// voltaUntil is the last vLLM built for Volta (7.0). Its release pipeline
// built with the Dockerfile's default arch list, "7.0 7.5 8.0 8.9 9.0 10.0
// 12.0"; from 0.20.0 every x86 build starts at 7.5, CUDA 12.9's included.
const voltaUntil = "0.19.1"
// minComputeFor is the oldest GPU a vLLM version's builds have kernels for
// on an architecture: what the project compiled, not what CUDA supports.
func minComputeFor(version, arch string) float64 {
if arch == "arm64" {
// Built for the Grace machines: 8.7 and up before 0.20, 8.0 and up
// since.
if versionBefore(version, "0.20") {
return 8.7
}
return 8.0
}
if versionBefore(version, "0.20") {
return minComputeForCUDA12
}
return minComputeForCUDA13
}
// cudaGate says whether a node can run this version of an engine, and which
// CUDA it is built for.
//
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
// nodes the older build (engine_overrides in the server's configuration)
// By version, because the line moved: a V100 cannot run vLLM 0.20 or later,
// and runs 0.19.1 — so a cluster with Volta and newer cards gives the Volta
// nodes the older version (engine_overrides in the server's configuration)
// rather than going without vLLM, or without the Volta nodes.
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
switch h.Arch {
@@ -159,21 +205,29 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
if versionBefore(version, cuda13Since[engine]) {
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
}
if engine == "vLLM" {
minCC = minComputeFor(version, h.Arch)
}
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
hint := ""
if cuda == "13" {
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
}
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, hint)
return "", fmt.Errorf("%s %s has kernels for GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, minCC, cc, olderHint(engine, cc))
}
return variant, nil
}
// olderHint says which version runs on a GPU too old for the one asked for.
func olderHint(engine string, cc float64) string {
if engine == "vLLM" && cc >= minComputeForCUDA12 && cc < minComputeForCUDA13 {
return "; the last vLLM built for it is " + voltaUntil +
": give these nodes that version with engine_overrides in the server's configuration"
}
return ""
}
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
@@ -202,22 +256,124 @@ func versionBefore(a, b string) bool {
// that load fails long after the installation looked finished.
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
func chooseVLLM(version string, h discovery.Host) (Build, error) {
variant, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
// version, built for the drivers and GPUs CUDA 13 left behind.
var VLLMVariants = []string{"cu13", "cu129"}
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
// chooseVLLM picks the vLLM build for a machine.
//
// Left to the machine (variant ""), it is CUDA 13 where that runs, and the
// same version's CUDA 12.9 build where only that does: one version across
// the cluster, which matters because the flags llmgw builds are checked
// against one version, and an older vLLM on the older cards would be a
// second set of flags nobody checked. A variant chosen for the node is that
// build, if the node can run it.
func chooseVLLM(version, variant string, h discovery.Host) (Build, error) {
// What no build of it gets past: a machine it is not published for, or
// one with no NVIDIA GPU. Only a CUDA requirement falls back to the
// CUDA 12.9 build.
switch h.Arch {
case "amd64", "arm64":
default:
return Build{}, fmt.Errorf("vLLM publishes no Linux build for %s", orUnknown(h.Arch))
}
return Build{
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"vllm==" + version},
Binary: "vllm",
Packages: compilerPackages,
FreeGB: 25,
}, nil
if h.NVIDIADriver == "" {
return Build{}, NotHere{"vLLM runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
}
pypi := func(v string) Build {
return Build{
Variant: v,
Python: pythonForEngines,
Requirements: []string{"vllm==" + version},
Binary: "vllm",
Packages: compilerPackages,
FreeGB: 25,
GPUCheck: true,
}
}
cu129 := func() (Build, error) {
if err := cudaFits("vLLM", version, "12.9", minDriverForCUDA12, minComputeFor(version, h.Arch), h); err != nil {
return Build{}, err
}
arch := map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch]
b := pypi("cu129")
b.Requirements = nil
// Found in the release's assets rather than named: the manylinux tag
// in the file's name changes between versions (2_28, 2_31, 2_34).
b.Wheel = &Wheel{
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + version,
Prefix: "vllm-" + version + "+cu129-", Suffix: "_" + arch + ".whl",
}
b.ExtraIndex = pytorchCU129
return b, nil
}
// From 0.20 the PyPI build is CUDA 13 and GitHub carries "+cu129".
modern := !versionBefore(version, cuda13Since["vLLM"])
switch variant {
case "":
v, err := cudaGate("vLLM", version, h)
if err == nil {
return pypi(v), nil
}
var nh NotHere
if errors.As(err, &nh) || !modern {
return Build{}, err
}
b, err129 := cu129()
if err129 != nil {
return Build{}, fmt.Errorf("%v; nor can its CUDA 12.9 build: %v", err, err129)
}
return b, nil
case "cu13":
if !modern {
return Build{}, fmt.Errorf("vLLM %s is built for CUDA 12; there is no CUDA 13 build of it", version)
}
if _, err := cudaGate("vLLM", version, h); err != nil {
return Build{}, err
}
return pypi("cu13"), nil
case "cu129":
if !modern {
// Before 0.20 PyPI's own build is CUDA 12, and is the one given.
v, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
}
return pypi(v), nil
}
if _, err := cudaGate("vLLM", version, h); err != nil {
var nh NotHere
if errors.As(err, &nh) {
return Build{}, err
}
}
return cu129()
}
return Build{}, fmt.Errorf("vLLM has no build %q; the builds are %s", variant, strings.Join(VLLMVariants, ", "))
}
func chooseSGLang(version string, h discovery.Host) (Build, error) {
// cudaFits checks a node against a CUDA's driver and GPU requirements.
func cudaFits(engine, version, cuda string, minDriver int, minCC float64, h discovery.Host) error {
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return fmt.Errorf("%s %s built for CUDA %s needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
return fmt.Errorf("%s %s built for CUDA %s needs GPUs of compute capability %.1f or newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, olderHint(engine, cc))
}
return nil
}
func chooseSGLang(version, chosen string, h discovery.Host) (Build, error) {
if chosen != "" {
return Build{}, fmt.Errorf("SGLang is installed in one build only, not %q", chosen)
}
variant, err := cudaGate("SGLang", version, h)
if err != nil {
return Build{}, err
@@ -252,7 +408,7 @@ func leadingInt(s string) int {
// else — no toolkit.
const cudaForLlamaCpp = "12.8"
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
func chooseLlamaCpp(version, _ string, h discovery.Host) (Build, error) {
// The published Linux builds are built on Ubuntu, but nothing about them
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
+1
View File
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
newRetryCmd(client),
newUpgradeCmd(client),
newEngineCheckCmd(),
newEngineBuildCmd(client),
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
newCapacityCmd(client, &asJSON),
+39
View File
@@ -0,0 +1,39 @@
package cli
import (
"encoding/json"
"fmt"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "engine-build <node> <engine> <build>",
Short: "Choose which build of an engine a GPU node installs",
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
" auto what the hardware runs (the default)\n" +
" cu13 the CUDA 13 build, from PyPI\n" +
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
" none no vLLM on this node\n\n" +
"The node is told at once and installs it in the background; a build it\n" +
"cannot compute with on its GPU fails to install, saying why, and the\n" +
"cluster page shows it. A V100 runs neither build of vLLM 0.20 or later;\n" +
"give it 0.19.1 with engine_overrides in the server's configuration.\n\n" +
" llmgw ctl engine-build gpu07 vllm cu129",
Args: cobra.ExactArgs(3),
RunE: func(cmd *cobra.Command, args []string) error {
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
args[0], args[1], args[2])
return nil
},
}
}
+6
View File
@@ -495,6 +495,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
// And what each model was measured to serve, for scaling and for
// choosing between profiles.
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
// The engine builds chosen per node on the cluster page.
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
log.Info("competing to be the control plane that places work",
"owner", lead.Owner())
}
@@ -504,6 +506,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
})
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
if ch := ctl.EngineChoices(); ch != nil {
svc.SetEngineChoices(ch.For)
ch.OnChange(svc.Reprovision)
}
// A standby must refuse agents as well as refusing to place. Commands
// travel down the stream the agent opened, so a standby holding one is a
// node the leader cannot command — and the leader's placements would fail
+57
View File
@@ -9,6 +9,7 @@ package adminapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -111,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
@@ -156,6 +158,10 @@ type NodeView struct {
// Provision is where installing each wanted engine stands. An engine is in
// Engines only once its installation here is ready.
Provision []inventory.EngineInstall `json:"provision,omitempty"`
// EngineBuilds are the builds chosen for this node, by engine: "auto"
// (its hardware decides), a build ("cu129"), or "none". Only on a node
// with GPUs, the only kind with a choice to make.
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
// OS is the operating system and architecture, as the agent read them.
OS string `json:"os,omitempty"`
// Warnings are facts about the machine that make it slow or unusable
@@ -473,6 +479,13 @@ func (a *API) Nodes() []NodeView {
}
slices.Sort(v.Engines)
v.Provision = n.Provision
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
chosen := ch.For(n.ID)
v.EngineBuilds = map[string]string{"vllm": "auto"}
for e, b := range chosen {
v.EngineBuilds[e] = b
}
}
if h := n.Host; h.OSID != "" {
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
@@ -1532,3 +1545,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
}
// NodeEngineRequest chooses one engine's build for a node.
type NodeEngineRequest struct {
Engine string `json:"engine"`
// Build is "auto", a build such as "cu129", or "none".
Build string `json:"build"`
}
// setNodeEngine chooses an engine's build for a node, and has the node
// install it.
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
ch := a.ctl.EngineChoices()
if ch == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
return
}
id := r.PathValue("id")
if _, ok := a.ctl.Inventory().Get(id); !ok {
writeError(w, http.StatusNotFound, "no node "+id)
return
}
var req NodeEngineRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, req)
}
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
ch := a.ctl.EngineChoices()
if ch == nil {
return errors.New("this control plane has no store to keep the choice in")
}
if _, ok := a.ctl.Inventory().Get(node); !ok {
return errors.New("no node " + node)
}
return ch.Set(ctx, node, engine, build, by)
}
@@ -0,0 +1,63 @@
package adminapi
import (
"context"
"log/slog"
"net/http"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// choiceStore keeps engine choices in memory.
type choiceStore struct{ m map[string]map[string]string }
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
return c.m, nil
}
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
if c.m[node] == nil {
c.m[node] = map[string]string{}
}
c.m[node][engine] = variant
return nil
}
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
var told []string
ch.OnChange(func(node string) error { told = append(told, node); return nil })
ctl.SetEngineChoices(ch)
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
t.Fatalf("status %d", code)
}
if len(told) != 1 || told[0] != "gpu07" {
t.Errorf("the node was not told: %v", told)
}
var nodes []NodeView
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
for _, n := range nodes {
switch n.ID {
case "gpu07":
if n.EngineBuilds["vllm"] != "cu129" {
t.Errorf("gpu07 shows %v", n.EngineBuilds)
}
case "cpu01":
if n.EngineBuilds != nil {
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
}
}
}
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
t.Errorf("%s: status %d", bad, code)
}
}
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
ch.For("gpu07")["vllm"] != "" {
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
}
}
+29 -3
View File
@@ -43,14 +43,40 @@ func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
}
return out
}
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" {
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
t.Errorf("the Volta node: %v", got)
}
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" {
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
t.Errorf("the Hopper node: %v", got)
}
// A node that disabled vLLM does not offer it, and is not asked for it.
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" {
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
t.Errorf("asked a node for an engine it does not install: %v", got)
}
}
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
s := &Service{}
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
s.SetEngineChoices(func(node string) map[string]string {
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
})
can := []string{"llamacpp", "vllm"}
variant := func(node string) (string, bool) {
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
if e.GetEngine() == "vllm" {
return e.GetVariant(), true
}
}
return "", false
}
if v, ok := variant("gpu07"); !ok || v != "cu129" {
t.Errorf("gpu07: %q, %v", v, ok)
}
if _, ok := variant("gpu08"); ok {
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
}
if v, ok := variant("gpu09"); !ok || v != "" {
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
}
}
+42 -5
View File
@@ -56,6 +56,10 @@ type Service struct {
// enginesFor is what a node is told to install, set once at start: the
// cluster's engines, with any override that picks the node laid over.
enginesFor func(node inventory.Node) map[string]string
// choices are the engine builds chosen per node, and installableBy what
// each connected node said it can install, for telling it again.
choices func(nodeID string) map[string]string
installableBy map[string][]string
// logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex
@@ -78,6 +82,29 @@ func (s *Service) SetEngines(e map[string]string) {
// of two GPU generations needs two vLLM builds, or vLLM on only one.
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
// SetEngineChoices sets where the builds chosen for each node come from: by
// engine, a build ("cu129") or "none". An engine not named is left to the
// node's hardware.
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
// Reprovision tells a connected node again what to install, after a choice
// for it changed; one not connected is told when it next connects.
func (s *Service) Reprovision(nodeID string) error {
if s.enginesFor == nil {
return nil
}
s.mu.RLock()
can, ok := s.installableBy[nodeID]
s.mu.RUnlock()
node, known := s.inv.Get(nodeID)
if !ok || !known {
return nil
}
return s.send(nodeID, &llmgwv1.ControlMessage{
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
})
}
// conn is one agent's open stream.
type conn struct {
nodeID string
@@ -188,7 +215,14 @@ func (s *Service) Connect(
// node rebuilt from nothing is told the same as one that has been here a
// month, and a changed version reaches every node on its next connect.
if s.enginesFor != nil {
p := s.provisionFor(node, reg)
can := installable(reg)
s.mu.Lock()
if s.installableBy == nil {
s.installableBy = map[string][]string{}
}
s.installableBy[nodeID] = can
s.mu.Unlock()
p := s.provisionFor(node, can)
if err := stream.Send(&llmgwv1.ControlMessage{
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
}); err != nil {
@@ -856,15 +890,18 @@ func rate(before, now uint64, secs float64) float64 {
// provisionFor is what a node is told to install: the engines decided for
// it, those it can install.
func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision {
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
p := &llmgwv1.Provision{}
can := installable(reg)
wants := s.enginesFor(node)
var chosen map[string]string
if s.choices != nil {
chosen = s.choices(node.ID)
}
for _, e := range slices.Sorted(maps.Keys(wants)) {
if !slices.Contains(can, e) {
if !slices.Contains(can, e) || chosen[e] == "none" {
continue
}
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]})
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
}
return p
}
+18 -9
View File
@@ -75,15 +75,18 @@ func DefaultConfig() Config {
// Controller owns the control plane's state.
type Controller struct {
cfg Config
log *slog.Logger
inv *inventory.Store
cat *catalog.Live
rec *reconciler.Reconciler
lead *Leadership
disp reconciler.Dispatcher
sink EndpointSink
usage UsageSource
// engineChoices are the engine builds chosen per node; nil without a
// store to keep them in.
engineChoices *EngineChoices
cfg Config
log *slog.Logger
inv *inventory.Store
cat *catalog.Live
rec *reconciler.Reconciler
lead *Leadership
disp reconciler.Dispatcher
sink EndpointSink
usage UsageSource
mu sync.Mutex
last string // fingerprint of the last published list
@@ -408,3 +411,9 @@ func EngineKeys(secret []byte) func(replicaID string) string {
// QuotaUse is every capacity quota beside what its scope holds.
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
// SetEngineChoices gives the controller the engine builds chosen per node.
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
// EngineChoices are the engine builds chosen per node, or nil.
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
+108
View File
@@ -0,0 +1,108 @@
package control
import (
"context"
"fmt"
"log/slog"
"maps"
"slices"
"strings"
"sync"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
)
// EngineChoiceStore is where the engine builds chosen per node are kept.
type EngineChoiceStore interface {
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
}
// EngineChoices are the engine builds chosen for nodes, rather than left to
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
// also install on, or no vLLM at all. Kept in the store, held here for the
// agent service to ask on every registration, and pushed to a node the
// moment its choice changes.
type EngineChoices struct {
st EngineChoiceStore
log *slog.Logger
mu sync.RWMutex
byNode map[string]map[string]string
notify func(node string) error
}
// NewEngineChoices reads what was chosen before.
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
if got, err := st.NodeEngines(ctx); err != nil {
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
} else {
c.byNode = got
}
return c
}
// OnChange is called with a node whose choice changed, to tell it.
func (c *EngineChoices) OnChange(f func(node string) error) {
c.mu.Lock()
c.notify = f
c.mu.Unlock()
}
// For is what was chosen for one node, by engine: a build, or "none".
func (c *EngineChoices) For(node string) map[string]string {
c.mu.RLock()
defer c.mu.RUnlock()
return maps.Clone(c.byNode[node])
}
// EngineBuilds are what can be chosen for each engine that has a choice:
// "auto" is the node's hardware deciding, "none" no engine at all.
func EngineBuilds(engine string) []string {
switch engine {
case "vllm":
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
case "sglang":
return []string{"auto", "none"}
}
return nil
}
// Set chooses an engine's build for a node, keeps it, and tells the node.
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
builds := EngineBuilds(engine)
if builds == nil {
return fmt.Errorf("%s has no builds to choose between", engine)
}
if !slices.Contains(builds, build) {
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
}
variant := build
if build == "auto" {
variant = ""
}
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
return err
}
c.mu.Lock()
if variant == "" {
delete(c.byNode[node], engine)
} else {
if c.byNode[node] == nil {
c.byNode[node] = map[string]string{}
}
c.byNode[node][engine] = variant
}
notify := c.notify
c.mu.Unlock()
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
if notify != nil {
if err := notify(node); err != nil {
// Kept and applied on its next connection; said, since a node that
// is not connected is the usual reason.
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
}
}
return nil
}
+1 -1
View File
@@ -581,7 +581,7 @@ func Evaluate(f Facts) []Finding {
// the same decision the agent makes, so the report and the node agree.
func vllmFindings(f Facts, add func(Level, string, string, string)) {
v := f.VLLMVersion
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
b, err := provision.Recipes["vllm"].Choose(v, "", f.Host)
var nh provision.NotHere
switch {
case errors.As(err, &nh):
+9 -5
View File
@@ -153,11 +153,15 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
// What vLLM cannot do keeps vLLM off the node, not the node out of
// the cluster: it still serves llama.cpp.
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"},
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"},
Warn, "before 0.26 is built for CUDA 12"},
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
// A V100 has no build of vLLM 0.20 or later, and is told which has.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.30.0"}, Warn, "the last vLLM built for it is 0.19.1"},
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
+5
View File
@@ -196,6 +196,11 @@ func (r *Router) Handler() http.Handler {
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
pages.SetCapacity(c)
}
// The engine build each GPU node installs, chosen on the cluster
// page.
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
pages.RegisterNodeEngines(mux, e)
}
pages.RegisterAdminQuotas(mux, r.store, r.acct)
// And the report of what that tree actually used.
pages.RegisterAdminUsage(mux, r.store, r.acct)
+1
View File
@@ -45,6 +45,7 @@ var copyOrder = []string{
// derived from this, and a new secret would lock the router out of all
// of them at the moment of the move.
"cluster_secrets",
"node_engines",
}
// notCopied are tables deliberately left behind.
@@ -0,0 +1,11 @@
-- The build of an engine chosen for a node, where it is not left to the
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
-- or no vLLM at all. A node with no row here installs what its hardware runs.
CREATE TABLE node_engines (
node_id TEXT NOT NULL,
engine TEXT NOT NULL,
variant TEXT NOT NULL,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (node_id, engine)
);
@@ -0,0 +1,11 @@
-- The build of an engine chosen for a node, where it is not left to the
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
-- or no vLLM at all. A node with no row here installs what its hardware runs.
CREATE TABLE node_engines (
node_id TEXT NOT NULL,
engine TEXT NOT NULL,
variant TEXT NOT NULL,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (node_id, engine)
);
+44
View File
@@ -0,0 +1,44 @@
package store
import (
"context"
"time"
)
// NodeEngines are the engine builds chosen per node: node id, then engine,
// then the build ("cu129") or "none". A node or engine not here is left to
// the node's hardware.
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]map[string]string{}
for rows.Next() {
var node, engine, variant string
if err := rows.Scan(&node, &engine, &variant); err != nil {
return nil, err
}
if out[node] == nil {
out[node] = map[string]string{}
}
out[node][engine] = variant
}
return out, rows.Err()
}
// SetNodeEngine chooses an engine's build for a node; an empty variant
// leaves it to the node's hardware again.
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
if variant == "" {
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
return err
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
node, engine, variant, time.Now().UnixMilli(), by)
return err
}
+24
View File
@@ -0,0 +1,24 @@
package store
import "testing"
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
t.Fatal(err)
}
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
t.Fatal(err)
}
got, err := db.NodeEngines(ctx)
if err != nil || got["gpu07"]["vllm"] != "none" {
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
}
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
t.Fatal(err)
}
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
t.Errorf("left to the node again, still %v", got)
}
}
+49
View File
@@ -0,0 +1,49 @@
package web
import (
"context"
"net/http"
"net/url"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
// such as "cu129", or "none".
type NodeEngineAdmin interface {
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
}
// RegisterNodeEngines mounts the choice of engine builds on the cluster
// page: for an administrator, with the session's CSRF token, and answered by
// going back to the page with what happened.
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
sess, _, _ := p.session(r)
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
q := url.Values{}
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
q.Set("problem", node+": "+err.Error())
} else {
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
"; the node is told now, and installs it in the background")
}
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
}))
}
func buildWords(b string) string {
switch b {
case "auto":
return "in the build its hardware runs"
case "none":
return "no longer"
}
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
}
+75
View File
@@ -0,0 +1,75 @@
package web
import (
"context"
"io"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
type buildFleet struct{ fakeFleet }
func (buildFleet) Nodes() []adminapi.NodeView {
return []adminapi.NodeView{
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
EngineBuilds: map[string]string{"vllm": "cu129"},
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
{ID: "cpu01", State: "ready"},
}
}
type recordedChoice struct{ node, engine, build, by string }
type fakeEngineAdmin struct{ got []recordedChoice }
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
f.got = append(f.got, recordedChoice{node, engine, build, by})
return nil
}
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
fe := &fakeEngineAdmin{}
p.RegisterNodeEngines(mux, fe)
body := mirrorGet(t, mux, "/admin").Body.String()
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
}
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
t.Error("a node with no GPU was offered a vLLM build")
}
if !strings.Contains(body, "CUDA 12.9 installing") {
t.Error("what is installed is not shown beside the choice")
}
// The read-only page on the admin listener has no session, so no form.
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
if strings.Contains(ro, "/engines\"") {
t.Error("the admin listener's page offers a change it cannot sign")
}
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
}
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
t.Errorf("redirected to %s, saying nothing", loc)
}
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
t.Errorf("a forged form was accepted: %d", w.Code)
}
}
+13 -7
View File
@@ -19,13 +19,16 @@ type DashboardData struct {
// of a GPU table that would only say there are none.
CPU cpuCapacity
// Menu is the shared top menu; Nav below is what the cluster page adds.
Menu Nav
Nodes []adminapi.NodeView
Models []adminapi.ModelView
Replicas []adminapi.ReplicaView
Unplaced []adminapi.ExplainView
Usage []store.KeyUsage
Summary adminapi.Summary
Menu Nav
// Notice and Problem are what the last change made from this page did,
// carried over the redirect back to it.
Notice, Problem string
Nodes []adminapi.NodeView
Models []adminapi.ModelView
Replicas []adminapi.ReplicaView
Unplaced []adminapi.ExplainView
Usage []store.KeyUsage
Summary adminapi.Summary
// Fabrics is the physical fabrics and what disagrees about them, from a
// source that can say.
Fabrics *fabrics.Report
@@ -170,6 +173,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
Now: time.Now().UTC().Format(time.RFC3339),
AuthEnabled: usage != nil,
}
if d.Menu.Admin {
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
}
if src != nil {
d.Nodes = src.Nodes()
d.CPU = cpuCapacityOf(d.Nodes)
+26 -1
View File
@@ -50,8 +50,9 @@
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
<h2>Nodes</h2>
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
{{if .Nodes}}
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
{{range .Nodes}}<tr>
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
@@ -66,6 +67,23 @@
was never asked about its disk must not read as a full one. */}}
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
What is installed beside it, so a choice and its outcome are read
together. Only a GPU node has one; changing it is for an
administrator, on the page reached through a session. */}}
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
{{if and $.Menu.CSRF .EngineBuilds}}
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<input type="hidden" name="engine" value="vllm">
<select name="build" title="the vLLM build this node installs">
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
</select>
<noscript><button type="submit">Save</button></noscript>
</form>
{{end}}
{{else}}—{{end}}</td>{{end}}
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
@@ -75,6 +93,13 @@
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
{{end}}</table>
<script>
// A build chosen is saved as it is chosen, with the page reloaded on the
// answer: one select, no separate button to forget.
document.querySelectorAll("form.build-choice select").forEach(function (s) {
s.addEventListener("change", function () { s.form.submit(); });
});
</script>
{{else}}<p class="empty">No agents have registered.</p>{{end}}
<h2>Models</h2>
+20 -1
View File
@@ -26,6 +26,7 @@ import (
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -118,7 +119,25 @@ var funcs = template.FuncMap{
},
// shortNode is a node's name without what every node's name starts with.
"shortNode": strings.TrimPrefix,
"age": age,
// engineBuilds are the builds an engine can be given on a node, and
// buildLabel says one in words.
"engineBuilds": control.EngineBuilds,
"buildLabel": func(b string) string {
switch b {
case "auto":
return "auto: what the hardware runs"
case "cu13":
return "CUDA 13"
case "cu129":
return "CUDA 12.9"
case "cu12":
return "CUDA 12"
case "none":
return "none"
}
return b
},
"age": age,
// pctbar is a percentage as the minibar template takes it: whole, and
// held to 0..100 so a measurement a little over does not draw past its box.
"pctbar": func(v float64) struct{ P int } {