Compare commits

..
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 dc483d8c68 Choose each engine's build from the node's hardware, for every GPU generation
Release / build (push) Successful in 1m12s
CI / test (push) Successful in 3m52s
Release / notes (push) Successful in 3s
Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:35:58 +02:00
Ilker ManapandClaude Opus 5.5 43d7c4eac1 Let the upgrade repair what the field's catalog already holds
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m51s
tp on a llama.cpp GPU profile is read as gpus_per_node rather than
refused, and a Hugging Face repository page as a vLLM source as the
repository id, so models written that way are placed after the upgrade
instead of dropped from the catalog. Engines adopted after an upgrade
are checked for computing on their GPUs too.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:20:11 +02:00
Ilker ManapandClaude Opus 5.5 86e9a1cb83 Give V100s the last vLLM built for them, not a build without their kernels
Release / build (push) Successful in 1m13s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m47s
Read from vLLM's release pipeline: every x86 build from 0.20.0 has
kernels for compute 7.5 and up, the CUDA 12.9 one included, and 0.19.1
is the last built for 7.0. PyPI's build is CUDA 13 from 0.20, not 0.26.
A V100 asked for a newer vLLM is refused, naming 0.19.1 for
engine_overrides; the CUDA 12.9 fallback is for older drivers on
Turing and newer. The cu129 wheel is found among the release's assets,
since its manylinux tag differs between versions.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 14:04:45 +02:00
Ilker ManapandClaude Opus 5.5 4f280a3546 Give each GPU node a vLLM build it can run, chosen on the cluster page
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m55s
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.

Refs #2.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 12:39:14 +02:00
38 changed files with 1540 additions and 277 deletions

No files matched your search

+57 -12
View File
@@ -34,6 +34,11 @@ reported ready while its GPU sat reserved and idle. Three checks now stop it:
(`llama-server --list-devices`). None, on a node with GPUs, is a failed
install, with what `ldd` says `libggml-cuda.so` is missing. The binary's
own `ldd` could not say it: the backend is opened at run time.
- **After the upgrade**, engines the agent finds already running are held to
the same check when it adopts them: one serving a GPU placement on the CPU
is stopped, and its replacement starts through the check. The installed
CUDA build is checked again within ten minutes, and installed afresh if it
sees no GPU.
- **Starting** a shard on GPUs is ready only when its process holds memory on
each of them (`nvidia-smi --query-compute-apps`, its process group
included). One that does not is stopped and reported failed: "placed on GPU
@@ -68,8 +73,10 @@ file of HTML.
it and the manifest carries it; a node checks every file of a set against
it before using the set, and fetches one that differs again.
- **`llmgw mirror add` takes several URLs** into one version.
- **A Hugging Face repository page is refused as a source**, saying to mirror
it or to give the repository id for vLLM to fetch itself.
- **A Hugging Face repository page given as a source** is read as the
repository id for a model vLLM or SGLang runs, which the engine downloads
itself (the node then needs to reach Hugging Face), and refused for one
llama.cpp runs, which needs a file.
With a vLLM profile of `tp: 4`, placement reserves four GPUs of one node for
such a model, as before.
@@ -123,6 +130,46 @@ leaving the model with nothing. Now:
stopping — is waited for, as before.
- **`ctl restart --in-place`** stops each replica first, whatever room there is.
### Every GPU node gets engine builds it can run, by itself
Which build of an engine a node installs follows its hardware — the CPU
architecture, the driver, and the oldest GPU on it — from what each project
publishes and how it was compiled. Each engine has a list of builds, best
first; the node takes the first its hardware meets, and the first of those
that installs and is seen to compute on the GPU. A build the table is wrong
about costs its download, and the next one is tried.
| | llama.cpp | vLLM |
|---|---|---|
| Driver 580+, Turing and newer | CUDA 13.3 | the version asked for, CUDA 13 (PyPI) |
| Driver 525–579 | CUDA 12.8 (x64): its machine code for 8.6, 8.9, 12.0; the GPUs it ships as PTX need 570 | the version asked for, CUDA 12.9 (vLLM's GitHub release, PyTorch's cu129 torch) |
| Volta (V100) | CUDA 12.8 on driver 570+, else Vulkan | **0.19.1**, the newest vLLM built for 7.0: no build of 0.20 or later has its kernels |
| Pascal, Maxwell | CUDA 12.8 on 570+, else Vulkan | none: no vLLM is built for them |
| arm64 GPU (GH200, Grace) | CUDA 13.3, else Vulkan — never the CPU build | CUDA 13 or 12.9, 8.0 and newer |
| No NVIDIA GPU | the CPU build | none |
Read from the projects' release pipelines: vLLM's x86 builds start at 7.5
from 0.20.0, its PyPI build is CUDA 13 from 0.20, 0.19.1 was the last
compiled for 7.0 (PyPI's torch 2.10 is too); llama.cpp's CUDA 12.8 build
carries 5.0–9.0 as PTX and 8.6/8.9/12.0 as machine code, its CUDA 13.3 build
starts at 7.5.
- **A node running an older vLLM says so**: the cluster page's *vLLM build*
column shows the version installed, and the installation's detail says why
("vLLM 0.19.1, the newest built for compute capability 7.0, instead of
0.30.0"). Profiles choose GPUs by class, so a model meant for the newer
vLLM is not placed on a V100 by accident.
- **The install computes on the GPU before it counts**: a small tensor
operation for vLLM, the devices llama.cpp lists for its GPU builds.
- **The build can still be chosen per node** on the cluster page (auto, CUDA
13, CUDA 12.9, none), with `llmgw ctl engine-build NODE vllm BUILD`, or
`POST /api/v1/nodes/{id}/engines`; kept in the store and sent to the node
at once.
- **Whether an engine computes on its GPUs is read from every process
nvidia-smi lists**, not compute processes alone: the Vulkan build is a
graphics one. A llama.cpp shard is given `GGML_VK_VISIBLE_DEVICES` beside
`CUDA_VISIBLE_DEVICES`.
### Engines per node, and nodes that do not run them all
A cluster with older GPUs could not add them cleanly: `node add` refused a
@@ -136,13 +183,10 @@ and `--ignore-checks` passed over every other problem with it.
written as `engines.disabled` in its `agent.yaml`, never installed or
advertised there, and not checked by doctor.
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
or labels a different version of an engine, or none: a CUDA 12 vLLM on
Volta beside a CUDA 13 one on newer cards.
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
the threshold had been CUDA 13's for every version, so the doctor's advice
to set an older version could not work. A refusal now says which versions
would.
or labels a different version of an engine, or none — for exceptions; the
build each GPU generation needs is chosen without it (above).
- **The requirements follow the version**, as the table above sets out; the
threshold had been CUDA 13's for every version.
- **An engine binary named in `agent.yaml` is used.** The agent had installed
its own beside it, which was then used first.
- The node page shows each GPU's compute capability.
@@ -176,9 +220,10 @@ placement reserves N cards of its `gpu_class` on one node, the engine sees
those N (`CUDA_VISIBLE_DEVICES`) and spreads the layers over them
(`--split-mode layer`, in proportion to each card's free memory, which for
whole cards is even). `tp` or `pp` on a llama.cpp profile was a width nothing
placed — the replica stayed unplaced or ran on one card — and is now refused
when the model is added, saying to write `gpus_per_node` instead; so is
`nodes` above one.
placed — the replica stayed unplaced or ran on one card. It is now read as
`gpus_per_node`, so a profile written that way is placed on that many cards
after the upgrade, rather than dropped; only both given and disagreeing, or
`nodes` above one, is refused.
## v0.10.6 — 2026-10-01
+16 -5
View File
@@ -2370,9 +2370,12 @@ func (x *Provision) GetEngines() []*EngineWant {
// EngineWant names one engine at one version.
type EngineWant struct {
state protoimpl.MessageState `protogen:"open.v1"`
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
state protoimpl.MessageState `protogen:"open.v1"`
Engine string `protobuf:"bytes,1,opt,name=engine,proto3" json:"engine,omitempty"` // "llamacpp"
Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` // "b10991"
// variant is the build to install, when one is chosen for the node rather
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
Variant string `protobuf:"bytes,3,opt,name=variant,proto3" json:"variant,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
@@ -2421,6 +2424,13 @@ func (x *EngineWant) GetVersion() string {
return ""
}
func (x *EngineWant) GetVariant() string {
if x != nil {
return x.Variant
}
return ""
}
// ProvisionStatus is where one engine's installation stands on a node.
type ProvisionStatus struct {
state protoimpl.MessageState `protogen:"open.v1"`
@@ -4068,11 +4078,12 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\";\n" +
"\tProvision\x12.\n" +
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\">\n" +
"\aengines\x18\x01 \x03(\v2\x14.llmgw.v1.EngineWantR\aengines\"X\n" +
"\n" +
"EngineWant\x12\x16\n" +
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\"\x8b\x01\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
"\avariant\x18\x03 \x01(\tR\avariant\"\x8b\x01\n" +
"\x0fProvisionStatus\x12\x16\n" +
"\x06engine\x18\x01 \x01(\tR\x06engine\x12\x18\n" +
"\aversion\x18\x02 \x01(\tR\aversion\x12\x18\n" +
+3
View File
@@ -465,6 +465,9 @@ message Provision {
message EngineWant {
string engine = 1; // "llamacpp"
string version = 2; // "b10991"
// variant is the build to install, when one is chosen for the node rather
// than left to its hardware: "cu13", "cu129". Empty is the agent's choice.
string variant = 3;
}
// ProvisionStatus is where one engine's installation stands on a node.
+31 -10
View File
@@ -367,7 +367,7 @@ Nothing to do: the agent installs them once the node joins, under
| Engine | On | How |
|---|---|---|
| llama.cpp | every node | the release build for its processor, CUDA where there is a GPU, checked against GitHub's checksum |
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
@@ -376,10 +376,17 @@ node gets the same dependencies whenever it is built, and installed with
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
the environment as `llmgw-requirements.txt`. It needs:
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
Volta (7.0);
- **the build its hardware runs**, chosen by the node. Driver 580 and newer:
the CUDA 13 build from PyPI. Driver 525 to 579: the same version built for
CUDA 12.9, from vLLM's GitHub release (checked against the digest GitHub
publishes) with PyTorch's CUDA 12.9 torch. Both have kernels for Turing
(7.5) and newer, 8.0 and newer on aarch64. **A V100** (Volta, 7.0) gets
**vLLM 0.19.1**, the newest built for it — no build from 0.20 on has its
kernels — and the cluster page shows it running that version; run `llmgw
ctl engine-check vllm` there once, since llmgw's flags are checked against
the current one. Pascal and older run no vLLM. Whatever is chosen, the
install ends by computing something on the GPU; a build that cannot is
passed over for the next;
- **about 25 GB free** under the engines directory while it installs;
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
@@ -400,6 +407,19 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
build** column — *auto* (what its hardware runs, the default), *CUDA 13*,
*CUDA 12.9*, or *none* — or from the command line:
```
llmgw ctl engine-build gpu07 vllm cu129
```
The choice is kept by the control plane and the node is told at once; it
installs the build in the background, and the column shows how that stands.
#### Different GPUs, different engines
A cluster of two GPU generations may need two builds of an engine, or one on
@@ -411,8 +431,8 @@ cluster's for them, in order:
engines:
vllm: "0.30.0" # CUDA 13: Turing and newer
engine_overrides:
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
- gpu_class: a10g # pin a class to a version you have tested
engines: { vllm: "0.29.0" }
- labels: { role: gguf } # nodes meant for llama.cpp alone
engines: { vllm: "" } # none
```
@@ -1253,9 +1273,10 @@ requests. On a slow model, set `timeouts.generation` higher.
(step 2) or every request costs zero, and a budget is something the dashboard
displays rather than something that stops anything. The quotas page says so
where it can, which is not the same as being told before you rely on it.
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
(step 4), or install one built for your CUDA yourself.
- **A V100 runs vLLM 0.19.1**, not the version the rest of the cluster runs:
no build from 0.20 on has Volta kernels, and the node picks the newest that
has. llmgw's flags are checked against the current version, so run `llmgw
ctl engine-check vllm` on a V100 node once.
- **Two control planes have not been run for long.** The leader election and
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
this has been run on in anger has had one. A single control plane is the
+11
View File
@@ -314,6 +314,17 @@ func (a *Agent) settleAdopted(sh *supervisor.Shard, o supervisor.Orphan) {
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
return
}
// An engine started by an agent older than the GPU check may be one
// serving a GPU placement on the CPU. Stopped, so its replacement starts
// through the check and fails there in words, if the build still sees
// no GPU — rather than going on unnoticed after the upgrade that could
// have caught it.
if err := a.onItsGPUs(ctx, eng, o.Spec, sh.Status().PID); err != nil {
a.log.Error("stopping an adopted engine that computes on the CPU though placed on GPUs",
"shard", o.ShardID, "replica", o.ReplicaID, "err", err)
_ = a.sup.Stop(ctx, o.ShardID)
return
}
a.log.Info("adopted engine is serving",
"shard", o.ShardID, "replica", o.ReplicaID)
}
+31 -32
View File
@@ -2,6 +2,7 @@ package discovery
import (
"context"
"encoding/xml"
"fmt"
"os/exec"
"strconv"
@@ -10,10 +11,14 @@ import (
// GPUProcesses reports which processes hold memory on each GPU, by the GPU's
// nvidia-smi index: what a process actually computes on, as opposed to what it
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose CUDA
// was told to use. A llama.cpp started with --n-gpu-layers on a GPU whose
// backend did not load runs on the CPU without a word, and this is where that
// shows — the process is on no GPU's list.
//
// From the full query rather than --query-compute-apps, which lists compute
// processes only: llama.cpp's Vulkan build is listed as a graphics one, and
// would have read as not on its GPU at all.
//
// A node with no nvidia-smi returns nil and no error, as discovery does.
func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
if binary == "" {
@@ -22,44 +27,38 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
}
binary = "nvidia-smi"
}
gpus, err := exec.CommandContext(ctx, binary, "--query-gpu=index,uuid", "--format=csv,noheader").Output()
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
apps, err := exec.CommandContext(ctx, binary, "--query-compute-apps=gpu_uuid,pid", "--format=csv,noheader").Output()
if err != nil {
return nil, fmt.Errorf("running %s: %w", binary, err)
}
return parseGPUProcesses(string(gpus), string(apps)), nil
return parseGPUProcesses(out)
}
// parseGPUProcesses joins nvidia-smi's GPU list (index, uuid) to its compute
// apps (gpu_uuid, pid): the apps are listed by UUID, placement speaks index.
func parseGPUProcesses(gpus, apps string) map[int][]int {
index := map[string]int{}
for _, line := range strings.Split(gpus, "\n") {
i, uuid, ok := strings.Cut(line, ",")
if !ok {
continue
}
n, err := strconv.Atoi(strings.TrimSpace(i))
if err != nil {
continue
}
index[strings.TrimSpace(uuid)] = n
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
// index order, and the processes on it.
type smiLog struct {
GPUs []struct {
Processes []struct {
PID string `xml:"pid"`
Type string `xml:"type"`
} `xml:"processes>process_info"`
} `xml:"gpu"`
}
// parseGPUProcesses reads the report. GPUs are listed in index order, which
// is nvidia-smi's own numbering — the one placement assigns by.
func parseGPUProcesses(report []byte) (map[int][]int, error) {
var log smiLog
if err := xml.Unmarshal(report, &log); err != nil {
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
}
out := map[int][]int{}
for _, line := range strings.Split(apps, "\n") {
uuid, p, ok := strings.Cut(line, ",")
if !ok {
continue
for i, g := range log.GPUs {
for _, p := range g.Processes {
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
out[i] = append(out[i], pid)
}
}
pid, err := strconv.Atoi(strings.TrimSpace(p))
i, known := index[strings.TrimSpace(uuid)]
if err != nil || !known {
continue
}
out[i] = append(out[i], pid)
}
return out
return out, nil
}
+25 -8
View File
@@ -5,17 +5,34 @@ import (
"testing"
)
func TestGPUProcessesAreListedByIndexNotUUID(t *testing.T) {
gpus := "0, GPU-aaa\n1, GPU-bbb\n2, GPU-ccc\n"
// Two processes on GPU 1, one on GPU 2, and a GPU this list has never
// heard of, which is dropped rather than guessed at.
apps := "GPU-bbb, 4100\nGPU-ccc, 4200\nGPU-bbb, 4300\nGPU-zzz, 9999\n"
got := parseGPUProcesses(gpus, apps)
func TestGPUProcessesOfEveryKindAreListedByIndex(t *testing.T) {
report := `<?xml version="1.0" ?>
<nvidia_smi_log>
<driver_version>580.65.06</driver_version>
<gpu id="00000000:17:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
<processes></processes>
</gpu>
<gpu id="00000000:65:00.0"><product_name>Tesla V100-SXM2-32GB</product_name>
<processes>
<process_info><pid>4100</pid><type>C</type><process_name>llama-server</process_name></process_info>
<process_info><pid>4300</pid><type>G</type><process_name>llama-server</process_name></process_info>
</processes>
</gpu>
<gpu id="00000000:B3:00.0"><processes>
<process_info><pid>4200</pid><type>C+G</type></process_info>
</processes></gpu>
</nvidia_smi_log>`
got, err := parseGPUProcesses([]byte(report))
if err != nil {
t.Fatal(err)
}
// A Vulkan engine is a graphics process ("G") and is on its GPU all the
// same; a compute-only query would have missed it.
want := map[int][]int{1: {4100, 4300}, 2: {4200}}
if !reflect.DeepEqual(got, want) {
t.Errorf("got %v, want %v", got, want)
}
if got := parseGPUProcesses(gpus, ""); len(got) != 0 {
t.Errorf("no compute apps should be no GPU in use: %v", got)
if _, err := parseGPUProcesses([]byte("not xml")); err == nil {
t.Error("a report that does not parse was read as no processes")
}
}
+8
View File
@@ -168,6 +168,14 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
if s.APIKey != "" {
env["LLAMA_API_KEY"] = s.APIKey
}
// The Vulkan build — what a GPU older than the CUDA builds gets — does
// not read CUDA_VISIBLE_DEVICES; this is its own. Vulkan numbers devices
// as the driver lists them, which on NVIDIA follows the same bus order
// as nvidia-smi; were it not to, the shard would not hold memory on its
// own GPUs, and starting it checks exactly that.
if len(s.GPUIndices) > 0 {
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
}
return []Command{{
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(env),
}}, nil
+1 -1
View File
@@ -88,7 +88,7 @@ func (a *Agent) setProvision(p *llmgwv1.Provision) {
// the installed build is used first — and quietly replace theirs.
continue
}
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion(), Variant: e.GetVariant()})
}
a.mu.Lock()
a.provWants = wants
+68 -16
View File
@@ -27,6 +27,9 @@ import (
type Want struct {
Engine string
Version string
// Variant is a build chosen for this node ("cu129"); empty leaves it to
// the node's hardware.
Variant string
}
// The states an engine's installation can be in.
@@ -135,7 +138,12 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
"; install it by hand and name it in the agent's configuration"
return st
}
build, err := recipe.Choose(w.Version, in.Host)
if w.Variant != "" && !validVersion.MatchString(w.Variant) {
st.State = Unsupported
st.Detail = fmt.Sprintf("%q is not a build this agent will install", w.Variant)
return st
}
builds, err := recipe.Candidates(w.Version, w.Variant, in.Host)
if err != nil {
st.State, st.Detail = Unsupported, err.Error()
var nh NotHere
@@ -144,14 +152,18 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
}
return st
}
st.Variant = build.Variant
dir := filepath.Join(in.root(), w.Engine, w.Version+"-"+build.Variant)
// Installed before: check it still runs, and trust nothing else about it.
if _, err := os.Stat(filepath.Join(dir, marker)); err == nil {
// Installed before: whichever candidate is, best first, checked that it
// still runs and trusted for nothing else.
for _, build := range builds {
dir := in.dirFor(w, build)
if _, err := os.Stat(filepath.Join(dir, marker)); err != nil {
continue
}
st := st.as(w, build)
// The packages are checked too. Removing one is exactly the kind of
// change an unrelated upgrade makes, and ldd below would catch it only
// as a missing library with no hint of which package held it.
// change an unrelated upgrade makes, and ldd below would catch it
// only as a missing library with no hint of which package held it.
if err := in.packages(ctx, build.Packages, &st); err != nil {
st.State, st.Detail = Failed, err.Error()
return st
@@ -161,24 +173,64 @@ func (in *Installer) one(ctx context.Context, w Want) Status {
return in.ready(st, build, bin, version)
}
in.log().Warn("an installed engine no longer runs; installing it again",
"engine", w.Engine, "version", w.Version, "err", err)
"engine", w.Engine, "version", st.Version, "variant", build.Variant, "err", err)
break
}
if err := in.install(ctx, recipe, w, build, dir, &st); err != nil {
st.State, st.Detail, st.Binary = Failed, err.Error(), ""
return st
// Each candidate in turn until one installs and is seen to work: a build
// the table was wrong about — a driver too old for it after all, a GPU
// it has no kernels for — costs its download, and the next is tried.
var tried []string
for _, build := range builds {
st := st.as(w, build)
if err := in.install(ctx, recipe, w, build, in.dirFor(w, build), &st); err != nil {
tried = append(tried, build.Variant+": "+err.Error())
in.log().Warn("an engine build did not work here; trying the next",
"engine", w.Engine, "variant", build.Variant, "err", err)
continue
}
bin, version, err := in.verify(ctx, in.dirFor(w, build), build)
if err != nil {
tried = append(tried, build.Variant+": "+err.Error())
continue
}
return in.ready(st, build, bin, version)
}
bin, version, err := in.verify(ctx, dir, build)
if err != nil {
st.State, st.Detail = Failed, err.Error()
return st
st.State, st.Binary = Failed, ""
st.Detail = strings.Join(tried, "; then ")
if len(builds) == 1 {
_, st.Detail, _ = strings.Cut(st.Detail, ": ")
}
return in.ready(st, build, bin, version)
return st
}
// dirFor is where a build of an engine is installed: by the version it is
// and its variant, so two builds of one version sit side by side.
func (in *Installer) dirFor(w Want, b Build) string {
return filepath.Join(in.root(), w.Engine, b.versionOr(w.Version)+"-"+b.Variant)
}
// versionOr is the version a build is: its own, when it is not the one asked
// for.
func (b Build) versionOr(asked string) string {
if b.Version != "" {
return b.Version
}
return asked
}
// as is a status for one build of what was asked for.
func (st Status) as(w Want, b Build) Status {
st.Variant, st.Version = b.Variant, b.versionOr(w.Version)
return st
}
func (in *Installer) ready(st Status, b Build, bin, version string) Status {
st.State, st.Binary = Ready, bin
st.Detail = strings.TrimSpace(filepath.Base(bin) + " " + version)
if b.Note != "" {
st.Detail += "; " + b.Note
}
var missing []string
for _, f := range b.CPUFlags {
if !in.Host.HasCPUFlag(f) {
+30 -5
View File
@@ -255,13 +255,38 @@ func TestAnArchiveCannotWriteOutsideItsDirectory(t *testing.T) {
func TestTheBuildFollowsTheHardware(t *testing.T) {
gpu := ubuntu
gpu.NVIDIADriver = "570.86.15"
b, err := chooseLlamaCpp("b1", gpu)
if err != nil || !strings.HasPrefix(b.Variant, "cuda") || len(b.Assets) != 2 {
t.Errorf("an NVIDIA node got %+v, %v; expected the CUDA build and its runtime", b, err)
got, err := llamaCppCandidates("b1", "", gpu)
if err != nil || got[0].Variant != "cuda-12.8" || len(got[0].Assets) != 2 || got[len(got)-1].Variant != "vulkan" {
t.Errorf("an NVIDIA node on driver 570 got %+v, %v; expected CUDA 12.8 with its runtime, then Vulkan", got, err)
}
if _, err := chooseLlamaCpp("b1", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
if _, err := llamaCppCandidates("b1", "", discovery.Host{OSID: "ubuntu", Arch: "riscv64"}); err == nil {
t.Error("a build was chosen for an architecture llama.cpp does not publish for")
}
for _, c := range []struct {
name string
host discovery.Host
want []string
}{
// Driver 580 and Turing: CUDA 13.3 first.
{"current", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 8.0}, []string{"cuda-13.3", "cuda-12.8", "vulkan"}},
// A V100 on driver 550: CUDA 12.8 ships 7.0 as PTX, which needs 570.
{"V100, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, []string{"vulkan"}},
// An RTX 3090 on 550: 8.6 is machine code in the CUDA 12.8 build.
{"3090, 550", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 8.6}, []string{"cuda-12.8", "vulkan"}},
// A Pascal card: CUDA 12.8, not 13.3.
{"P40", discovery.Host{Arch: "amd64", NVIDIADriver: "575.51.03", MinComputeCapability: 6.1}, []string{"cuda-12.8", "vulkan"}},
// arm64 with a GPU: never the CPU build.
{"GH200", discovery.Host{Arch: "arm64", NVIDIADriver: "580.65.06", MinComputeCapability: 9.0}, []string{"cuda-13.3", "vulkan"}},
} {
got, err := llamaCppCandidates("b1", "", c.host)
var names []string
for _, b := range got {
names = append(names, b.Variant)
}
if err != nil || strings.Join(names, " ") != strings.Join(c.want, " ") {
t.Errorf("%s: %v, %v; want %v", c.name, names, err, c.want)
}
}
slow := ubuntu
slow.CPUFlags = nil
@@ -400,7 +425,7 @@ func TestALibrarysVersionLinkIsStillUnpacked(t *testing.T) {
func TestACUDABuildThatSeesNoGPUIsNotInstalled(t *testing.T) {
// Found in the field: libggml-cuda.so in the build, cuBLAS nowhere the
// loader looks, and llama-server serving a GPU placement on the CPU.
cuda := "bin-ubuntu-cuda-" + cudaForLlamaCpp + "-x64.tar.gz"
cuda := "bin-ubuntu-cuda-12.8-x64.tar.gz"
rel := newRelease(map[string][]byte{
"llama-b1-" + cuda: tarGz(t, map[string]string{
"llama-b1/llama-server": "#!/bin/sh\n", "llama-b1/libggml-cuda.so": "x"}, nil),
+76 -10
View File
@@ -36,6 +36,10 @@ const lockName = "llmgw-requirements.txt"
func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Build, dir string, st *Status) error {
ctx, cancel := context.WithTimeout(ctx, pythonInstallTimeout)
defer cancel()
// The version this build is, which for a GPU older than every build of
// the one asked for is an older one: its publication date is the one
// the dependencies are resolved as of.
w.Version = b.versionOr(w.Version)
if err := in.packages(ctx, b.Packages, st); err != nil {
return err
@@ -86,28 +90,67 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
}
python := filepath.Join(tmp, "bin", "python")
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(b.Requirements, " "), asOf[:10])
reqs := b.Requirements
if b.Wheel != nil {
// The engine from its GitHub release, pinned to the digest GitHub
// publishes for the file: the same trust as an archive, and uv
// refuses a download that does not match it.
in.progress(st, "asking GitHub for %s %s's %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
sums, err := in.checksums(ctx, b.Wheel.Repo, b.Wheel.Tag)
if err != nil {
return err
}
var asset, sum string
for name, s := range sums {
if strings.HasPrefix(name, b.Wheel.Prefix) && strings.HasSuffix(name, b.Wheel.Suffix) {
asset, sum = name, s
}
}
if asset == "" {
return fmt.Errorf("%s %s publishes no %s…%s", b.Wheel.Repo, b.Wheel.Tag, b.Wheel.Prefix, b.Wheel.Suffix)
}
url := strings.TrimSuffix(in.downloadBase(), "/") + "/" + b.Wheel.Repo + "/releases/download/" +
b.Wheel.Tag + "/" + asset
reqs = append([]string{b.Wheel.Project + " @ " + url + "#sha256=" + sum}, reqs...)
}
// A second index, for the torch the engine was built on. Every package
// is taken from whichever of the two has the version resolved — both are
// the publishers' own — and each file is still pinned by its hash.
var index []string
if b.ExtraIndex != "" {
index = []string{"--extra-index-url", b.ExtraIndex, "--index-strategy", "unsafe-best-match"}
}
in.progress(st, "resolving %s against PyPI as of %s", strings.Join(reqs, " "), asOf[:10])
lock := filepath.Join(tmp, lockName)
args := []string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
"--generate-hashes", "--quiet", "--output-file", lock, "-"}
if out, err := in.runStdin(ctx, env, strings.Join(b.Requirements, "\n")+"\n", uv.Binary, args...); err != nil {
return fmt.Errorf("resolving %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
args := append([]string{"pip", "compile", "--python", python, "--exclude-newer", asOf,
"--generate-hashes", "--quiet", "--output-file", lock}, index...)
args = append(args, "-")
if out, err := in.runStdin(ctx, env, strings.Join(reqs, "\n")+"\n", uv.Binary, args...); err != nil {
return fmt.Errorf("resolving %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
in.progress(st, "downloading and installing %s and its dependencies: several gigabytes, which takes minutes",
strings.Join(b.Requirements, " "))
if out, err := in.run(ctx, env, uv.Binary, "pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock); err != nil {
return fmt.Errorf("installing %s: %v: %s", strings.Join(b.Requirements, " "), err, tail(out))
strings.Join(reqs, " "))
install := append([]string{"pip", "install", "--python", python,
"--require-hashes", "--no-deps", "-r", lock}, index...)
if out, err := in.run(ctx, env, uv.Binary, install...); err != nil {
return fmt.Errorf("installing %s: %v: %s", strings.Join(reqs, " "), err, tail(out))
}
in.progress(st, "checking that %s runs", w.Engine)
if _, _, err := in.verify(ctx, tmp, b); err != nil {
return err
}
if b.GPUCheck && in.Host.NVIDIADriver != "" {
in.progress(st, "checking that %s (%s) computes on this node's GPU", w.Engine, b.Variant)
if err := in.gpuCheck(ctx, python, w, b); err != nil {
return err
}
}
m, _ := json.MarshalIndent(markerFile{
Engine: w.Engine, Version: w.Version, Variant: b.Variant,
Binary: filepath.Join("bin", b.Binary), Requirements: b.Requirements,
Binary: filepath.Join("bin", b.Binary), Requirements: reqs,
ExcludeNewer: asOf, UV: UVVersion, Installed: time.Now().UTC(),
}, "", " ")
if err := os.WriteFile(filepath.Join(tmp, marker), m, 0o644); err != nil { //nolint:gosec // not secret
@@ -203,3 +246,26 @@ func tail(b []byte) string {
}
return strings.Join(lines, " / ")
}
// gpuProbe is what the environment is asked to run on the GPU: one small
// operation, which needs the torch to have kernels for the card.
const gpuProbe = `import torch
assert torch.cuda.is_available(), "torch sees no CUDA device"
x = torch.ones(64, device="cuda")
print(float((x * 2).sum()), torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))`
// gpuCheck runs gpuProbe in a freshly installed environment.
func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Build) error {
ctx, cancel := context.WithTimeout(ctx, 3*time.Minute)
defer cancel()
out, err := in.run(ctx, nil, python, "-c", gpuProbe)
if err != nil {
hint := ""
if strings.Contains(string(out), "no kernel image") {
hint = "; this build has no kernels for this GPU, so try the other build for this node"
}
return fmt.Errorf("%s %s (%s) installed but cannot compute on this node's GPU: %s%s",
w.Engine, w.Version, b.Variant, tail(out), hint)
}
return nil
}
+130 -2
View File
@@ -2,6 +2,7 @@ package provision
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"os"
@@ -23,6 +24,10 @@ type fakePython struct {
pmu sync.Mutex
commands []string
failPip bool
// noKernel makes the GPU check fail as a torch without kernels for the
// card does: in every build, or in noKernelIn only ("-cu13.").
noKernel bool
noKernelIn string
}
func (f *fakePython) run(ctx context.Context, env []string, name string, args ...string) ([]byte, error) {
@@ -43,6 +48,11 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
return []byte("0.30.0\n"), nil
case strings.HasSuffix(name, "/bin/python") && args[0] == "-c":
if f.noKernel || (f.noKernelIn != "" && strings.Contains(name, f.noKernelIn)) {
return []byte("RuntimeError: CUDA error: no kernel image is available for execution on the device\n"), errExit
}
return []byte("128.0 Tesla V100-SXM2-32GB (7, 0)\n"), nil
}
return f.fakeSystem.run(ctx, env, name, args...)
}
@@ -59,6 +69,8 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
uv := newRelease(map[string][]byte{
"uv-x86_64-unknown-linux-gnu.tar.gz": tarGz(t,
map[string]string{"uv-x86_64-unknown-linux-gnu/uv": "#!/bin/sh\n"}, nil),
// The fake GitHub serves every repository's release from this one.
"vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl": []byte("the cu129 wheel"),
})
gh := uv.serve(t)
mux := http.NewServeMux()
@@ -66,6 +78,9 @@ func pythonInstaller(t *testing.T, host discovery.Host, sys *fakePython) (*Insta
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-09-22T05:22:21.647189Z"},
{"upload_time_iso_8601":"2026-09-22T05:23:10.039243Z"}]}`))
})
mux.HandleFunc("/pypi/vllm/0.19.1/json", func(w http.ResponseWriter, _ *http.Request) {
_, _ = w.Write([]byte(`{"urls":[{"upload_time_iso_8601":"2026-04-02T10:00:00Z"}]}`))
})
pypi := httptest.NewServer(mux)
t.Cleanup(pypi.Close)
return &Installer{
@@ -130,8 +145,10 @@ func TestVLLMIsNotForEveryNode(t *testing.T) {
says string
}{
{"no GPU", ubuntu, Skipped, "has none"},
{"old driver", discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, Unsupported, "driver 580"},
{"V100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, Unsupported, "7.5"},
// Too old for CUDA 13 and for its CUDA 12.9 build both.
{"a driver too old for either", discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, Unsupported, "driver 525"},
// No vLLM was ever built for Pascal.
{"P100", discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.0}, Unsupported, "no vLLM is built for GPUs of compute capability 6.0"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
@@ -182,3 +199,114 @@ func TestAVersionPyPIDoesNotHaveIsSaidSo(t *testing.T) {
t.Errorf("got %s %q", got[0].State, got[0].Detail)
}
}
func TestAnOlderDriverGetsTheSameVLLMBuiltForCUDA12(t *testing.T) {
// A Turing card on a driver too old for CUDA 13: the same vLLM's CUDA
// 12.9 build, from the project's GitHub release, with the torch PyTorch
// publishes for 12.9.
volta := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5}
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, gh := pythonInstaller(t, volta, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Ready || got[0].Variant != "cu129" {
t.Fatalf("a Turing node on driver 550: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu129", marker)); err != nil {
t.Errorf("not installed under its own build's name: %v", err)
}
all := strings.Join(sys.commands, "\n")
wheel := "vllm-0.30.0+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"
for _, want := range []string{
// Pinned to the digest GitHub publishes for the wheel.
"vllm @ " + in.DownloadBase + "/vllm-project/vllm/releases/download/v0.30.0/" + wheel + "#sha256=" +
strings.TrimPrefix(gh.digests[wheel], "sha256:"),
"--extra-index-url https://download.pytorch.org/whl/cu129 --index-strategy unsafe-best-match -",
"--require-hashes --no-deps -r",
// Computing on the card, not only importing.
"/bin/python -c import torch",
} {
if !strings.Contains(all, want) {
t.Errorf("no %q in:\n%s", want, all)
}
}
for _, c := range sys.commands {
if strings.Contains(c, "pip install") && !strings.Contains(c, "download.pytorch.org/whl/cu129") {
t.Errorf("installed without the index its torch comes from: %s", c)
}
}
}
func TestABuildChosenForANodeIsTheOneInstalled(t *testing.T) {
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, _ := pythonInstaller(t, gpuNode, sys)
if got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0", Variant: "cu129"}}); got[0].Variant != "cu129" || got[0].State != Ready {
t.Errorf("cu129 chosen for a node that runs cu13 too: %+v", got[0])
}
if _, err := vllmCandidates("0.30.0", "cu200", gpuNode); err == nil {
t.Error("a build that does not exist was accepted")
}
}
func TestABuildThatCannotComputeOnTheCardIsNotInstalled(t *testing.T) {
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernel: true}
in, _ := pythonInstaller(t, gpuNode, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})
if got[0].State != Failed || !strings.Contains(got[0].Detail, "no kernels for this GPU") {
t.Fatalf("a build without kernels for the card: %+v", got[0])
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.30.0-cu13", marker)); err == nil {
t.Error("marked installed")
}
}
func TestOnlyACUDARequirementFallsBackToTheCUDA12Build(t *testing.T) {
// A machine vLLM is not published for is refused, not given the other
// build to fail with.
if _, err := vllmCandidates("0.30.0", "", discovery.Host{NVIDIADriver: "580.65.06"}); err == nil ||
!strings.Contains(err.Error(), "no Linux build") {
t.Errorf("an unknown architecture: %v", err)
}
if _, err := vllmCandidates("0.30.0", "cu129", ubuntu); !errors.As(err, new(NotHere)) {
t.Errorf("a node with no GPU chosen cu129: %v", err)
}
// Driver 580 and Turing: CUDA 13 first, the CUDA 12.9 build behind it.
got, err := vllmCandidates("0.30.0", "", gpuNode)
if err != nil || len(got) != 2 || got[0].Variant != "cu13" || got[1].Variant != "cu129" {
t.Errorf("candidates on a current node: %+v, %v", got, err)
}
}
func TestAV100GetsTheNewestVLLMBuiltForItByItself(t *testing.T) {
// 0.30.0 asked for; no build of it has Volta kernels. The node installs
// 0.19.1, the newest that has, and says so.
v100 := discovery.Host{OSID: "ubuntu", Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
in, _ := pythonInstaller(t, v100, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
if got.State != Ready || got.Version != "0.19.1" || got.Variant != "cu12" ||
!strings.Contains(got.Detail, "the newest built for compute capability 7.0, instead of 0.30.0") {
t.Fatalf("a V100: %+v", got)
}
if _, err := os.Stat(filepath.Join(in.Root, "vllm", "0.19.1-cu12", marker)); err != nil {
t.Errorf("not installed as what it is: %v", err)
}
all := strings.Join(sys.commands, "\n")
if !strings.Contains(all, "<<< vllm==0.19.1") || strings.Contains(all, "download.pytorch.org") {
t.Errorf("0.19.1 is PyPI's CUDA 12 build, with PyPI's torch:\n%s", all)
}
// Chosen cu13, which 0.19.1 has none of: refused, saying why.
if _, err := vllmCandidates("0.30.0", "cu13", v100); err == nil {
t.Error("a CUDA 13 build was offered to a V100")
}
}
func TestABuildThatFailsOnTheCardGivesWayToTheNext(t *testing.T) {
// The table says CUDA 13 runs here; the card says otherwise. The next
// build is installed rather than the node going without.
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}, noKernelIn: "-cu13."}
in, _ := pythonInstaller(t, gpuNode, sys)
got := in.Converge(t.Context(), []Want{{Engine: "vllm", Version: "0.30.0"}})[0]
if got.State != Ready || got.Variant != "cu129" {
t.Fatalf("after the CUDA 13 build failed on the card: %+v", got)
}
}
+323 -120
View File
@@ -48,12 +48,36 @@ type Build struct {
// build must list a device with it, or it is not installed — a llama.cpp
// whose CUDA backend cannot load runs on the CPU without complaint.
GPUBackend string
// Wheel, when set, is the engine itself as a wheel from a GitHub
// release rather than from PyPI, checked against the digest GitHub
// publishes for it. ExtraIndex is a package index besides PyPI that the
// rest resolves from too: the CUDA 12.9 torch the wheel was built on.
Wheel *Wheel
ExtraIndex string
// GPUCheck, on a node with an NVIDIA GPU, has the installed environment
// compute something on one before the build counts as installed. A build
// with no kernels for the card — a GPU generation its torch dropped —
// imports and prints its version, and fails on its first model.
GPUCheck bool
// Version is the engine version this build is, when it is not the one
// asked for: the newest built for this machine's GPUs, where the one
// asked for has no kernels for them. Empty is the version asked for.
Version string
// Note says why this build, when it is not the plain one: shown beside
// it wherever the installation is.
Note string
// FreeGB is the disk space the installation needs before it starts: a
// Python engine is several gigabytes, and one that fills the disk halfway
// leaves the node worse off than one that never began.
FreeGB int
}
// Wheel is a package published as a GitHub release asset: the asset whose
// name starts with Prefix and ends with Suffix.
type Wheel struct {
Project, Repo, Tag, Prefix, Suffix string
}
// Recipe knows how to install one engine.
type Recipe struct {
// Repo is the GitHub repository whose releases carry the builds; empty
@@ -61,8 +85,12 @@ type Recipe struct {
Repo string
// PyPI is the project an engine installed from PyPI is published as.
PyPI string
// Choose picks the build for a machine, or says why none fits.
Choose func(version string, h discovery.Host) (Build, error)
// Candidates are the builds that may run on a machine, best first, or
// why none can. The installer takes the first that installs and is seen
// to work — computing on the GPU, where there is one — so a build this
// table is wrong about costs a download, not a node. variant is a build
// chosen for the node; empty leaves the choice to the machine.
Candidates func(version, variant string, h discovery.Host) ([]Build, error)
}
// Recipes are the engines this agent can install.
@@ -71,10 +99,10 @@ type Recipe struct {
// from PyPI by uv, which is itself installed like llama.cpp: a GitHub release
// checked against its published checksum.
var Recipes = map[string]Recipe{
"llamacpp": {Repo: "ggml-org/llama.cpp", Choose: chooseLlamaCpp},
"vllm": {PyPI: "vllm", Choose: chooseVLLM},
"sglang": {PyPI: "sglang", Choose: chooseSGLang},
"uv": {Repo: "astral-sh/uv", Choose: chooseUV},
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
}
// Installable are the engines this agent can install, as it tells the
@@ -104,7 +132,15 @@ type NotHere struct{ Reason string }
func (e NotHere) Error() string { return e.Reason }
func chooseUV(version string, h discovery.Host) (Build, error) {
func uvCandidates(version, _ string, h discovery.Host) ([]Build, error) {
b, err := chooseUV(version, h)
if err != nil {
return nil, err
}
return []Build{b}, nil
}
func chooseUV(_ string, h discovery.Host) (Build, error) {
var arch string
switch h.Arch {
case "amd64":
@@ -121,32 +157,190 @@ func chooseUV(version string, h discovery.Host) (Build, error) {
}, nil
}
// minDriverForCUDA13 is the oldest NVIDIA driver CUDA 13 runs on. Every vLLM
// since 0.26 and SGLang 0.5 is built on torch for CUDA 13.
const minDriverForCUDA13 = 580
// minComputeForCUDA13 is the oldest GPU CUDA 13 supports: Turing. It dropped
// Volta, so a V100 cannot run these builds at all.
const minComputeForCUDA13 = 7.5
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
// version compatibility, and on Volta.
// What builds need of the NVIDIA side of a machine, read from what each
// project publishes and how it was built:
//
// - CUDA 13 needs driver 580; CUDA 12 builds run on driver 525 and newer,
// by CUDA's minor version compatibility — except code shipped only as
// PTX, which the driver compiles and so needs a driver as new as the
// CUDA it was compiled with.
// - Which GPUs a build has kernels for is the build's own list, not what
// its CUDA supports.
const (
minDriverForCUDA12 = 525
minComputeForCUDA12 = 7.0
minDriverForCUDA13 = 580
minDriverForCUDA12 = 525
)
// cuda13Since is the first version of each Python engine built for CUDA 13.
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
// gpuNeeds is what one build needs of a machine's GPUs and driver.
type gpuNeeds struct {
driver int
cc float64
}
// cudaGate says whether a node can run this version of an engine, and which
// CUDA it is built for.
// fits says why a machine cannot run a build, or nil.
func (n gpuNeeds) fits(what string, h discovery.Host) error {
if major := leadingInt(h.NVIDIADriver); n.driver > 0 && major > 0 && major < n.driver {
return fmt.Errorf("%s needs NVIDIA driver %d or newer; this node has %s", what, n.driver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; n.cc > 0 && cc > 0 && cc < n.cc {
return fmt.Errorf("%s has kernels for compute capability %.1f and newer; this node has %.1f", what, n.cc, cc)
}
return nil
}
// candidate is a build and what it needs.
type candidate struct {
b Build
needs gpuNeeds
what string
}
// pick keeps the candidates a machine can run, in order, and those a
// variant chosen for the node names; with none left, it says why each went.
func pick(engine, variant string, cands []candidate, h discovery.Host) ([]Build, error) {
var out []Build
var why []string
for _, c := range cands {
if variant != "" && c.b.Variant != variant {
continue
}
if err := c.needs.fits(c.what, h); err != nil {
why = append(why, err.Error())
continue
}
out = append(out, c.b)
}
switch {
case len(out) > 0:
return out, nil
case len(why) == 0 && variant != "":
var names []string
for _, c := range cands {
names = append(names, c.b.Variant)
}
return nil, fmt.Errorf("%s has no build %q here; it has %s", engine, variant, strings.Join(names, ", "))
}
return nil, fmt.Errorf("no build of %s runs on this node: %s", engine, strings.Join(why, "; "))
}
// --- vLLM ---
// VLLMVariants are the builds of vLLM a node can be given by choice: CUDA
// 13 from PyPI, and CUDA 12.9 from the project's GitHub release — the same
// version, built for drivers CUDA 13 does not run on.
var VLLMVariants = []string{"cu13", "cu129"}
// pytorchCU129 is where the CUDA 12.9 build's torch is published. PyPI's
// torch is built for CUDA 13; the cu129 vLLM wheel needs this one.
const pytorchCU129 = "https://download.pytorch.org/whl/cu129"
// vllmCUDA13Since is the first vLLM whose PyPI build is CUDA 13, read from
// its releases: 0.19.1 publishes "+cu130" beside a CUDA 12 default, 0.20.0
// "+cu129" beside a CUDA 13 one.
const vllmCUDA13Since = "0.20"
// vllmMinCompute is the oldest GPU a vLLM version's builds have kernels for,
// from its release pipeline: on x86, 7.0 (Volta) up to 0.19.1 — the
// Dockerfile's default "7.0 7.5 8.0 8.9 9.0 10.0 12.0" — and 7.5 from 0.20.0,
// CUDA 12.9 build included; on arm64, built for Grace machines, 8.7 before
// 0.20 and 8.0 since.
func vllmMinCompute(version, arch string) float64 {
old := versionBefore(version, vllmCUDA13Since)
switch {
case arch == "arm64" && old:
return 8.7
case arch == "arm64":
return 8.0
case old:
return 7.0
}
return 7.5
}
// vllmNewestFor is the newest vLLM built for GPUs of compute capability cc
// on an architecture, for GPUs older than every build of the version asked
// for: a V100 gets 0.19.1, where 0.20 and later have nothing for it. Nothing
// for a GPU no vLLM was built for.
var vllmNewestFor = []struct {
arch string
from float64 // the oldest GPU it covers
version string
}{
{"amd64", 7.0, "0.19.1"},
}
// vllmCandidates are the vLLM builds for a machine.
//
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
// nodes the older build (engine_overrides in the server's configuration)
// rather than going without vLLM, or without the Volta nodes.
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
// The version asked for, in the build the machine runs: CUDA 13 where the
// driver allows, the same version's CUDA 12.9 build where it does not. One
// version across the cluster wherever the GPUs allow it, because the flags
// llmgw builds are checked against it. Where they do not — GPUs older than
// every build of that version — the newest version built for them, said so
// beside the installation: vLLM on a V100 at all, rather than not.
func vllmCandidates(version, variant string, h discovery.Host) ([]Build, error) {
arch, err := gpuArch("vLLM", h)
if err != nil {
return nil, err
}
v, note := version, ""
if cc := h.MinComputeCapability; cc > 0 && cc < vllmMinCompute(version, h.Arch) {
older := ""
for _, n := range vllmNewestFor {
if n.arch == h.Arch && cc >= n.from {
older = n.version
}
}
if older == "" {
return nil, fmt.Errorf("no vLLM is built for GPUs of compute capability %.1f on %s; "+
"the node serves llama.cpp models", cc, arch)
}
v, note = older, fmt.Sprintf("vLLM %s, the newest built for compute capability %.1f, instead of %s", older, cc, version)
}
pypi := func(variant string) Build {
b := Build{
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"vllm==" + v},
Binary: "vllm",
Packages: compilerPackages,
FreeGB: 25,
GPUCheck: true,
Note: note,
}
if v != version {
b.Version = v
}
return b
}
minCC := vllmMinCompute(v, h.Arch)
if versionBefore(v, vllmCUDA13Since) {
// PyPI's own build is CUDA 12, and is the one there is; asked for
// cu129, this is it.
if variant == "cu129" {
variant = "cu12"
}
return pick("vLLM "+v, variant, []candidate{
{pypi("cu12"), gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v},
}, h)
}
cu129 := pypi("cu129")
cu129.Requirements = nil
// Found in the release's assets rather than named: the manylinux tag
// in the file's name changes between versions (2_28, 2_31, 2_34).
cu129.Wheel = &Wheel{
Project: "vllm", Repo: "vllm-project/vllm", Tag: "v" + v,
Prefix: "vllm-" + v + "+cu129-", Suffix: "_" + map[string]string{"amd64": "x86_64", "arm64": "aarch64"}[h.Arch] + ".whl",
}
cu129.ExtraIndex = pytorchCU129
return pick("vLLM "+v, variant, []candidate{
{pypi("cu13"), gpuNeeds{minDriverForCUDA13, minCC}, "vLLM " + v + " built for CUDA 13"},
{cu129, gpuNeeds{minDriverForCUDA12, minCC}, "vLLM " + v + " built for CUDA 12.9"},
}, h)
}
// gpuArch says whether a Python GPU engine is published for this machine
// and whether it has an NVIDIA GPU at all.
func gpuArch(engine string, h discovery.Host) (string, error) {
switch h.Arch {
case "amd64", "arm64":
default:
@@ -155,25 +349,42 @@ func cudaGate(engine, version string, h discovery.Host) (variant string, err err
if h.NVIDIADriver == "" {
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
}
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
if versionBefore(version, cuda13Since[engine]) {
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
}
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
engine, version, cuda, minDriver, h.NVIDIADriver)
}
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
hint := ""
if cuda == "13" {
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
}
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
engine, version, cuda, minCC, cc, hint)
}
return variant, nil
return h.Arch, nil
}
// --- SGLang ---
// sglangCUDA13Since is the first SGLang built for CUDA 13.
const sglangCUDA13Since = "0.5"
func sglangCandidates(version, variant string, h discovery.Host) ([]Build, error) {
if _, err := gpuArch("SGLang", h); err != nil {
return nil, err
}
b := Build{
Variant: "cu13",
Python: pythonForEngines,
Requirements: []string{"sglang==" + version},
// SGLang is launched as a module, so what the agent is given is the
// environment's interpreter, and the proof is importing it.
Binary: "python",
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
Packages: compilerPackages,
FreeGB: 25,
GPUCheck: true,
}
needs := gpuNeeds{minDriverForCUDA13, 7.5}
if versionBefore(version, sglangCUDA13Since) {
b.Variant, needs = "cu12", gpuNeeds{minDriverForCUDA12, 7.0}
}
return pick("SGLang "+version, variant, []candidate{{b, needs, "SGLang " + version}}, h)
}
// Python engines need a C compiler at run time: Triton builds a small
// launcher for its kernels the first time a model loads, and without gcc
// that load fails long after the installation looked finished.
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
// versionBefore reports whether version a is older than b, comparing the
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
// number counts as zero.
@@ -197,44 +408,6 @@ func versionBefore(a, b string) bool {
return false
}
// Python engines need a C compiler at run time: Triton builds a small
// launcher for its kernels the first time a model loads, and without gcc
// that load fails long after the installation looked finished.
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
func chooseVLLM(version string, h discovery.Host) (Build, error) {
variant, err := cudaGate("vLLM", version, h)
if err != nil {
return Build{}, err
}
return Build{
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"vllm==" + version},
Binary: "vllm",
Packages: compilerPackages,
FreeGB: 25,
}, nil
}
func chooseSGLang(version string, h discovery.Host) (Build, error) {
variant, err := cudaGate("SGLang", version, h)
if err != nil {
return Build{}, err
}
// SGLang is launched as a module, so what the agent is given is the
// environment's interpreter, and the proof is importing it.
return Build{
Variant: variant,
Python: pythonForEngines,
Requirements: []string{"sglang==" + version},
Binary: "python",
VersionArgs: []string{"-c", "import sglang; print('sglang', sglang.__version__)"},
Packages: compilerPackages,
FreeGB: 25,
}, nil
}
// leadingInt is the number a version string starts with: 580 of "580.65.06".
func leadingInt(s string) int {
n := 0
@@ -247,17 +420,30 @@ func leadingInt(s string) int {
return n
}
// cudaForLlamaCpp is the CUDA build taken on an NVIDIA node. The release also
// carries a matching cudart archive, so the node needs a driver and nothing
// else — no toolkit.
const cudaForLlamaCpp = "12.8"
// --- llama.cpp ---
func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
// The published Linux builds are built on Ubuntu, but nothing about them
// is Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which
// Debian 12 and RHEL 10 have and RHEL 9 does not. So the distribution's
// name decides nothing here. The build is tried everywhere, and ldd says
// whether this system can load it — in the words of the missing symbol.
// llamaCUDA12Real are the GPUs llama.cpp's CUDA 12.8 build carries machine
// code for. The rest of what it covers — 5.0, 6.1, 7.0, 7.5, 8.0, 9.0 — it
// carries as PTX, which a driver older than the CUDA it came from cannot
// compile.
var llamaCUDA12Real = []float64{8.6, 8.9, 12.0}
// llamaCppCandidates are the llama.cpp builds for a machine.
//
// On an NVIDIA node, the GPU builds best first: CUDA 13.3 (driver 580,
// Turing and newer), CUDA 12.8 on x64 (Maxwell to Blackwell; driver 570 for
// the GPUs it ships as PTX, 525 for those it ships as machine code), and
// Vulkan, which runs on any card whose driver has Vulkan — the generations
// CUDA builds no longer cover. Never the CPU build on an NVIDIA node: that is
// a GPU placement served on the CPU, which is what installing checks for.
// The CPU build where there is no NVIDIA GPU.
//
// The published Linux builds are built on Ubuntu, but nothing about them is
// Ubuntu's: they need glibc 2.34 and the libstdc++ of GCC 12, which Debian 12
// and RHEL 10 have and RHEL 9 does not. So the distribution's name decides
// nothing here. The build is tried everywhere, and ldd says whether this
// system can load it — in the words of the missing symbol.
func llamaCppCandidates(version, variant string, h discovery.Host) ([]Build, error) {
var arch string
switch h.Arch {
case "amd64":
@@ -265,36 +451,53 @@ func chooseLlamaCpp(version string, h discovery.Host) (Build, error) {
case "arm64":
arch = "arm64"
default:
return Build{}, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
return nil, fmt.Errorf("llama.cpp publishes no Linux build for %s", orUnknown(h.Arch))
}
b := Build{
Variant: "cpu",
Assets: []string{fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)},
// Everything else the build links is in the archive or in any current
// glibc system; OpenMP's runtime is the one a cloud image lacks, and
// the server does not start without it.
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
Binary: "llama-server",
}
if arch == "x64" {
// The archive carries a backend per CPU generation and loads the best
// one, down to SSE4.2. Below AVX2 it runs, several times slower.
b.CPUFlags = []string{"avx2"}
}
// The CUDA build is published for x64 at this version; on arm64 it is a
// newer CUDA that the driver may not support, so an arm64 GPU node runs
// the CPU build until someone decides otherwise.
if h.NVIDIADriver != "" && arch == "x64" {
name := fmt.Sprintf("bin-ubuntu-cuda-%s-x64.tar.gz", cudaForLlamaCpp)
b.Variant = "cuda-" + cudaForLlamaCpp
b.GPUBackend = "libggml-cuda.so"
b.Assets = []string{
fmt.Sprintf("llama-%s-%s", version, name),
fmt.Sprintf("cudart-llama-%s-%s", version, name),
base := func(variant string, assets ...string) Build {
b := Build{
Variant: variant,
Assets: assets,
// Everything else the build links is in the archive or in any
// current glibc system; OpenMP's runtime is the one a cloud image
// lacks, and the server does not start without it.
Packages: map[string][]string{"deb": {"libgomp1"}, "rpm": {"libgomp"}},
Binary: "llama-server",
}
if arch == "x64" {
// The archive carries a backend per CPU generation and loads the
// best one, down to SSE4.2. Below AVX2 it runs, several times
// slower.
b.CPUFlags = []string{"avx2"}
}
return b
}
return b, nil
if h.NVIDIADriver == "" {
return pick("llama.cpp", variant, []candidate{
{base("cpu", fmt.Sprintf("llama-%s-bin-ubuntu-%s.tar.gz", version, arch)), gpuNeeds{}, "llama.cpp"},
}, h)
}
cuda := func(cv string) Build {
name := fmt.Sprintf("bin-ubuntu-cuda-%s-%s.tar.gz", cv, arch)
// The release carries the CUDA runtime beside the build, so the node
// needs a driver and nothing else — no toolkit.
b := base("cuda-"+cv, fmt.Sprintf("llama-%s-%s", version, name), fmt.Sprintf("cudart-llama-%s-%s", version, name))
b.GPUBackend = "libggml-cuda.so"
return b
}
vulkan := base("vulkan", fmt.Sprintf("llama-%s-bin-ubuntu-vulkan-%s.tar.gz", version, arch))
vulkan.GPUBackend = "libggml-vulkan.so"
vulkan.Packages = map[string][]string{"deb": {"libgomp1", "libvulkan1"}, "rpm": {"libgomp", "vulkan-loader"}}
cuda12Driver := 570
if slices.Contains(llamaCUDA12Real, h.MinComputeCapability) {
cuda12Driver = minDriverForCUDA12
}
cands := []candidate{{cuda("13.3"), gpuNeeds{minDriverForCUDA13, 7.5}, "llama.cpp built for CUDA 13.3"}}
if arch == "x64" {
cands = append(cands, candidate{cuda("12.8"), gpuNeeds{cuda12Driver, 5.0}, "llama.cpp built for CUDA 12.8"})
}
cands = append(cands, candidate{vulkan, gpuNeeds{}, "llama.cpp built for Vulkan"})
return pick("llama.cpp", variant, cands, h)
}
func orUnknown(s string) string {
+1
View File
@@ -143,6 +143,7 @@ func newCtlCmd() *cobra.Command {
newRetryCmd(client),
newUpgradeCmd(client),
newEngineCheckCmd(),
newEngineBuildCmd(client),
newToolCheckCmd(),
newWeightsCmd(client, &asJSON),
newCapacityCmd(client, &asJSON),
+39
View File
@@ -0,0 +1,39 @@
package cli
import (
"encoding/json"
"fmt"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
)
func newEngineBuildCmd(client func() *ctlClient) *cobra.Command {
return &cobra.Command{
Use: "engine-build <node> <engine> <build>",
Short: "Choose which build of an engine a GPU node installs",
Long: "A node installs the build of vLLM its hardware runs: CUDA 13 where the\n" +
"driver and GPUs allow, else the same version built for CUDA 12.9. This\n" +
"chooses instead, as the cluster page's vLLM build column does:\n\n" +
" auto what the hardware runs (the default)\n" +
" cu13 the CUDA 13 build, from PyPI\n" +
" cu129 the CUDA 12.9 build, from vLLM's GitHub release\n" +
" none no vLLM on this node\n\n" +
"The node is told at once and installs it in the background; a build it\n" +
"cannot compute with on its GPU fails to install, saying why, and the\n" +
"cluster page shows it. A V100 runs no build of vLLM 0.20 or later, and\n" +
"gets 0.19.1, the newest built for it, by itself.\n\n" +
" llmgw ctl engine-build gpu07 vllm cu129",
Args: cobra.ExactArgs(3),
RunE: func(cmd *cobra.Command, args []string) error {
body, _ := json.Marshal(adminapi.NodeEngineRequest{Engine: args[1], Build: args[2]})
if err := client().post(cmd.Context(), "/api/v1/nodes/"+args[0]+"/engines", body, nil); err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s build %s; the node installs it in the background (llmgw ctl nodes shows it)\n",
args[0], args[1], args[2])
return nil
},
}
}
+6
View File
@@ -495,6 +495,8 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
// And what each model was measured to serve, for scaling and for
// choosing between profiles.
ctl.Reconciler().SetMeasurements(control.Measurements(st, log))
// The engine builds chosen per node on the cluster page.
ctl.SetEngineChoices(control.NewEngineChoices(ctx, st, log))
log.Info("competing to be the control plane that places work",
"owner", lead.Owner())
}
@@ -504,6 +506,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
})
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
if ch := ctl.EngineChoices(); ch != nil {
svc.SetEngineChoices(ch.For)
ch.OnChange(svc.Reprovision)
}
// A standby must refuse agents as well as refusing to place. Commands
// travel down the stream the agent opened, so a standby holding one is a
// node the leader cannot command — and the leader's placements would fail
+57
View File
@@ -9,6 +9,7 @@ package adminapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
@@ -111,6 +112,7 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("GET /api/v1/explain-placement", a.explainPlacement)
mux.HandleFunc("POST /api/v1/nodes/{id}/state", a.setNodeState)
mux.HandleFunc("POST /api/v1/nodes/{id}/upgrade", a.upgradeNode)
mux.HandleFunc("POST /api/v1/nodes/{id}/engines", a.setNodeEngine)
mux.HandleFunc("DELETE /api/v1/nodes/{id}", a.removeNode)
mux.HandleFunc("POST /api/v1/fabric-test", a.fabricTest)
mux.HandleFunc("GET /api/v1/nodes/{id}/logs", a.nodeLogs)
@@ -156,6 +158,10 @@ type NodeView struct {
// Provision is where installing each wanted engine stands. An engine is in
// Engines only once its installation here is ready.
Provision []inventory.EngineInstall `json:"provision,omitempty"`
// EngineBuilds are the builds chosen for this node, by engine: "auto"
// (its hardware decides), a build ("cu129"), or "none". Only on a node
// with GPUs, the only kind with a choice to make.
EngineBuilds map[string]string `json:"engine_builds,omitempty"`
// OS is the operating system and architecture, as the agent read them.
OS string `json:"os,omitempty"`
// Warnings are facts about the machine that make it slow or unusable
@@ -473,6 +479,13 @@ func (a *API) Nodes() []NodeView {
}
slices.Sort(v.Engines)
v.Provision = n.Provision
if ch := a.ctl.EngineChoices(); ch != nil && len(n.GPUs) > 0 {
chosen := ch.For(n.ID)
v.EngineBuilds = map[string]string{"vllm": "auto"}
for e, b := range chosen {
v.EngineBuilds[e] = b
}
}
if h := n.Host; h.OSID != "" {
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
@@ -1532,3 +1545,47 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
}
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
}
// NodeEngineRequest chooses one engine's build for a node.
type NodeEngineRequest struct {
Engine string `json:"engine"`
// Build is "auto", a build such as "cu129", or "none".
Build string `json:"build"`
}
// setNodeEngine chooses an engine's build for a node, and has the node
// install it.
func (a *API) setNodeEngine(w http.ResponseWriter, r *http.Request) {
ch := a.ctl.EngineChoices()
if ch == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store to keep the choice in")
return
}
id := r.PathValue("id")
if _, ok := a.ctl.Inventory().Get(id); !ok {
writeError(w, http.StatusNotFound, "no node "+id)
return
}
var req NodeEngineRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<12)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
if err := ch.Set(r.Context(), id, req.Engine, req.Build, actorOf(r)); err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, req)
}
// SetNodeEngine chooses an engine's build for a node, for the cluster page.
func (a *API) SetNodeEngine(ctx context.Context, node, engine, build, by string) error {
ch := a.ctl.EngineChoices()
if ch == nil {
return errors.New("this control plane has no store to keep the choice in")
}
if _, ok := a.ctl.Inventory().Get(node); !ok {
return errors.New("no node " + node)
}
return ch.Set(ctx, node, engine, build, by)
}
@@ -0,0 +1,63 @@
package adminapi
import (
"context"
"log/slog"
"net/http"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
)
// choiceStore keeps engine choices in memory.
type choiceStore struct{ m map[string]map[string]string }
func (c *choiceStore) NodeEngines(context.Context) (map[string]map[string]string, error) {
return c.m, nil
}
func (c *choiceStore) SetNodeEngine(_ context.Context, node, engine, variant, _ string) error {
if c.m[node] == nil {
c.m[node] = map[string]string{}
}
c.m[node][engine] = variant
return nil
}
func TestAnEngineBuildIsChosenForANodeAndTheNodeIsTold(t *testing.T) {
a, ctl := setup(t, testModels(), gpuNode("gpu07", 8), inventory.Node{ID: "cpu01", State: inventory.Ready})
ch := control.NewEngineChoices(t.Context(), &choiceStore{m: map[string]map[string]string{}}, slog.New(slog.DiscardHandler))
var told []string
ch.OnChange(func(node string) error { told = append(told, node); return nil })
ctl.SetEngineChoices(ch)
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"cu129"}`, nil); code != http.StatusOK {
t.Fatalf("status %d", code)
}
if len(told) != 1 || told[0] != "gpu07" {
t.Errorf("the node was not told: %v", told)
}
var nodes []NodeView
call(t, a, http.MethodGet, "/api/v1/nodes", "", &nodes)
for _, n := range nodes {
switch n.ID {
case "gpu07":
if n.EngineBuilds["vllm"] != "cu129" {
t.Errorf("gpu07 shows %v", n.EngineBuilds)
}
case "cpu01":
if n.EngineBuilds != nil {
t.Errorf("a node with no GPU has no build to choose: %v", n.EngineBuilds)
}
}
}
for _, bad := range []string{`{"engine":"vllm","build":"cu200"}`, `{"engine":"llamacpp","build":"cu129"}`} {
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", bad, nil); code != http.StatusBadRequest {
t.Errorf("%s: status %d", bad, code)
}
}
if code := call(t, a, http.MethodPost, "/api/v1/nodes/gpu07/engines", `{"engine":"vllm","build":"auto"}`, nil); code != http.StatusOK ||
ch.For("gpu07")["vllm"] != "" {
t.Errorf("auto should leave it to the node again: %d %v", code, ch.For("gpu07"))
}
}
+29 -3
View File
@@ -43,14 +43,40 @@ func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
}
return out
}
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" {
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, installable(reg))); got["vllm"] != "0.25.1" {
t.Errorf("the Volta node: %v", got)
}
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" {
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, installable(reg))); got["vllm"] != "0.30.0" {
t.Errorf("the Hopper node: %v", got)
}
// A node that disabled vLLM does not offer it, and is not asked for it.
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" {
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, []string{"llamacpp"})); got["vllm"] != "" {
t.Errorf("asked a node for an engine it does not install: %v", got)
}
}
func TestABuildChosenForANodeIsWhatItIsTold(t *testing.T) {
s := &Service{}
s.SetEngines(map[string]string{"llamacpp": "b1", "vllm": "0.30.0"})
s.SetEngineChoices(func(node string) map[string]string {
return map[string]map[string]string{"gpu07": {"vllm": "cu129"}, "gpu08": {"vllm": "none"}}[node]
})
can := []string{"llamacpp", "vllm"}
variant := func(node string) (string, bool) {
for _, e := range s.provisionFor(inventory.Node{ID: node}, can).GetEngines() {
if e.GetEngine() == "vllm" {
return e.GetVariant(), true
}
}
return "", false
}
if v, ok := variant("gpu07"); !ok || v != "cu129" {
t.Errorf("gpu07: %q, %v", v, ok)
}
if _, ok := variant("gpu08"); ok {
t.Error("gpu08 was asked for a vLLM it was chosen to run none of")
}
if v, ok := variant("gpu09"); !ok || v != "" {
t.Errorf("gpu09, left to its hardware: %q, %v", v, ok)
}
}
+42 -5
View File
@@ -56,6 +56,10 @@ type Service struct {
// enginesFor is what a node is told to install, set once at start: the
// cluster's engines, with any override that picks the node laid over.
enginesFor func(node inventory.Node) map[string]string
// choices are the engine builds chosen per node, and installableBy what
// each connected node said it can install, for telling it again.
choices func(nodeID string) map[string]string
installableBy map[string][]string
// logWaits are the log requests awaiting a reply, by request id.
logMu sync.Mutex
@@ -78,6 +82,29 @@ func (s *Service) SetEngines(e map[string]string) {
// of two GPU generations needs two vLLM builds, or vLLM on only one.
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
// SetEngineChoices sets where the builds chosen for each node come from: by
// engine, a build ("cu129") or "none". An engine not named is left to the
// node's hardware.
func (s *Service) SetEngineChoices(f func(nodeID string) map[string]string) { s.choices = f }
// Reprovision tells a connected node again what to install, after a choice
// for it changed; one not connected is told when it next connects.
func (s *Service) Reprovision(nodeID string) error {
if s.enginesFor == nil {
return nil
}
s.mu.RLock()
can, ok := s.installableBy[nodeID]
s.mu.RUnlock()
node, known := s.inv.Get(nodeID)
if !ok || !known {
return nil
}
return s.send(nodeID, &llmgwv1.ControlMessage{
Msg: &llmgwv1.ControlMessage_Provision{Provision: s.provisionFor(node, can)},
})
}
// conn is one agent's open stream.
type conn struct {
nodeID string
@@ -188,7 +215,14 @@ func (s *Service) Connect(
// node rebuilt from nothing is told the same as one that has been here a
// month, and a changed version reaches every node on its next connect.
if s.enginesFor != nil {
p := s.provisionFor(node, reg)
can := installable(reg)
s.mu.Lock()
if s.installableBy == nil {
s.installableBy = map[string][]string{}
}
s.installableBy[nodeID] = can
s.mu.Unlock()
p := s.provisionFor(node, can)
if err := stream.Send(&llmgwv1.ControlMessage{
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
}); err != nil {
@@ -856,15 +890,18 @@ func rate(before, now uint64, secs float64) float64 {
// provisionFor is what a node is told to install: the engines decided for
// it, those it can install.
func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision {
func (s *Service) provisionFor(node inventory.Node, can []string) *llmgwv1.Provision {
p := &llmgwv1.Provision{}
can := installable(reg)
wants := s.enginesFor(node)
var chosen map[string]string
if s.choices != nil {
chosen = s.choices(node.ID)
}
for _, e := range slices.Sorted(maps.Keys(wants)) {
if !slices.Contains(can, e) {
if !slices.Contains(can, e) || chosen[e] == "none" {
continue
}
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]})
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e], Variant: chosen[e]})
}
return p
}
+25 -5
View File
@@ -305,6 +305,16 @@ func (m Model) normalize() Model {
if p.Nodes <= 0 {
p.Nodes = 1
}
// tp on a llama.cpp GPU profile, written before gpus_per_node meant
// anything for it, is read as the width it was meant as: N cards of
// one node. Refusing it would have dropped a catalog's model on the
// upgrade that made the width work. Only both given, disagreeing, is
// refused below, as ambiguous.
if p.Engine == "llamacpp" && p.GPUClass != "" && (p.TP > 1 || p.PP > 1) && p.Nodes == 1 {
if n := p.TP * max(p.PP, 1); p.GPUsPerNode == 0 || p.GPUsPerNode == n {
p.GPUsPerNode, p.TP, p.PP = n, 1, 0
}
}
switch {
case p.CPUOnly():
// Nothing to divide up: this profile asks for no cards at all.
@@ -323,6 +333,13 @@ func (m Model) normalize() Model {
p.Weight = 1
}
}
// A Hugging Face repository page for a model vLLM or SGLang runs is the
// repository it names: the engine downloads a repository id itself,
// where the page was fetched as one file of HTML and failed at start.
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
m.Source = repo
}
return m
}
@@ -335,9 +352,11 @@ func (m Model) validate() error {
}
if repo, ok := repositoryPage(m.Source); ok {
// A page, not weights: a node fetches the HTML as one file and the
// engine fails at start, on a path that looks right.
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not its files; "+
"mirror it and use the version's source ending in /, or give %q for vLLM to fetch itself",
// engine fails at start, on a path that looks right. For vLLM it was
// read as the repository id when the model was made; only a model
// llama.cpp runs is left here, which needs a file.
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
m.Name, m.Source, repo)
}
if err := validateAdapters(m); err != nil {
@@ -398,8 +417,9 @@ func (m Model) validate() error {
return fmt.Errorf("model %q profile %q: llama.cpp does not span nodes; use vllm or sglang for a %d-node replica",
m.Name, p.Name, p.Nodes)
case p.TP > 1 || p.PP > 1:
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, not by tp or pp: "+
"write gpus_per_node: %d and leave tp and pp out", m.Name, p.Name, max(p.TP, 1)*max(p.PP, 1))
return fmt.Errorf("model %q profile %q: llama.cpp splits a model across a node's GPUs by layer, and "+
"tp %d does not agree with gpus_per_node %d: give gpus_per_node alone", m.Name, p.Name,
max(p.TP, 1)*max(p.PP, 1), p.GPUsPerNode)
}
continue
}
+34 -15
View File
@@ -332,37 +332,56 @@ func TestALlamaCppReplicaOnSeveralGPUsSaysHowManyAsGPUsPerNode(t *testing.T) {
if pl := m.PlacementProfiles(); len(pl) != 1 || pl[0].GPUsPerNode != 4 {
t.Errorf("placement was not asked for four GPUs: %+v", pl)
}
// tp is a width llama.cpp has no way to use; refused, saying what to
// write instead.
// tp, written before gpus_per_node meant anything for llama.cpp, is read
// as the same width rather than refused: refusing dropped the model on
// the upgrade that made the width work.
old := model("old", Profile{Name: "t4", Engine: "llamacpp", GPUClass: "t4", TP: 4})
c, err = New([]Model{old})
if err != nil {
t.Fatalf("tp 4 on llama.cpp: %v", err)
}
got, _ := c.Resolve("old")
if p := got.PlacementProfiles()[0]; p.GPUsPerNode != 4 {
t.Errorf("tp 4 should place on four GPUs: %+v", p)
}
// Both given and disagreeing is ambiguous, and refused; so is more than
// one node.
for _, p := range []Profile{
{Name: "tp", Engine: "llamacpp", GPUClass: "t4", TP: 4},
{Name: "both", Engine: "llamacpp", GPUClass: "t4", TP: 4, GPUsPerNode: 2},
{Name: "nodes", Engine: "llamacpp", GPUClass: "t4", Nodes: 2, GPUsPerNode: 1},
} {
_, err := New([]Model{model("x", p)})
if err == nil {
if _, err := New([]Model{model("x", p)}); err == nil {
t.Errorf("%s: accepted", p.Name)
}
if p.TP == 4 && (err == nil || !strings.Contains(err.Error(), "gpus_per_node: 4")) {
t.Errorf("tp 4 should point at gpus_per_node: %v", err)
}
}
}
func TestARepositoryPageIsNotASource(t *testing.T) {
func TestARepositoryPageIsReadAsTheRepositoryForVLLM(t *testing.T) {
// The field: an AWQ model's page given as a vLLM source, fetched as one
// file of HTML. vLLM downloads a repository id itself.
for _, src := range []string{
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ",
"https://huggingface.co/Qwen/Qwen2.5-32B-Instruct-AWQ/tree/main",
} {
m := model("awq")
m.Source = src
_, err := New([]Model{m})
if err == nil || !strings.Contains(err.Error(), `"Qwen/Qwen2.5-32B-Instruct-AWQ"`) {
t.Errorf("%s: %v", src, err)
c, err := New([]Model{m})
if err != nil {
t.Fatalf("%s: %v", src, err)
}
if got, _ := c.Resolve("awq"); got.Source != "Qwen/Qwen2.5-32B-Instruct-AWQ" {
t.Errorf("%s became %q", src, got.Source)
}
}
m := model("file")
m.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
if _, err := New([]Model{m}); err != nil {
// llama.cpp needs a file, which a page is not.
g := model("gguf", Profile{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 4})
g.Source = "https://huggingface.co/Qwen/X-GGUF"
if _, err := New([]Model{g}); err == nil || !strings.Contains(err.Error(), "needs a .gguf") {
t.Errorf("a page for llama.cpp: %v", err)
}
f := model("file")
f.Source = "https://huggingface.co/Qwen/X-GGUF/resolve/abc123/x.gguf"
if _, err := New([]Model{f}); err != nil {
t.Errorf("a file under /resolve/ is a source: %v", err)
}
}
+18 -9
View File
@@ -75,15 +75,18 @@ func DefaultConfig() Config {
// Controller owns the control plane's state.
type Controller struct {
cfg Config
log *slog.Logger
inv *inventory.Store
cat *catalog.Live
rec *reconciler.Reconciler
lead *Leadership
disp reconciler.Dispatcher
sink EndpointSink
usage UsageSource
// engineChoices are the engine builds chosen per node; nil without a
// store to keep them in.
engineChoices *EngineChoices
cfg Config
log *slog.Logger
inv *inventory.Store
cat *catalog.Live
rec *reconciler.Reconciler
lead *Leadership
disp reconciler.Dispatcher
sink EndpointSink
usage UsageSource
mu sync.Mutex
last string // fingerprint of the last published list
@@ -408,3 +411,9 @@ func EngineKeys(secret []byte) func(replicaID string) string {
// QuotaUse is every capacity quota beside what its scope holds.
func (c *Controller) QuotaUse() []reconciler.QuotaUse { return c.rec.QuotaUse() }
// SetEngineChoices gives the controller the engine builds chosen per node.
func (c *Controller) SetEngineChoices(e *EngineChoices) { c.engineChoices = e }
// EngineChoices are the engine builds chosen per node, or nil.
func (c *Controller) EngineChoices() *EngineChoices { return c.engineChoices }
+108
View File
@@ -0,0 +1,108 @@
package control
import (
"context"
"fmt"
"log/slog"
"maps"
"slices"
"strings"
"sync"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
)
// EngineChoiceStore is where the engine builds chosen per node are kept.
type EngineChoiceStore interface {
NodeEngines(ctx context.Context) (map[string]map[string]string, error)
SetNodeEngine(ctx context.Context, node, engine, variant, by string) error
}
// EngineChoices are the engine builds chosen for nodes, rather than left to
// their hardware: vLLM's CUDA 12.9 build on a node its CUDA 13 build would
// also install on, or no vLLM at all. Kept in the store, held here for the
// agent service to ask on every registration, and pushed to a node the
// moment its choice changes.
type EngineChoices struct {
st EngineChoiceStore
log *slog.Logger
mu sync.RWMutex
byNode map[string]map[string]string
notify func(node string) error
}
// NewEngineChoices reads what was chosen before.
func NewEngineChoices(ctx context.Context, st EngineChoiceStore, log *slog.Logger) *EngineChoices {
c := &EngineChoices{st: st, log: log, byNode: map[string]map[string]string{}}
if got, err := st.NodeEngines(ctx); err != nil {
log.Error("could not read the engine builds chosen per node; every node installs what its hardware runs", "err", err)
} else {
c.byNode = got
}
return c
}
// OnChange is called with a node whose choice changed, to tell it.
func (c *EngineChoices) OnChange(f func(node string) error) {
c.mu.Lock()
c.notify = f
c.mu.Unlock()
}
// For is what was chosen for one node, by engine: a build, or "none".
func (c *EngineChoices) For(node string) map[string]string {
c.mu.RLock()
defer c.mu.RUnlock()
return maps.Clone(c.byNode[node])
}
// EngineBuilds are what can be chosen for each engine that has a choice:
// "auto" is the node's hardware deciding, "none" no engine at all.
func EngineBuilds(engine string) []string {
switch engine {
case "vllm":
return append(append([]string{"auto"}, provision.VLLMVariants...), "none")
case "sglang":
return []string{"auto", "none"}
}
return nil
}
// Set chooses an engine's build for a node, keeps it, and tells the node.
func (c *EngineChoices) Set(ctx context.Context, node, engine, build, by string) error {
builds := EngineBuilds(engine)
if builds == nil {
return fmt.Errorf("%s has no builds to choose between", engine)
}
if !slices.Contains(builds, build) {
return fmt.Errorf("%s has no build %q; choose %s", engine, build, strings.Join(builds, ", "))
}
variant := build
if build == "auto" {
variant = ""
}
if err := c.st.SetNodeEngine(ctx, node, engine, variant, by); err != nil {
return err
}
c.mu.Lock()
if variant == "" {
delete(c.byNode[node], engine)
} else {
if c.byNode[node] == nil {
c.byNode[node] = map[string]string{}
}
c.byNode[node][engine] = variant
}
notify := c.notify
c.mu.Unlock()
c.log.Info("engine build chosen for a node", "node", node, "engine", engine, "build", build, "by", by)
if notify != nil {
if err := notify(node); err != nil {
// Kept and applied on its next connection; said, since a node that
// is not connected is the usual reason.
c.log.Warn("the node was not told yet; it is when it next connects", "node", node, "err", err)
}
}
return nil
}
+17 -5
View File
@@ -581,7 +581,16 @@ func Evaluate(f Facts) []Finding {
// the same decision the agent makes, so the report and the node agree.
func vllmFindings(f Facts, add func(Level, string, string, string)) {
v := f.VLLMVersion
b, err := provision.Recipes["vllm"].Choose(v, f.Host)
builds, err := provision.Recipes["vllm"].Candidates(v, "", f.Host)
var b provision.Build
if err == nil {
// The first the agent would try; it moves to the next by itself if
// that one does not work here.
b = builds[0]
if b.Version != "" {
v = b.Version
}
}
var nh provision.NotHere
switch {
case errors.As(err, &nh):
@@ -592,8 +601,7 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
// llama.cpp. Refusing it here kept older GPU generations out of a
// cluster for an engine nobody meant to run on them.
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
"add it with --disable-engine vllm if it is meant for llama.cpp alone")
return
case f.VLLMInstalled:
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
@@ -605,8 +613,12 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
"free space there, or point engines_dir in agent.yaml at a larger disk")
return
}
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s",
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir), "")
note := ""
if b.Note != "" {
note = "; " + b.Note
}
add(OK, "engines", fmt.Sprintf("vLLM %s (%s) can be installed here: driver %s, %.1f GB free under %s%s",
v, b.Variant, f.Host.NVIDIADriver, f.EnginesFreeGB, f.EnginesDir, note), "")
}
func rank(l Level) int {
+12 -5
View File
@@ -153,11 +153,18 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
// What vLLM cannot do keeps vLLM off the node, not the node out of
// the cluster: it still serves llama.cpp.
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"},
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"},
Warn, "before 0.26 is built for CUDA 12"},
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"a driver too old for either build", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "470.256.02"}, VLLMVersion: "0.30.0"}, Warn, "driver 525"},
// A driver too old for CUDA 13 runs the same version's CUDA 12.9 build.
{"turing, old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.5},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu129) can be installed"},
// A V100 has no build of vLLM 0.20 or later; it gets the newest
// that has Volta kernels, and doctor says which.
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "vLLM 0.19.1 (cu12) can be installed here"},
{"pascal", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 6.1},
VLLMVersion: "0.30.0"}, Warn, "no vLLM is built for GPUs of compute capability 6.1"},
{"volta, the last vLLM built for it", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
VLLMVersion: "0.19.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
+5
View File
@@ -196,6 +196,11 @@ func (r *Router) Handler() http.Handler {
if c, ok := r.clusterView.(web.CapacityAdmin); ok {
pages.SetCapacity(c)
}
// The engine build each GPU node installs, chosen on the cluster
// page.
if e, ok := r.clusterView.(web.NodeEngineAdmin); ok {
pages.RegisterNodeEngines(mux, e)
}
pages.RegisterAdminQuotas(mux, r.store, r.acct)
// And the report of what that tree actually used.
pages.RegisterAdminUsage(mux, r.store, r.acct)
+1
View File
@@ -45,6 +45,7 @@ var copyOrder = []string{
// derived from this, and a new secret would lock the router out of all
// of them at the moment of the move.
"cluster_secrets",
"node_engines",
}
// notCopied are tables deliberately left behind.
@@ -0,0 +1,11 @@
-- The build of an engine chosen for a node, where it is not left to the
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
-- or no vLLM at all. A node with no row here installs what its hardware runs.
CREATE TABLE node_engines (
node_id TEXT NOT NULL,
engine TEXT NOT NULL,
variant TEXT NOT NULL,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (node_id, engine)
);
@@ -0,0 +1,11 @@
-- The build of an engine chosen for a node, where it is not left to the
-- node's hardware: vLLM's CUDA 12.9 build on a node that could run CUDA 13's,
-- or no vLLM at all. A node with no row here installs what its hardware runs.
CREATE TABLE node_engines (
node_id TEXT NOT NULL,
engine TEXT NOT NULL,
variant TEXT NOT NULL,
updated_at BIGINT NOT NULL,
updated_by TEXT NOT NULL,
PRIMARY KEY (node_id, engine)
);
+44
View File
@@ -0,0 +1,44 @@
package store
import (
"context"
"time"
)
// NodeEngines are the engine builds chosen per node: node id, then engine,
// then the build ("cu129") or "none". A node or engine not here is left to
// the node's hardware.
func (s *DB) NodeEngines(ctx context.Context) (map[string]map[string]string, error) {
rows, err := s.db.QueryContext(ctx, `SELECT node_id, engine, variant FROM node_engines`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]map[string]string{}
for rows.Next() {
var node, engine, variant string
if err := rows.Scan(&node, &engine, &variant); err != nil {
return nil, err
}
if out[node] == nil {
out[node] = map[string]string{}
}
out[node][engine] = variant
}
return out, rows.Err()
}
// SetNodeEngine chooses an engine's build for a node; an empty variant
// leaves it to the node's hardware again.
func (s *DB) SetNodeEngine(ctx context.Context, node, engine, variant, by string) error {
if variant == "" {
_, err := s.db.ExecContext(ctx, `DELETE FROM node_engines WHERE node_id = ? AND engine = ?`, node, engine)
return err
}
_, err := s.db.ExecContext(ctx,
`INSERT INTO node_engines (node_id, engine, variant, updated_at, updated_by) VALUES (?, ?, ?, ?, ?)
ON CONFLICT (node_id, engine) DO UPDATE SET variant = excluded.variant,
updated_at = excluded.updated_at, updated_by = excluded.updated_by`,
node, engine, variant, time.Now().UnixMilli(), by)
return err
}
+24
View File
@@ -0,0 +1,24 @@
package store
import "testing"
func TestAnEngineBuildIsChosenPerNodeAndCanBeLeftToItAgain(t *testing.T) {
db := capacityStore(t)
ctx := t.Context()
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "cu129", "ilker"); err != nil {
t.Fatal(err)
}
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "none", "ilker"); err != nil {
t.Fatal(err)
}
got, err := db.NodeEngines(ctx)
if err != nil || got["gpu07"]["vllm"] != "none" {
t.Fatalf("got %v, %v; the second choice replaces the first", got, err)
}
if err := db.SetNodeEngine(ctx, "gpu07", "vllm", "", "ilker"); err != nil {
t.Fatal(err)
}
if got, _ := db.NodeEngines(ctx); len(got["gpu07"]) != 0 {
t.Errorf("left to the node again, still %v", got)
}
}
+49
View File
@@ -0,0 +1,49 @@
package web
import (
"context"
"net/http"
"net/url"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// NodeEngineAdmin chooses an engine's build for a node: "auto", a build
// such as "cu129", or "none".
type NodeEngineAdmin interface {
SetNodeEngine(ctx context.Context, node, engine, build, by string) error
}
// RegisterNodeEngines mounts the choice of engine builds on the cluster
// page: for an administrator, with the session's CSRF token, and answered by
// going back to the page with what happened.
func (p *AccountPages) RegisterNodeEngines(mux *http.ServeMux, a NodeEngineAdmin) {
mux.HandleFunc("POST /admin/nodes/{id}/engines", p.adminOnly(func(w http.ResponseWriter, r *http.Request, admin store.User) {
sess, _, _ := p.session(r)
if !p.accounts.CheckCSRF(sess, r.FormValue("csrf")) {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
node, engine, build := r.PathValue("id"), r.FormValue("engine"), r.FormValue("build")
q := url.Values{}
if err := a.SetNodeEngine(r.Context(), node, engine, build, admin.Email); err != nil {
p.log.Warn("an engine build was not chosen", "admin", admin.Email, "node", node, "err", err)
q.Set("problem", node+": "+err.Error())
} else {
p.log.Info("an engine build was chosen", "admin", admin.Email, "node", node, "engine", engine, "build", build)
q.Set("notice", node+" installs "+engine+" "+buildWords(build)+
"; the node is told now, and installs it in the background")
}
http.Redirect(w, r, "/admin?"+q.Encode(), http.StatusSeeOther)
}))
}
func buildWords(b string) string {
switch b {
case "auto":
return "in the build its hardware runs"
case "none":
return "no longer"
}
return "built for " + map[string]string{"cu13": "CUDA 13", "cu129": "CUDA 12.9"}[b]
}
+75
View File
@@ -0,0 +1,75 @@
package web
import (
"context"
"io"
"log/slog"
"net/http"
"net/url"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
// buildFleet is one GPU node whose vLLM build was chosen, and one CPU node.
type buildFleet struct{ fakeFleet }
func (buildFleet) Nodes() []adminapi.NodeView {
return []adminapi.NodeView{
{ID: "gpu07", State: "ready", GPUTotal: 8, GPUFree: 8,
EngineBuilds: map[string]string{"vllm": "cu129"},
Provision: []inventory.EngineInstall{{Engine: "vllm", Version: "0.30.0", Variant: "cu129", State: "installing"}}},
{ID: "cpu01", State: "ready"},
}
}
type recordedChoice struct{ node, engine, build, by string }
type fakeEngineAdmin struct{ got []recordedChoice }
func (f *fakeEngineAdmin) SetNodeEngine(_ context.Context, node, engine, build, by string) error {
f.got = append(f.got, recordedChoice{node, engine, build, by})
return nil
}
func TestTheVLLMBuildIsChosenOnTheClusterPage(t *testing.T) {
a := fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com", IsAdmin: true}}
p := NewAccountPages(a, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
mux := http.NewServeMux()
mux.HandleFunc("GET /admin", p.AdminPage(Dashboard(buildFleet{}, nil, "v1", time.Second, false, NavExtras{}, nil)))
fe := &fakeEngineAdmin{}
p.RegisterNodeEngines(mux, fe)
body := mirrorGet(t, mux, "/admin").Body.String()
if !strings.Contains(body, `action="/admin/nodes/gpu07/engines"`) ||
!strings.Contains(body, `<option value="cu129" selected>CUDA 12.9</option>`) {
t.Fatalf("no choice of build for the GPU node, with its current one selected: %s", body)
}
if strings.Contains(body, `action="/admin/nodes/cpu01/engines"`) {
t.Error("a node with no GPU was offered a vLLM build")
}
if !strings.Contains(body, "0.30.0 CUDA 12.9 installing") {
t.Error("what is installed is not shown beside the choice")
}
// The read-only page on the admin listener has no session, so no form.
_, ro := get(t, Dashboard(buildFleet{}, nil, "v1", time.Second, true, NavExtras{}, nil), "http://127.0.0.1:9090/", nil)
if strings.Contains(ro, "/engines\"") {
t.Error("the admin listener's page offers a change it cannot sign")
}
w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"tok"}, "engine": {"vllm"}, "build": {"none"}})
if w.Code != http.StatusSeeOther || len(fe.got) != 1 || fe.got[0] != (recordedChoice{"gpu07", "vllm", "none", "a@example.com"}) {
t.Fatalf("status %d, chose %+v", w.Code, fe.got)
}
if loc := w.Header().Get("Location"); !strings.Contains(loc, "notice=") {
t.Errorf("redirected to %s, saying nothing", loc)
}
if w := mirrorPost(t, mux, "/admin/nodes/gpu07/engines", url.Values{"csrf": {"bad"}, "engine": {"vllm"}, "build": {"none"}}); w.Code != http.StatusBadRequest || len(fe.got) != 1 {
t.Errorf("a forged form was accepted: %d", w.Code)
}
}
+13 -7
View File
@@ -19,13 +19,16 @@ type DashboardData struct {
// of a GPU table that would only say there are none.
CPU cpuCapacity
// Menu is the shared top menu; Nav below is what the cluster page adds.
Menu Nav
Nodes []adminapi.NodeView
Models []adminapi.ModelView
Replicas []adminapi.ReplicaView
Unplaced []adminapi.ExplainView
Usage []store.KeyUsage
Summary adminapi.Summary
Menu Nav
// Notice and Problem are what the last change made from this page did,
// carried over the redirect back to it.
Notice, Problem string
Nodes []adminapi.NodeView
Models []adminapi.ModelView
Replicas []adminapi.ReplicaView
Unplaced []adminapi.ExplainView
Usage []store.KeyUsage
Summary adminapi.Summary
// Fabrics is the physical fabrics and what disagrees about them, from a
// source that can say.
Fabrics *fabrics.Report
@@ -170,6 +173,9 @@ func Dashboard(src Source, usage UsageSource, version string, refresh time.Durat
Now: time.Now().UTC().Format(time.RFC3339),
AuthEnabled: usage != nil,
}
if d.Menu.Admin {
d.Notice, d.Problem = r.URL.Query().Get("notice"), r.URL.Query().Get("problem")
}
if src != nil {
d.Nodes = src.Nodes()
d.CPU = cpuCapacityOf(d.Nodes)
+26 -1
View File
@@ -50,8 +50,9 @@
{{else}}<p class="empty">No nodes have registered.</p>{{end}}
<h2>Nodes</h2>
{{with .Notice}}<p class="notice">{{.}}</p>{{end}}{{with .Problem}}<p class="error">{{.}}</p>{{end}}
{{if .Nodes}}
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
<table class="nodes rows"><tr><th>Node</th><th>State</th><th>CPU</th><th>RAM</th>{{if .AnyGPUs}}<th class="num">GPUs</th>{{end}}<th class="num">Disk</th><th>Engines</th>{{if .AnyGPUs}}<th>vLLM build</th>{{end}}{{if .AnyClasses}}<th>Classes</th>{{end}}{{if .AnyFabrics}}<th>Fabrics</th>{{end}}<th class="num">Models</th><th class="num">Replicas</th><th>Seen</th></tr>
{{range .Nodes}}<tr>
<td><a href="{{if $.OnAdminListener}}/nodes/{{.ID}}{{else}}/admin/nodes/{{.ID}}{{end}}" title="{{.ID}}"><code>{{shortNode .ID $.NodePrefix}}</code></a>{{if .MgmtAddr}} <span class="note">{{.MgmtAddr}}</span>{{end}}
<a class="note" href="{{if $.OnAdminListener}}/nodes/{{.ID}}/logs{{else}}/admin/nodes/{{.ID}}/logs{{end}}">logs</a></td>
@@ -66,6 +67,23 @@
was never asked about its disk must not read as a full one. */}}
<td class="num">{{if .DiskTotalGB}}{{.DiskFreeGB}}G/{{.DiskTotalGB}}G{{else}}—{{end}}</td>
<td>{{if .Engines}}{{join .Engines}}{{else}}<span class="note">none yet</span>{{end}}</td>
{{/* Which vLLM build the node installs: left to its hardware, or chosen.
What is installed beside it, so a choice and its outcome are read
together. Only a GPU node has one; changing it is for an
administrator, on the page reached through a session. */}}
{{if $.AnyGPUs}}<td class="clip">{{if .GPUTotal}}{{$node := .}}
{{range .Provision}}{{if eq .Engine "vllm"}}<span class="note" title="{{.Detail}}">{{.Version}} {{with .Variant}}{{buildLabel .}} {{end}}{{.State}}</span>{{end}}{{end}}
{{if and $.Menu.CSRF .EngineBuilds}}
<form method="post" action="/admin/nodes/{{.ID}}/engines" class="inline build-choice">
<input type="hidden" name="csrf" value="{{$.Menu.CSRF}}">
<input type="hidden" name="engine" value="vllm">
<select name="build" title="the vLLM build this node installs">
{{range engineBuilds "vllm"}}<option value="{{.}}"{{if eq . (index $node.EngineBuilds "vllm")}} selected{{end}}>{{buildLabel .}}</option>{{end}}
</select>
<noscript><button type="submit">Save</button></noscript>
</form>
{{end}}
{{else}}—{{end}}</td>{{end}}
{{if $.AnyClasses}}<td>{{join .Classes}}</td>{{end}}
{{if $.AnyFabrics}}<td>{{join .Fabrics}}</td>{{end}}
<td class="num">{{if .Models}}{{len .Models}}{{else}}—{{end}}</td>
@@ -75,6 +93,13 @@
{{range .Provision}}{{if and (ne .State "ready") (ne .State "skipped")}}<tr><td colspan="12" class="note{{if or (eq .State "failed") (eq .State "unsupported")}} bad{{end}}">{{.Engine}} {{.Version}}{{if .Variant}} ({{.Variant}}){{end}} {{.State}}: {{.Detail}}</td></tr>{{end}}{{end}}
{{range .Warnings}}<tr><td colspan="12" class="note bad">{{.}}</td></tr>{{end}}
{{end}}</table>
<script>
// A build chosen is saved as it is chosen, with the page reloaded on the
// answer: one select, no separate button to forget.
document.querySelectorAll("form.build-choice select").forEach(function (s) {
s.addEventListener("change", function () { s.form.submit(); });
});
</script>
{{else}}<p class="empty">No agents have registered.</p>{{end}}
<h2>Models</h2>
+20 -1
View File
@@ -26,6 +26,7 @@ import (
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -118,7 +119,25 @@ var funcs = template.FuncMap{
},
// shortNode is a node's name without what every node's name starts with.
"shortNode": strings.TrimPrefix,
"age": age,
// engineBuilds are the builds an engine can be given on a node, and
// buildLabel says one in words.
"engineBuilds": control.EngineBuilds,
"buildLabel": func(b string) string {
switch b {
case "auto":
return "auto: what the hardware runs"
case "cu13":
return "CUDA 13"
case "cu129":
return "CUDA 12.9"
case "cu12":
return "CUDA 12"
case "none":
return "none"
}
return b
},
"age": age,
// pctbar is a percentage as the minibar template takes it: whole, and
// held to 0..100 so a measurement a little over does not draw past its box.
"pctbar": func(v float64) struct{ P int } {