diff --git a/CHANGELOG.md b/CHANGELOG.md index 25877815..dd28a976 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -47,6 +47,30 @@ as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a node with two kinds of GPU, a shard could get a different card from the one it was given. +### Engines per node, and nodes that do not run them all + +A cluster with older GPUs could not add them cleanly: `node add` refused a +V100 node for the vLLM it could not run, though it was meant for llama.cpp, +and `--ignore-checks` passed over every other problem with it. + +- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the + node joins without it and serves what it does run. `node add` checks the + vLLM the control plane installs, not the binary's own default. +- **`node add --disable-engine vllm`** keeps an engine off a node for good: + written as `engines.disabled` in its `agent.yaml`, never installed or + advertised there, and not checked by doctor. +- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id + or labels a different version of an engine, or none: a CUDA 12 vLLM on + Volta beside a CUDA 13 one on newer cards. +- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang + before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta; + the threshold had been CUDA 13's for every version, so the doctor's advice + to set an older version could not work. A refusal now says which versions + would. +- **An engine binary named in `agent.yaml` is used.** The agent had installed + its own beside it, which was then used first. +- The node page shows each GPU's compute capability. + ### Serving from the mirror says where a model will run The serve form put every GGUF model on the CPU: it offered no choice, and its diff --git a/docs/install.md b/docs/install.md index 1ef91d6f..be66d9c9 100644 --- a/docs/install.md +++ b/docs/install.md @@ -376,8 +376,10 @@ node gets the same dependencies whenever it is built, and installed with `--require-hashes` against the sha256 PyPI publishes. The resolved list stays in the environment as `llmgw-requirements.txt`. It needs: -- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5): - vLLM is built for CUDA 13. A V100 cannot run it; +- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5) + for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run + them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and + Volta (7.0); - **about 25 GB free** under the engines directory while it installs; - **gcc**, which Triton builds its kernel launcher with. The agent installs it. @@ -394,8 +396,38 @@ engines: ``` A changed version reaches every node when it next connects. To use an engine -you installed yourself instead, set its version to `""` and name its path in -the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`). +you installed yourself on a node, name its path in that node's `agent.yaml` +(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent +then installs none of its own beside it. + +#### Different GPUs, different engines + +A cluster of two GPU generations may need two builds of an engine, or one on +only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by +id, or by labels, every one given having to match — and lays engines over the +cluster's for them, in order: + +```yaml +engines: + vllm: "0.30.0" # CUDA 13: Turing and newer +engine_overrides: + - gpu_class: v100-32g # Volta, which CUDA 13 dropped + engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM + - labels: { role: gguf } # nodes meant for llama.cpp alone + engines: { vllm: "" } # none +``` + +A node told nothing it can run of an engine joins without it: `llmgw doctor` +and `node add` warn, and the node serves what it does run. To keep an engine +off a node for good, whatever the control plane says, add the node with +`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]` +into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower +than `--ignore-checks`, which passes over every other problem as well. + +A llama.cpp CUDA build is checked for the GPU it is meant for: installing it +asks it which devices it sees, and a replica placed on GPUs is not ready until +its process holds memory on each of them. A build missing a CUDA library fails +there, naming the library, rather than serving on the CPU. After changing a version, check it still takes the flags llmgw builds for it, which drift between releases: @@ -1216,9 +1248,9 @@ requests. On a slow model, set `timeouts.generation` higher. (step 2) or every request costs zero, and a budget is something the dashboard displays rather than something that stops anything. The quotas page says so where it can, which is not the same as being told before you rely on it. -- **vLLM needs CUDA 13.** The version the agent installs needs driver 580 - and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and - install a vLLM built for your CUDA yourself. +- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer. + Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides` + (step 4), or install one built for your CUDA yourself. - **Two control planes have not been run for long.** The leader election and agent failover work and there is a pattern in `deploy/ha/`, but every cluster this has been run on in anger has had one. A single control plane is the diff --git a/docs/reference.md b/docs/reference.md index 225ec8b5..45e780ff 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -81,6 +81,10 @@ rather than repeated. | `pool` | `string` | [Endpoint](#endpoint) | | `weight` | `float64` | [Endpoint](#endpoint) | | `priority` | `int` | [Endpoint](#endpoint) | +| `gpu_class` | `string` | [EngineOverride](#engineoverride) | +| `nodes` | `[]string` | [EngineOverride](#engineoverride) | +| `labels` | `map[string]string` | [EngineOverride](#engineoverride) | +| `engines` | `map[string]string` | [EngineOverride](#engineoverride) | | `url` | `string` | [External](#external) | | `model` | `string` | [External](#external) | | `api_key_file` | `string` | [External](#external) | @@ -183,6 +187,7 @@ rather than repeated. | `tracing` | `Tracing` | [Server](#server) | | `mirror` | `Mirror` | [Server](#server) | | `engines` | `map[string]string` | [Server](#server) | +| `engine_overrides` | `[]EngineOverride` | [Server](#server) | | `backup_dir` | `string` | [Server](#server) | | `releases` | `string` | [Server](#server) | | `router` | `Router` | [Server](#server) | @@ -645,6 +650,32 @@ called. Empty means vLLM's names. `int` +## EngineOverride + +### `gpu_class` + +`string` + +GPUClass picks nodes with a GPU of this class ("v100-32g"). + +### `nodes` + +`[]string` + +Nodes picks nodes by id. + +### `labels` + +`map[string]string` + +Labels picks nodes carrying all of these. + +### `engines` + +`map[string]string` + +Engines are laid over the cluster's: a version, or "" for none. + ## External ### `url` @@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there. Unset keeps DefaultEngines. A version replaces the default; an empty one leaves that engine to be installed by hand. +### `engine_overrides` + +`[]EngineOverride` + +EngineOverrides give some nodes different engines from the rest: a +vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at +all on nodes meant only for llama.cpp. Each picks nodes and lays its +engines over the cluster's, in order, later ones last; an empty version +removes that engine from the nodes it picks. + ### `backup_dir` `string` diff --git a/internal/agent/agent.go b/internal/agent/agent.go index 19f9d273..a522901f 100644 --- a/internal/agent/agent.go +++ b/internal/agent/agent.go @@ -450,7 +450,7 @@ func (a *Agent) session(ctx context.Context, server string) error { // vLLM accepted a vLLM shard and failed it minutes later. reg.EngineVersions = a.engines(parent) reg.EnginesProbed = true - reg.InstallableEngines = provision.Installable() + reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled) reg.Features = []string{FeatureFabricTest} reg.RunningShards = a.runningShards() a.mu.Lock() diff --git a/internal/agent/disabled_test.go b/internal/agent/disabled_test.go new file mode 100644 index 00000000..350c9208 --- /dev/null +++ b/internal/agent/disabled_test.go @@ -0,0 +1,32 @@ +package agent + +import ( + "testing" + + llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines" +) + +func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) { + a := &Agent{} + a.cfg.Engines.Disabled = []string{"vllm"} + a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{ + {Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"}, + }}) + if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" { + t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants) + } + a.configuredVers = map[string]string{"vllm": "0.25.0"} + a.installedVers = map[string]string{"llamacpp": "b1"} + if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" { + t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got) + } +} + +func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) { + a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})} + a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}}) + if len(a.provWants) != 0 { + t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants) + } +} diff --git a/internal/agent/engines/engine.go b/internal/agent/engines/engine.go index 330d6d32..edc88294 100644 --- a/internal/agent/engines/engine.go +++ b/internal/agent/engines/engine.go @@ -213,6 +213,10 @@ func (r *Registry) Names() []string { // Config configures the engine adapters. type Config struct { + // Disabled are engines this node never installs or advertises, whatever + // the control plane asks: vLLM on a node of cards no build of it runs on, + // that is meant for llama.cpp alone. + Disabled []string `koanf:"disabled"` VLLM VLLMConfig `koanf:"vllm"` SGLang SGLangConfig `koanf:"sglang"` LlamaCpp LlamaCppConfig `koanf:"llamacpp"` diff --git a/internal/agent/provision.go b/internal/agent/provision.go index a0e876da..ea3f4486 100644 --- a/internal/agent/provision.go +++ b/internal/agent/provision.go @@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string { out = map[string]string{} } maps.Copy(out, a.installedVers) + maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) }) return out } +// byHand reports whether agent.yaml names this engine's binary, which opts +// it out of being installed. +func (a *Agent) byHand(engine string) bool { + if a.reg == nil { + return false + } + e, err := a.reg.Get(engine) + if err != nil { + return false + } + c, ok := e.(engines.Configured) + return ok && c.Configured() +} + +// disabled reports whether agent.yaml says this node never runs an engine: +// not installed when asked, and not advertised when found. +func (a *Agent) disabled(engine string) bool { + return slices.Contains(a.cfg.Engines.Disabled, engine) +} + // provisionEvery is how often installed engines are checked again. A library // removed by an unrelated upgrade is found here rather than by the next // replica to start. @@ -58,6 +79,15 @@ const provisionEvery = 10 * time.Minute func (a *Agent) setProvision(p *llmgwv1.Provision) { wants := []provision.Want{} for _, e := range p.GetEngines() { + if a.disabled(e.GetEngine()) { + continue // a control plane older than the node's say in it + } + if a.byHand(e.GetEngine()) { + // The operator named a binary for it: a local build, a CUDA + // runtime of their own. Installing one beside it would win — + // the installed build is used first — and quietly replace theirs. + continue + } wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()}) } a.mu.Lock() diff --git a/internal/agent/provision/recipe.go b/internal/agent/provision/recipe.go index e6355eb4..49c3c665 100644 --- a/internal/agent/provision/recipe.go +++ b/internal/agent/provision/recipe.go @@ -129,25 +129,72 @@ const minDriverForCUDA13 = 580 // Volta, so a V100 cannot run these builds at all. const minComputeForCUDA13 = 7.5 -// cudaGate says whether a node can run an engine built on CUDA 13. -func cudaGate(engine string, h discovery.Host) error { +// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor +// version compatibility, and on Volta. +const ( + minDriverForCUDA12 = 525 + minComputeForCUDA12 = 7.0 +) + +// cuda13Since is the first version of each Python engine built for CUDA 13. +var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"} + +// cudaGate says whether a node can run this version of an engine, and which +// CUDA it is built for. +// +// By version, because the line moved: a V100 cannot run vLLM 0.26 or later, +// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta +// nodes the older build (engine_overrides in the server's configuration) +// rather than going without vLLM, or without the Volta nodes. +func cudaGate(engine, version string, h discovery.Host) (variant string, err error) { switch h.Arch { case "amd64", "arm64": default: - return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch)) + return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch)) } if h.NVIDIADriver == "" { - return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"} + return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"} } - if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 { - return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s", - engine, minDriverForCUDA13, h.NVIDIADriver) + variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13 + if versionBefore(version, cuda13Since[engine]) { + variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12 } - if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 { - return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f", - engine, minComputeForCUDA13, cc) + if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver { + return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s", + engine, version, cuda, minDriver, h.NVIDIADriver) } - return nil + if cc := h.MinComputeCapability; cc > 0 && cc < minCC { + hint := "" + if cuda == "13" { + hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine]) + } + return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s", + engine, version, cuda, minCC, cc, hint) + } + return variant, nil +} + +// versionBefore reports whether version a is older than b, comparing the +// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a +// number counts as zero. +func versionBefore(a, b string) bool { + if b == "" { + return false + } + pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".") + for i := 0; i < max(len(pa), len(pb)); i++ { + x, y := 0, 0 + if i < len(pa) { + x = leadingInt(pa[i]) + } + if i < len(pb) { + y = leadingInt(pb[i]) + } + if x != y { + return x < y + } + } + return false } // Python engines need a C compiler at run time: Triton builds a small @@ -156,11 +203,12 @@ func cudaGate(engine string, h discovery.Host) error { var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}} func chooseVLLM(version string, h discovery.Host) (Build, error) { - if err := cudaGate("vLLM", h); err != nil { + variant, err := cudaGate("vLLM", version, h) + if err != nil { return Build{}, err } return Build{ - Variant: "cu13", + Variant: variant, Python: pythonForEngines, Requirements: []string{"vllm==" + version}, Binary: "vllm", @@ -170,13 +218,14 @@ func chooseVLLM(version string, h discovery.Host) (Build, error) { } func chooseSGLang(version string, h discovery.Host) (Build, error) { - if err := cudaGate("SGLang", h); err != nil { + variant, err := cudaGate("SGLang", version, h) + if err != nil { return Build{}, err } // SGLang is launched as a module, so what the agent is given is the // environment's interpreter, and the proof is importing it. return Build{ - Variant: "cu13", + Variant: variant, Python: pythonForEngines, Requirements: []string{"sglang==" + version}, Binary: "python", diff --git a/internal/cli/doctor.go b/internal/cli/doctor.go index d5c51df0..f61a8f8c 100644 --- a/internal/cli/doctor.go +++ b/internal/cli/doctor.go @@ -38,8 +38,11 @@ that file when the report is not enough to go on.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { doctor.SetConfigLoader(config.LoadAgent) - if opts.VLLMVersion == "" { + switch opts.VLLMVersion { + case "": opts.VLLMVersion = config.DefaultEngines["vllm"] + case "none": + opts.VLLMVersion = "" } ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second) defer cancel() diff --git a/internal/cli/node.go b/internal/cli/node.go index 712beaa9..2e4c14bf 100644 --- a/internal/cli/node.go +++ b/internal/cli/node.go @@ -12,6 +12,7 @@ import ( "github.com/spf13/cobra" + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision" "gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/onboard" @@ -113,13 +114,14 @@ engines are kept.`, f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)") f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)") f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs") - f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml") + f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs") f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed") f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH") f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband") f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority") f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from") f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem") + f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable") f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)") f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address") f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp") @@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error { } } } + for _, e := range o.DisableEngines { + if !slices.Contains(provision.Installable(), e) { + return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", ")) + } + } + // What the control plane will ask the node to install, so doctor checks + // that vLLM rather than its own default. Its overrides for a GPU class + // are not known until the node says what it has, and are not applied + // here; doctor's verdict on vLLM is a warning either way. + if o.VLLMVersion == "" { + if cfg, err := config.Load(na.serverCfg); err == nil { + o.VLLMVersion = cfg.WantedEngines()["vllm"] + if o.VLLMVersion == "" { + o.VLLMVersion = "none" + } + } + } o.Version = version.Version if o.Binary == "" { exe, err := os.Executable() diff --git a/internal/cli/server.go b/internal/cli/server.go index 1eb13411..ceff217b 100644 --- a/internal/cli/server.go +++ b/internal/cli/server.go @@ -5,19 +5,23 @@ import ( "crypto/tls" "errors" "fmt" - "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" - "gitea.mynodes.xyz/ilker/llmgw/internal/gguf" - "gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck" - "github.com/prometheus/client_golang/prometheus" "log/slog" + "maps" "net" "net/http" "os/signal" + "slices" "strings" "sync" "syscall" "time" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" + "gitea.mynodes.xyz/ilker/llmgw/internal/gguf" + "gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck" + "github.com/prometheus/client_golang/prometheus" + "github.com/spf13/cobra" "golang.org/x/net/http2" "golang.org/x/net/http2/h2c" @@ -496,8 +500,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live } } svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log) - svc.SetEngines(cfg.WantedEngines()) - log.Info("nodes install these engines", "engines", cfg.WantedEngines()) + svc.SetEnginesFor(func(n inventory.Node) map[string]string { + return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels) + }) + log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides)) // A standby must refuse agents as well as refusing to place. Commands // travel down the stream the agent opened, so a standby holding one is a // node the leader cannot command — and the leader's placements would fail diff --git a/internal/config/config.go b/internal/config/config.go index 32952463..24fefc24 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -8,6 +8,7 @@ package config import ( "fmt" "os" + "slices" "strings" "time" @@ -85,6 +86,13 @@ type Server struct { // one leaves that engine to be installed by hand. Engines map[string]string `koanf:"engines"` + // EngineOverrides give some nodes different engines from the rest: a + // vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at + // all on nodes meant only for llama.cpp. Each picks nodes and lays its + // engines over the cluster's, in order, later ones last; an empty version + // removes that engine from the nodes it picks. + EngineOverrides []EngineOverride `koanf:"engine_overrides"` + // BackupDir is where `llmgw server backup` writes, and where the newest // archive's age is read for llmgw_backup_last_success_timestamp_seconds. // Empty means /var/backups/llmgw-server. @@ -548,6 +556,14 @@ func Load(path string) (Server, error) { // Validate rejects configurations that would otherwise fail confusingly at // request time. func (s Server) Validate() error { + for i, o := range s.EngineOverrides { + if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 { + return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i) + } + if len(o.Engines) == 0 { + return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i) + } + } if s.Listen == "" { return fmt.Errorf("listen is empty") } @@ -626,6 +642,58 @@ var DefaultEngines = map[string]string{ "vllm": "0.30.0", } +// EngineOverride gives the nodes it picks different engines. Every selector +// set must match; at least one must be set. +type EngineOverride struct { + // GPUClass picks nodes with a GPU of this class ("v100-32g"). + GPUClass string `koanf:"gpu_class"` + // Nodes picks nodes by id. + Nodes []string `koanf:"nodes"` + // Labels picks nodes carrying all of these. + Labels map[string]string `koanf:"labels"` + // Engines are laid over the cluster's: a version, or "" for none. + Engines map[string]string `koanf:"engines"` +} + +// Picks reports whether the override applies to a node with this id, these +// GPU classes and these labels. +func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool { + if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 { + return false + } + if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) { + return false + } + if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) { + return false + } + for k, v := range o.Labels { + if labels[k] != v { + return false + } + } + return true +} + +// EnginesFor is what one node installs: WantedEngines with every override +// that picks it laid over, in order. +func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string { + out := s.WantedEngines() + for _, o := range s.EngineOverrides { + if !o.Picks(node, classes, labels) { + continue + } + for k, v := range o.Engines { + if v == "" { + delete(out, k) + continue + } + out[k] = v + } + } + return out +} + // WantedEngines is DefaultEngines with the configuration laid over it. func (s Server) WantedEngines() map[string]string { out := map[string]string{} diff --git a/internal/config/engines_test.go b/internal/config/engines_test.go new file mode 100644 index 00000000..8f79a94f --- /dev/null +++ b/internal/config/engines_test.go @@ -0,0 +1,38 @@ +package config + +import ( + "strings" + "testing" +) + +func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) { + s := Server{ + Engines: map[string]string{"vllm": "0.30.0"}, + EngineOverrides: []EngineOverride{ + // Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs. + {GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}}, + // Nodes meant for llama.cpp alone. + {Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}}, + }, + } + if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" { + t.Errorf("an H100 node: %v", got) + } + if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" { + t.Errorf("a V100 node: %v", got) + } + got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"}) + if _, has := got["vllm"]; has || got["llamacpp"] == "" { + t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got) + } + + for _, bad := range []EngineOverride{ + {Engines: map[string]string{"vllm": ""}}, + {GPUClass: "v100-32g"}, + } { + s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}} + if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") { + t.Errorf("%+v: %v", bad, err) + } + } +} diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go index d47b51f0..76d548e0 100644 --- a/internal/control/adminapi/api.go +++ b/internal/control/adminapi/api.go @@ -208,10 +208,13 @@ type NodeView struct { // GPUView is one card as a node's page shows it. type GPUView struct { - Index int `json:"index"` - Name string `json:"name"` - NUMA int `json:"numa"` - NVLinkPeers []int `json:"nvlink_peers,omitempty"` + Index int `json:"index"` + Name string `json:"name"` + // ComputeCapability is what decides which engine builds run on the card: + // CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones. + ComputeCapability string `json:"compute_capability,omitempty"` + NUMA int `json:"numa"` + NVLinkPeers []int `json:"nvlink_peers,omitempty"` // NearestNIC is the adapter closest to the card, and how close. NearestNIC string `json:"nearest_nic,omitempty"` NICPath string `json:"nic_path,omitempty"` @@ -497,7 +500,8 @@ func (a *API) Nodes() []NodeView { v.Load = loadView(n.Load, time.Now()) v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels for _, g := range n.GPUs { - gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers} + gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability, + NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers} for hca, rel := range g.NICPaths { if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) || (discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) { diff --git a/internal/control/agentsvc/installable_test.go b/internal/control/agentsvc/installable_test.go index e560a969..5ee5a40b 100644 --- a/internal/control/agentsvc/installable_test.go +++ b/internal/control/agentsvc/installable_test.go @@ -6,6 +6,7 @@ import ( llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1" "gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision" + "gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory" ) func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) { @@ -25,3 +26,31 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) { t.Error("uv, a tool for the others, is offered as an engine") } } + +func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) { + s := &Service{} + s.SetEnginesFor(func(n inventory.Node) map[string]string { + if n.ID == "volta" { + return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"} + } + return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"} + }) + reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}} + want := func(p *llmgwv1.Provision) map[string]string { + out := map[string]string{} + for _, e := range p.GetEngines() { + out[e.GetEngine()] = e.GetVersion() + } + return out + } + if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" { + t.Errorf("the Volta node: %v", got) + } + if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" { + t.Errorf("the Hopper node: %v", got) + } + // A node that disabled vLLM does not offer it, and is not asked for it. + if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" { + t.Errorf("asked a node for an engine it does not install: %v", got) + } +} diff --git a/internal/control/agentsvc/service.go b/internal/control/agentsvc/service.go index 88c44b90..f40f7e57 100644 --- a/internal/control/agentsvc/service.go +++ b/internal/control/agentsvc/service.go @@ -53,8 +53,9 @@ type Service struct { mu sync.RWMutex conns map[string]*conn - // engines is what every node is told to install, set once at start. - engines map[string]string + // enginesFor is what a node is told to install, set once at start: the + // cluster's engines, with any override that picks the node laid over. + enginesFor func(node inventory.Node) map[string]string // logWaits are the log requests awaiting a reply, by request id. logMu sync.Mutex @@ -68,7 +69,14 @@ type Service struct { } // SetEngines sets the engines every node installs, by name and version. -func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) } +func (s *Service) SetEngines(e map[string]string) { + e = maps.Clone(e) + s.enginesFor = func(inventory.Node) map[string]string { return e } +} + +// SetEnginesFor sets what each node installs, decided per node: a cluster +// of two GPU generations needs two vLLM builds, or vLLM on only one. +func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f } // conn is one agent's open stream. type conn struct { @@ -179,15 +187,8 @@ func (s *Service) Connect( // What this node should have installed. Sent on every registration, so a // node rebuilt from nothing is told the same as one that has been here a // month, and a changed version reaches every node on its next connect. - if s.engines != nil { - p := &llmgwv1.Provision{} - can := installable(reg) - for _, e := range slices.Sorted(maps.Keys(s.engines)) { - if !slices.Contains(can, e) { - continue - } - p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]}) - } + if s.enginesFor != nil { + p := s.provisionFor(node, reg) if err := stream.Send(&llmgwv1.ControlMessage{ Msg: &llmgwv1.ControlMessage_Provision{Provision: p}, }); err != nil { @@ -853,6 +854,21 @@ func rate(before, now uint64, secs float64) float64 { return float64(now-before) / secs } +// provisionFor is what a node is told to install: the engines decided for +// it, those it can install. +func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision { + p := &llmgwv1.Provision{} + can := installable(reg) + wants := s.enginesFor(node) + for _, e := range slices.Sorted(maps.Keys(wants)) { + if !slices.Contains(can, e) { + continue + } + p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]}) + } + return p +} + // installable are the engines a node can be asked to install. An agent older // than the list said nothing, and could install llama.cpp alone: asked for // vLLM too, it reported it unsupported, and a cluster half way through an diff --git a/internal/doctor/doctor.go b/internal/doctor/doctor.go index 191f9731..6cb5afcd 100644 --- a/internal/doctor/doctor.go +++ b/internal/doctor/doctor.go @@ -86,10 +86,12 @@ type Facts struct { // What installing vLLM here would meet: the version this llmgw installs, // where, the space there, and whether it already has. - VLLMVersion string `json:"vllm_version,omitempty"` - EnginesDir string `json:"engines_dir,omitempty"` - EnginesFreeGB float64 `json:"engines_free_gb,omitempty"` - VLLMInstalled bool `json:"vllm_installed"` + VLLMVersion string `json:"vllm_version,omitempty"` + // DisabledEngines are those agent.yaml says this node never runs. + DisabledEngines []string `json:"disabled_engines,omitempty"` + EnginesDir string `json:"engines_dir,omitempty"` + EnginesFreeGB float64 `json:"engines_free_gb,omitempty"` + VLLMInstalled bool `json:"vllm_installed"` } // Modules checked, and why each matters. @@ -238,6 +240,7 @@ func (f *Facts) readConfig(o Options) { _, err = net.InterfaceByName(f.MgmtIface) f.IfaceExists = err == nil f.EnginesDir = cfg.EnginesDir + f.DisabledEngines = cfg.Engines.Disabled f.LaunchMode = cfg.Engines.VLLM.LaunchMode if f.LaunchMode == "" { f.LaunchMode = "mp" @@ -525,7 +528,9 @@ func Evaluate(f Facts) []Finding { } } // --- engines --- - if f.VLLMVersion != "" { + // An engine disabled on this node: whether it could be installed is not + // a question anybody is asking. + if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") { vllmFindings(f, add) } @@ -549,15 +554,19 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) { add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "") return case err != nil: - add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(), - "upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one") + // A warning, not a failure: the node is still a node, and serves + // llama.cpp. Refusing it here kept older GPU generations out of a + // cluster for an engine nobody meant to run on them. + add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(), + "give this node a vLLM it can run with engine_overrides in the server's configuration, "+ + "or add it with --disable-engine vllm if it is meant for llama.cpp alone") return case f.VLLMInstalled: add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "") return } if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) { - add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB", + add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is", v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB), "free space there, or point engines_dir in agent.yaml at a larger disk") return diff --git a/internal/doctor/doctor_test.go b/internal/doctor/doctor_test.go index 87041dc2..22c69f75 100644 --- a/internal/doctor/doctor_test.go +++ b/internal/doctor/doctor_test.go @@ -151,8 +151,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) { says string }{ {"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"}, - {"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"}, - {"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"}, + // What vLLM cannot do keeps vLLM off the node, not the node out of + // the cluster: it still serves llama.cpp. + {"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"}, + {"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"}, + Warn, "before 0.26 is built for CUDA 12"}, + {"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0}, + VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"}, + {"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"}, {"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"}, {"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"}, } @@ -200,3 +206,15 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) { t.Errorf("ARM: %+v", f) } } + +func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) { + // The field: a V100 node meant for llama.cpp, refused for a vLLM it was + // never going to run. + f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, + VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}} + for _, fd := range Evaluate(f) { + if fd.Area == "engines" { + t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message) + } + } +} diff --git a/internal/onboard/onboard.go b/internal/onboard/onboard.go index 3ba5337e..ff1fa5a5 100644 --- a/internal/onboard/onboard.go +++ b/internal/onboard/onboard.go @@ -18,7 +18,6 @@ import ( "encoding/pem" "errors" "fmt" - "gitea.mynodes.xyz/ilker/llmgw/internal/release" "net" "os" "path/filepath" @@ -27,6 +26,8 @@ import ( "strings" "time" + "gitea.mynodes.xyz/ilker/llmgw/internal/release" + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery" "gitea.mynodes.xyz/ilker/llmgw/internal/doctor" "gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf" @@ -86,6 +87,15 @@ type Options struct { Releases string // IgnoreChecks goes on when doctor finds a problem. IgnoreChecks bool + // DisableEngines are engines this node never installs or advertises, + // written into its agent.yaml, and not checked by doctor: vLLM on a node + // of cards it does not run on, meant for llama.cpp alone. Narrower than + // IgnoreChecks, which also passes over everything else. + DisableEngines []string + // VLLMVersion is the vLLM the control plane has nodes install, for + // doctor to check against; "none" when it has them install none, empty + // for doctor's own default. + VLLMVersion string // SSHWait is how long an unreachable machine is waited for. SSHWait time.Duration @@ -390,6 +400,13 @@ func (r *Runner) check(ctx context.Context, res *Result) error { if r.Opts.IBRoot != "" { args += " --ib-root " + shellQuote(r.Opts.IBRoot) } + switch { + case slices.Contains(r.Opts.DisableEngines, "vllm"): + args += " --vllm none" + r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "") + case r.Opts.VLLMVersion != "": + args += " --vllm " + shellQuote(r.Opts.VLLMVersion) + } out, err := r.Remote.Run(ctx, args, nil) var rep struct { Findings []doctor.Finding `json:"findings"` @@ -596,8 +613,11 @@ func AgentYAML(o Options, res *Result) string { fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot) } } - if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" { + if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 { b.WriteString("engines:\n") + if len(o.DisableEngines) > 0 { + fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", ")) + } if o.LlamaCppBin != "" { fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin) } diff --git a/internal/onboard/onboard_test.go b/internal/onboard/onboard_test.go index 8a40cd73..0d69b9b2 100644 --- a/internal/onboard/onboard_test.go +++ b/internal/onboard/onboard_test.go @@ -280,3 +280,31 @@ func said(events []Event, level, s string) bool { } return false } + +func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) { + m := newFake() + r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"}) + if _, err := r.Run(t.Context()); err != nil { + t.Fatal(err) + } + if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") { + t.Errorf("agent.yaml:\n%s", yaml) + } + if !m.ran("sudo llmgw doctor --json --vllm none") { + t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts) + } + if !said(*events, Info, "joins with llama.cpp alone") { + t.Error("the reduced engine set went unsaid") + } + + // Not disabled: doctor checks the vLLM the control plane installs, not + // its own default. + m = newFake() + r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"}) + if _, err := r.Run(t.Context()); err != nil { + t.Fatal(err) + } + if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") { + t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts) + } +} diff --git a/internal/web/templates/node.html b/internal/web/templates/node.html index 89f7f6b4..c316e7d0 100644 --- a/internal/web/templates/node.html +++ b/internal/web/templates/node.html @@ -82,9 +82,9 @@ the adapter came up: a number that grows is the one to look at.
{{end}}| GPU | Name | NUMA | Nearest adapter | Multi-node NVLink | ||||
|---|---|---|---|---|---|---|---|---|
| GPU | Name | Compute | NUMA | Nearest adapter | Multi-node NVLink | |||
| {{.Index}} | {{.Name}} | {{.NUMA}} | +{{.Index}} | {{.Name}} | {{.ComputeCapability}} | {{.NUMA}} | {{if .NearestNIC}}{{.NearestNIC}} {{.NICPath}}{{else}}—{{end}} |
{{if .Domain}}joined{{if and .Bandwidth (ne .Bandwidth "Full")}} {{.Bandwidth}} bandwidth{{end}}{{else if .FabricState}}{{.FabricState}}{{else}}—{{end}} |