Decide engines per node, and let a node join without one it cannot run
engine_overrides give nodes picked by GPU class, id or labels their own engine versions; node add --disable-engine keeps one off a node; what vLLM cannot do on a node is a warning rather than a refused node; the CUDA requirement follows the version; a hand-named binary is no longer replaced by an installed one. Refs #2 (items 4 and 7). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
cba3e329fc
commit
e313f32696
21 files changed
+532
-62
No files matched your search
@@ -47,6 +47,30 @@ as nvidia-smi does, by PCI bus, and CUDA's own order is fastest first — on a
|
||||
node with two kinds of GPU, a shard could get a different card from the one it
|
||||
was given.
|
||||
|
||||
### Engines per node, and nodes that do not run them all
|
||||
|
||||
A cluster with older GPUs could not add them cleanly: `node add` refused a
|
||||
V100 node for the vLLM it could not run, though it was meant for llama.cpp,
|
||||
and `--ignore-checks` passed over every other problem with it.
|
||||
|
||||
- **What vLLM cannot do is a warning**, in `llmgw doctor` and `node add`: the
|
||||
node joins without it and serves what it does run. `node add` checks the
|
||||
vLLM the control plane installs, not the binary's own default.
|
||||
- **`node add --disable-engine vllm`** keeps an engine off a node for good:
|
||||
written as `engines.disabled` in its `agent.yaml`, never installed or
|
||||
advertised there, and not checked by doctor.
|
||||
- **`engine_overrides`** in `server.yaml` give nodes picked by GPU class, id
|
||||
or labels a different version of an engine, or none: a CUDA 12 vLLM on
|
||||
Volta beside a CUDA 13 one on newer cards.
|
||||
- **The CUDA requirement follows the version.** vLLM before 0.26 and SGLang
|
||||
before 0.5 are built for CUDA 12 and are installed on driver 525 and Volta;
|
||||
the threshold had been CUDA 13's for every version, so the doctor's advice
|
||||
to set an older version could not work. A refusal now says which versions
|
||||
would.
|
||||
- **An engine binary named in `agent.yaml` is used.** The agent had installed
|
||||
its own beside it, which was then used first.
|
||||
- The node page shows each GPU's compute capability.
|
||||
|
||||
### Serving from the mirror says where a model will run
|
||||
|
||||
The serve form put every GGUF model on the CPU: it offered no choice, and its
|
||||
|
||||
+39
-7
@@ -376,8 +376,10 @@ node gets the same dependencies whenever it is built, and installed with
|
||||
`--require-hashes` against the sha256 PyPI publishes. The resolved list stays in
|
||||
the environment as `llmgw-requirements.txt`. It needs:
|
||||
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5):
|
||||
vLLM is built for CUDA 13. A V100 cannot run it;
|
||||
- **driver 580 or newer and a Turing GPU or newer** (compute capability 7.5)
|
||||
for vLLM 0.26 and later, which are built for CUDA 13. A V100 cannot run
|
||||
them; vLLM before 0.26 is built for CUDA 12, and runs on driver 525 and
|
||||
Volta (7.0);
|
||||
- **about 25 GB free** under the engines directory while it installs;
|
||||
- **gcc**, which Triton builds its kernel launcher with. The agent installs it.
|
||||
|
||||
@@ -394,8 +396,38 @@ engines:
|
||||
```
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself instead, set its version to `""` and name its path in
|
||||
the node's `agent.yaml` (`engines.vllm.bin`, `engines.sglang.python`).
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
|
||||
#### Different GPUs, different engines
|
||||
|
||||
A cluster of two GPU generations may need two builds of an engine, or one on
|
||||
only some nodes. `engine_overrides` picks nodes — by a GPU class they have, by
|
||||
id, or by labels, every one given having to match — and lays engines over the
|
||||
cluster's for them, in order:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
vllm: "0.30.0" # CUDA 13: Turing and newer
|
||||
engine_overrides:
|
||||
- gpu_class: v100-32g # Volta, which CUDA 13 dropped
|
||||
engines: { vllm: "0.25.1" } # the last CUDA 12 vLLM
|
||||
- labels: { role: gguf } # nodes meant for llama.cpp alone
|
||||
engines: { vllm: "" } # none
|
||||
```
|
||||
|
||||
A node told nothing it can run of an engine joins without it: `llmgw doctor`
|
||||
and `node add` warn, and the node serves what it does run. To keep an engine
|
||||
off a node for good, whatever the control plane says, add the node with
|
||||
`llmgw node add --disable-engine vllm`, which writes `engines.disabled: [vllm]`
|
||||
into its `agent.yaml` and leaves vLLM out of doctor's checks. That is narrower
|
||||
than `--ignore-checks`, which passes over every other problem as well.
|
||||
|
||||
A llama.cpp CUDA build is checked for the GPU it is meant for: installing it
|
||||
asks it which devices it sees, and a replica placed on GPUs is not ready until
|
||||
its process holds memory on each of them. A build missing a CUDA library fails
|
||||
there, naming the library, rather than serving on the CPU.
|
||||
|
||||
After changing a version, check it still takes the flags llmgw builds for it,
|
||||
which drift between releases:
|
||||
@@ -1216,9 +1248,9 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
(step 2) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **vLLM needs CUDA 13.** The version the agent installs needs driver 580
|
||||
and a Turing GPU or newer. On older hardware, set `engines.vllm` to `""` and
|
||||
install a vLLM built for your CUDA yourself.
|
||||
- **vLLM 0.26 and later need CUDA 13**, driver 580 and a Turing GPU or newer.
|
||||
Give older hardware a CUDA 12 vLLM (before 0.26) with `engine_overrides`
|
||||
(step 4), or install one built for your CUDA yourself.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
|
||||
@@ -81,6 +81,10 @@ rather than repeated.
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -183,6 +187,7 @@ rather than repeated.
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
| `router` | `Router` | [Server](#server) |
|
||||
@@ -645,6 +650,32 @@ called. Empty means vLLM's names.
|
||||
|
||||
`int`
|
||||
|
||||
## EngineOverride
|
||||
|
||||
### `gpu_class`
|
||||
|
||||
`string`
|
||||
|
||||
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]string`
|
||||
|
||||
Nodes picks nodes by id.
|
||||
|
||||
### `labels`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Labels picks nodes carrying all of these.
|
||||
|
||||
### `engines`
|
||||
|
||||
`map[string]string`
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -1412,6 +1443,16 @@ never trusted to already have what somebody once put there.
|
||||
Unset keeps DefaultEngines. A version replaces the default; an empty
|
||||
one leaves that engine to be installed by hand.
|
||||
|
||||
### `engine_overrides`
|
||||
|
||||
`[]EngineOverride`
|
||||
|
||||
EngineOverrides give some nodes different engines from the rest: a
|
||||
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
engines over the cluster's, in order, later ones last; an empty version
|
||||
removes that engine from the nodes it picks.
|
||||
|
||||
### `backup_dir`
|
||||
|
||||
`string`
|
||||
|
||||
@@ -450,7 +450,7 @@ func (a *Agent) session(ctx context.Context, server string) error {
|
||||
// vLLM accepted a vLLM shard and failed it minutes later.
|
||||
reg.EngineVersions = a.engines(parent)
|
||||
reg.EnginesProbed = true
|
||||
reg.InstallableEngines = provision.Installable()
|
||||
reg.InstallableEngines = slices.DeleteFunc(provision.Installable(), a.disabled)
|
||||
reg.Features = []string{FeatureFabricTest}
|
||||
reg.RunningShards = a.runningShards()
|
||||
a.mu.Lock()
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNeitherInstalledNorAdvertised(t *testing.T) {
|
||||
a := &Agent{}
|
||||
a.cfg.Engines.Disabled = []string{"vllm"}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{
|
||||
{Engine: "llamacpp", Version: "b1"}, {Engine: "vllm", Version: "0.30.0"},
|
||||
}})
|
||||
if len(a.provWants) != 1 || a.provWants[0].Engine != "llamacpp" {
|
||||
t.Errorf("wants %+v: vLLM is disabled here and must not be installed", a.provWants)
|
||||
}
|
||||
a.configuredVers = map[string]string{"vllm": "0.25.0"}
|
||||
a.installedVers = map[string]string{"llamacpp": "b1"}
|
||||
if got := a.engineReportLocked(); got["vllm"] != "" || got["llamacpp"] != "b1" {
|
||||
t.Errorf("advertised %v: a disabled engine found on disk is still not offered", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineNamedByHandIsNotInstalledOverIt(t *testing.T) {
|
||||
a := &Agent{reg: engines.NewRegistry(engines.Config{LlamaCpp: engines.LlamaCppConfig{Bin: "/opt/own/llama-server"}})}
|
||||
a.setProvision(&llmgwv1.Provision{Engines: []*llmgwv1.EngineWant{{Engine: "llamacpp", Version: "b1"}}})
|
||||
if len(a.provWants) != 0 {
|
||||
t.Errorf("wants %+v: an installed build would be used over the operator's own", a.provWants)
|
||||
}
|
||||
}
|
||||
@@ -213,6 +213,10 @@ func (r *Registry) Names() []string {
|
||||
|
||||
// Config configures the engine adapters.
|
||||
type Config struct {
|
||||
// Disabled are engines this node never installs or advertises, whatever
|
||||
// the control plane asks: vLLM on a node of cards no build of it runs on,
|
||||
// that is meant for llama.cpp alone.
|
||||
Disabled []string `koanf:"disabled"`
|
||||
VLLM VLLMConfig `koanf:"vllm"`
|
||||
SGLang SGLangConfig `koanf:"sglang"`
|
||||
LlamaCpp LlamaCppConfig `koanf:"llamacpp"`
|
||||
|
||||
@@ -46,9 +46,30 @@ func (a *Agent) engineReportLocked() map[string]string {
|
||||
out = map[string]string{}
|
||||
}
|
||||
maps.Copy(out, a.installedVers)
|
||||
maps.DeleteFunc(out, func(e, _ string) bool { return a.disabled(e) })
|
||||
return out
|
||||
}
|
||||
|
||||
// byHand reports whether agent.yaml names this engine's binary, which opts
|
||||
// it out of being installed.
|
||||
func (a *Agent) byHand(engine string) bool {
|
||||
if a.reg == nil {
|
||||
return false
|
||||
}
|
||||
e, err := a.reg.Get(engine)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
c, ok := e.(engines.Configured)
|
||||
return ok && c.Configured()
|
||||
}
|
||||
|
||||
// disabled reports whether agent.yaml says this node never runs an engine:
|
||||
// not installed when asked, and not advertised when found.
|
||||
func (a *Agent) disabled(engine string) bool {
|
||||
return slices.Contains(a.cfg.Engines.Disabled, engine)
|
||||
}
|
||||
|
||||
// provisionEvery is how often installed engines are checked again. A library
|
||||
// removed by an unrelated upgrade is found here rather than by the next
|
||||
// replica to start.
|
||||
@@ -58,6 +79,15 @@ const provisionEvery = 10 * time.Minute
|
||||
func (a *Agent) setProvision(p *llmgwv1.Provision) {
|
||||
wants := []provision.Want{}
|
||||
for _, e := range p.GetEngines() {
|
||||
if a.disabled(e.GetEngine()) {
|
||||
continue // a control plane older than the node's say in it
|
||||
}
|
||||
if a.byHand(e.GetEngine()) {
|
||||
// The operator named a binary for it: a local build, a CUDA
|
||||
// runtime of their own. Installing one beside it would win —
|
||||
// the installed build is used first — and quietly replace theirs.
|
||||
continue
|
||||
}
|
||||
wants = append(wants, provision.Want{Engine: e.GetEngine(), Version: e.GetVersion()})
|
||||
}
|
||||
a.mu.Lock()
|
||||
|
||||
@@ -129,25 +129,72 @@ const minDriverForCUDA13 = 580
|
||||
// Volta, so a V100 cannot run these builds at all.
|
||||
const minComputeForCUDA13 = 7.5
|
||||
|
||||
// cudaGate says whether a node can run an engine built on CUDA 13.
|
||||
func cudaGate(engine string, h discovery.Host) error {
|
||||
// The CUDA 12 builds before them run on any driver from 525, by CUDA's minor
|
||||
// version compatibility, and on Volta.
|
||||
const (
|
||||
minDriverForCUDA12 = 525
|
||||
minComputeForCUDA12 = 7.0
|
||||
)
|
||||
|
||||
// cuda13Since is the first version of each Python engine built for CUDA 13.
|
||||
var cuda13Since = map[string]string{"vLLM": "0.26", "SGLang": "0.5"}
|
||||
|
||||
// cudaGate says whether a node can run this version of an engine, and which
|
||||
// CUDA it is built for.
|
||||
//
|
||||
// By version, because the line moved: a V100 cannot run vLLM 0.26 or later,
|
||||
// and runs 0.25 — so a cluster with Volta and newer cards gives the Volta
|
||||
// nodes the older build (engine_overrides in the server's configuration)
|
||||
// rather than going without vLLM, or without the Volta nodes.
|
||||
func cudaGate(engine, version string, h discovery.Host) (variant string, err error) {
|
||||
switch h.Arch {
|
||||
case "amd64", "arm64":
|
||||
default:
|
||||
return fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
return "", fmt.Errorf("%s publishes no Linux build for %s", engine, orUnknown(h.Arch))
|
||||
}
|
||||
if h.NVIDIADriver == "" {
|
||||
return NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
return "", NotHere{engine + " runs on NVIDIA GPUs and this node has none; it serves llama.cpp models"}
|
||||
}
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriverForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, minDriverForCUDA13, h.NVIDIADriver)
|
||||
variant, cuda, minDriver, minCC := "cu13", "13", minDriverForCUDA13, minComputeForCUDA13
|
||||
if versionBefore(version, cuda13Since[engine]) {
|
||||
variant, cuda, minDriver, minCC = "cu12", "12", minDriverForCUDA12, minComputeForCUDA12
|
||||
}
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minComputeForCUDA13 {
|
||||
return fmt.Errorf("%s is built for CUDA 13, which supports GPUs of compute capability %.1f and newer; this node has %.1f",
|
||||
engine, minComputeForCUDA13, cc)
|
||||
if major := leadingInt(h.NVIDIADriver); major > 0 && major < minDriver {
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which needs NVIDIA driver %d or newer; this node has %s",
|
||||
engine, version, cuda, minDriver, h.NVIDIADriver)
|
||||
}
|
||||
return nil
|
||||
if cc := h.MinComputeCapability; cc > 0 && cc < minCC {
|
||||
hint := ""
|
||||
if cuda == "13" {
|
||||
hint = fmt.Sprintf("; %s before %s is built for CUDA 12 and runs on it", engine, cuda13Since[engine])
|
||||
}
|
||||
return "", fmt.Errorf("%s %s is built for CUDA %s, which supports GPUs of compute capability %.1f and newer; this node has %.1f%s",
|
||||
engine, version, cuda, minCC, cc, hint)
|
||||
}
|
||||
return variant, nil
|
||||
}
|
||||
|
||||
// versionBefore reports whether version a is older than b, comparing the
|
||||
// dotted numbers in turn ("0.25.1" before "0.26"). A part that is not a
|
||||
// number counts as zero.
|
||||
func versionBefore(a, b string) bool {
|
||||
if b == "" {
|
||||
return false
|
||||
}
|
||||
pa, pb := strings.Split(strings.TrimPrefix(a, "v"), "."), strings.Split(b, ".")
|
||||
for i := 0; i < max(len(pa), len(pb)); i++ {
|
||||
x, y := 0, 0
|
||||
if i < len(pa) {
|
||||
x = leadingInt(pa[i])
|
||||
}
|
||||
if i < len(pb) {
|
||||
y = leadingInt(pb[i])
|
||||
}
|
||||
if x != y {
|
||||
return x < y
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Python engines need a C compiler at run time: Triton builds a small
|
||||
@@ -156,11 +203,12 @@ func cudaGate(engine string, h discovery.Host) error {
|
||||
var compilerPackages = map[string][]string{"deb": {"gcc"}, "rpm": {"gcc"}}
|
||||
|
||||
func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("vLLM", h); err != nil {
|
||||
variant, err := cudaGate("vLLM", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"vllm==" + version},
|
||||
Binary: "vllm",
|
||||
@@ -170,13 +218,14 @@ func chooseVLLM(version string, h discovery.Host) (Build, error) {
|
||||
}
|
||||
|
||||
func chooseSGLang(version string, h discovery.Host) (Build, error) {
|
||||
if err := cudaGate("SGLang", h); err != nil {
|
||||
variant, err := cudaGate("SGLang", version, h)
|
||||
if err != nil {
|
||||
return Build{}, err
|
||||
}
|
||||
// SGLang is launched as a module, so what the agent is given is the
|
||||
// environment's interpreter, and the proof is importing it.
|
||||
return Build{
|
||||
Variant: "cu13",
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Requirements: []string{"sglang==" + version},
|
||||
Binary: "python",
|
||||
|
||||
@@ -38,8 +38,11 @@ that file when the report is not enough to go on.`,
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
doctor.SetConfigLoader(config.LoadAgent)
|
||||
if opts.VLLMVersion == "" {
|
||||
switch opts.VLLMVersion {
|
||||
case "":
|
||||
opts.VLLMVersion = config.DefaultEngines["vllm"]
|
||||
case "none":
|
||||
opts.VLLMVersion = ""
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
|
||||
+20
-1
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
@@ -113,13 +114,14 @@ engines are kept.`,
|
||||
f.StringVar(&na.opts.MgmtAddr, "mgmt-addr", "", "address engines are reached at (default: found by the agent)")
|
||||
f.StringVar(&na.opts.MgmtIface, "mgmt-if", "", "interface NCCL bootstraps over (default: the one holding that address)")
|
||||
f.StringVar(&na.opts.LlamaCppBin, "llamacpp-bin", "", "a llama-server you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed; also set engines.vllm: \"\" in server.yaml")
|
||||
f.StringVar(&na.opts.VLLMBin, "vllm-bin", "", "a vllm you installed, instead of the one the agent installs")
|
||||
f.StringVar(&na.opts.SGLangPython, "sglang-python", "", "python of an SGLang virtualenv you installed")
|
||||
f.StringVar(&na.opts.NvidiaSMI, "nvidia-smi", "", "where nvidia-smi is, when it is not on PATH")
|
||||
f.StringVar(&na.opts.IBRoot, "ib-root", "", "the sysfs InfiniBand directory, when not /sys/class/infiniband")
|
||||
f.StringVar(&na.opts.PKI, "pki", defaultPKIDir, "directory holding the certificate authority")
|
||||
f.StringVar(&na.opts.Releases, "releases", "https://gitea.mynodes.xyz/ilker/llmgw/releases/download", "where release files are downloaded from")
|
||||
f.BoolVar(&na.opts.IgnoreChecks, "ignore-checks", false, "add the node even when doctor finds a problem")
|
||||
f.StringArrayVar(&na.opts.DisableEngines, "disable-engine", nil, "an engine this node never installs, e.g. vllm on GPUs it does not run on; repeatable")
|
||||
f.StringVar(&na.admin, "admin", "", "the control plane's admin API, to watch the node join (default: from server.yaml, else 127.0.0.1:9090)")
|
||||
f.StringVar(&na.serverCfg, "config", "/etc/llmgw/server.yaml", "the control plane's configuration, for the admin API's address")
|
||||
f.BoolVar(&na.waitAll, "wait", false, "wait for every engine, vLLM included, not only llama.cpp")
|
||||
@@ -148,6 +150,23 @@ func (na *nodeAdd) prepare() error {
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, e := range o.DisableEngines {
|
||||
if !slices.Contains(provision.Installable(), e) {
|
||||
return fmt.Errorf("--disable-engine %s: the engines are %s", e, strings.Join(provision.Installable(), ", "))
|
||||
}
|
||||
}
|
||||
// What the control plane will ask the node to install, so doctor checks
|
||||
// that vLLM rather than its own default. Its overrides for a GPU class
|
||||
// are not known until the node says what it has, and are not applied
|
||||
// here; doctor's verdict on vLLM is a warning either way.
|
||||
if o.VLLMVersion == "" {
|
||||
if cfg, err := config.Load(na.serverCfg); err == nil {
|
||||
o.VLLMVersion = cfg.WantedEngines()["vllm"]
|
||||
if o.VLLMVersion == "" {
|
||||
o.VLLMVersion = "none"
|
||||
}
|
||||
}
|
||||
}
|
||||
o.Version = version.Version
|
||||
if o.Binary == "" {
|
||||
exe, err := os.Executable()
|
||||
|
||||
+12
-6
@@ -5,19 +5,23 @@ import (
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
"log/slog"
|
||||
"maps"
|
||||
"net"
|
||||
"net/http"
|
||||
"os/signal"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"github.com/prometheus/client_golang/prometheus"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"golang.org/x/net/http2"
|
||||
"golang.org/x/net/http2/h2c"
|
||||
@@ -496,8 +500,10 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
}
|
||||
}
|
||||
svc := agentsvc.New(ctl.Inventory(), ctl.Reconciler(), requireIdentity, log)
|
||||
svc.SetEngines(cfg.WantedEngines())
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines())
|
||||
svc.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
return cfg.EnginesFor(n.ID, slices.Sorted(maps.Keys(n.GPUsByClass())), n.Labels)
|
||||
})
|
||||
log.Info("nodes install these engines", "engines", cfg.WantedEngines(), "overrides", len(cfg.EngineOverrides))
|
||||
// A standby must refuse agents as well as refusing to place. Commands
|
||||
// travel down the stream the agent opened, so a standby holding one is a
|
||||
// node the leader cannot command — and the leader's placements would fail
|
||||
|
||||
@@ -8,6 +8,7 @@ package config
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -85,6 +86,13 @@ type Server struct {
|
||||
// one leaves that engine to be installed by hand.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
|
||||
// EngineOverrides give some nodes different engines from the rest: a
|
||||
// vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
||||
// all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
||||
// engines over the cluster's, in order, later ones last; an empty version
|
||||
// removes that engine from the nodes it picks.
|
||||
EngineOverrides []EngineOverride `koanf:"engine_overrides"`
|
||||
|
||||
// BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
// archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
// Empty means /var/backups/llmgw-server.
|
||||
@@ -548,6 +556,14 @@ func Load(path string) (Server, error) {
|
||||
// Validate rejects configurations that would otherwise fail confusingly at
|
||||
// request time.
|
||||
func (s Server) Validate() error {
|
||||
for i, o := range s.EngineOverrides {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] picks no nodes: give it a gpu_class, nodes or labels", i)
|
||||
}
|
||||
if len(o.Engines) == 0 {
|
||||
return fmt.Errorf("engine_overrides[%d] changes no engine: give it engines, with \"\" for none", i)
|
||||
}
|
||||
}
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
@@ -626,6 +642,58 @@ var DefaultEngines = map[string]string{
|
||||
"vllm": "0.30.0",
|
||||
}
|
||||
|
||||
// EngineOverride gives the nodes it picks different engines. Every selector
|
||||
// set must match; at least one must be set.
|
||||
type EngineOverride struct {
|
||||
// GPUClass picks nodes with a GPU of this class ("v100-32g").
|
||||
GPUClass string `koanf:"gpu_class"`
|
||||
// Nodes picks nodes by id.
|
||||
Nodes []string `koanf:"nodes"`
|
||||
// Labels picks nodes carrying all of these.
|
||||
Labels map[string]string `koanf:"labels"`
|
||||
// Engines are laid over the cluster's: a version, or "" for none.
|
||||
Engines map[string]string `koanf:"engines"`
|
||||
}
|
||||
|
||||
// Picks reports whether the override applies to a node with this id, these
|
||||
// GPU classes and these labels.
|
||||
func (o EngineOverride) Picks(node string, classes []string, labels map[string]string) bool {
|
||||
if o.GPUClass == "" && len(o.Nodes) == 0 && len(o.Labels) == 0 {
|
||||
return false
|
||||
}
|
||||
if o.GPUClass != "" && !slices.Contains(classes, o.GPUClass) {
|
||||
return false
|
||||
}
|
||||
if len(o.Nodes) > 0 && !slices.Contains(o.Nodes, node) {
|
||||
return false
|
||||
}
|
||||
for k, v := range o.Labels {
|
||||
if labels[k] != v {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// EnginesFor is what one node installs: WantedEngines with every override
|
||||
// that picks it laid over, in order.
|
||||
func (s Server) EnginesFor(node string, classes []string, labels map[string]string) map[string]string {
|
||||
out := s.WantedEngines()
|
||||
for _, o := range s.EngineOverrides {
|
||||
if !o.Picks(node, classes, labels) {
|
||||
continue
|
||||
}
|
||||
for k, v := range o.Engines {
|
||||
if v == "" {
|
||||
delete(out, k)
|
||||
continue
|
||||
}
|
||||
out[k] = v
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// WantedEngines is DefaultEngines with the configuration laid over it.
|
||||
func (s Server) WantedEngines() map[string]string {
|
||||
out := map[string]string{}
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestEachNodeGetsTheEnginesItsOverridesSay(t *testing.T) {
|
||||
s := Server{
|
||||
Engines: map[string]string{"vllm": "0.30.0"},
|
||||
EngineOverrides: []EngineOverride{
|
||||
// Volta: CUDA 13 dropped it, CUDA 12 vLLM still runs.
|
||||
{GPUClass: "v100-32g", Engines: map[string]string{"vllm": "0.25.1"}},
|
||||
// Nodes meant for llama.cpp alone.
|
||||
{Labels: map[string]string{"role": "gguf"}, Engines: map[string]string{"vllm": ""}},
|
||||
},
|
||||
}
|
||||
if got := s.EnginesFor("h1", []string{"h100-80g"}, nil); got["vllm"] != "0.30.0" || got["llamacpp"] == "" {
|
||||
t.Errorf("an H100 node: %v", got)
|
||||
}
|
||||
if got := s.EnginesFor("v1", []string{"v100-32g"}, nil); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("a V100 node: %v", got)
|
||||
}
|
||||
got := s.EnginesFor("v2", []string{"v100-32g"}, map[string]string{"role": "gguf"})
|
||||
if _, has := got["vllm"]; has || got["llamacpp"] == "" {
|
||||
t.Errorf("a V100 node for llama.cpp alone, the later override last: %v", got)
|
||||
}
|
||||
|
||||
for _, bad := range []EngineOverride{
|
||||
{Engines: map[string]string{"vllm": ""}},
|
||||
{GPUClass: "v100-32g"},
|
||||
} {
|
||||
s := Server{Listen: ":8080", EngineOverrides: []EngineOverride{bad}}
|
||||
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "engine_overrides[0]") {
|
||||
t.Errorf("%+v: %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -208,10 +208,13 @@ type NodeView struct {
|
||||
|
||||
// GPUView is one card as a node's page shows it.
|
||||
type GPUView struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
// ComputeCapability is what decides which engine builds run on the card:
|
||||
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
|
||||
ComputeCapability string `json:"compute_capability,omitempty"`
|
||||
NUMA int `json:"numa"`
|
||||
NVLinkPeers []int `json:"nvlink_peers,omitempty"`
|
||||
// NearestNIC is the adapter closest to the card, and how close.
|
||||
NearestNIC string `json:"nearest_nic,omitempty"`
|
||||
NICPath string `json:"nic_path,omitempty"`
|
||||
@@ -497,7 +500,8 @@ func (a *API) Nodes() []NodeView {
|
||||
v.Load = loadView(n.Load, time.Now())
|
||||
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
|
||||
for _, g := range n.GPUs {
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
for hca, rel := range g.NICPaths {
|
||||
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
|
||||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
@@ -25,3 +26,31 @@ func TestANodeIsAskedOnlyForEnginesItCanInstall(t *testing.T) {
|
||||
t.Error("uv, a tool for the others, is offered as an engine")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachNodeIsToldTheEnginesDecidedForIt(t *testing.T) {
|
||||
s := &Service{}
|
||||
s.SetEnginesFor(func(n inventory.Node) map[string]string {
|
||||
if n.ID == "volta" {
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.25.1"}
|
||||
}
|
||||
return map[string]string{"llamacpp": "b1", "vllm": "0.30.0"}
|
||||
})
|
||||
reg := &llmgwv1.Register{InstallableEngines: []string{"llamacpp", "vllm"}}
|
||||
want := func(p *llmgwv1.Provision) map[string]string {
|
||||
out := map[string]string{}
|
||||
for _, e := range p.GetEngines() {
|
||||
out[e.GetEngine()] = e.GetVersion()
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "volta"}, reg)); got["vllm"] != "0.25.1" {
|
||||
t.Errorf("the Volta node: %v", got)
|
||||
}
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, reg)); got["vllm"] != "0.30.0" {
|
||||
t.Errorf("the Hopper node: %v", got)
|
||||
}
|
||||
// A node that disabled vLLM does not offer it, and is not asked for it.
|
||||
if got := want(s.provisionFor(inventory.Node{ID: "hopper"}, &llmgwv1.Register{InstallableEngines: []string{"llamacpp"}})); got["vllm"] != "" {
|
||||
t.Errorf("asked a node for an engine it does not install: %v", got)
|
||||
}
|
||||
}
|
||||
@@ -53,8 +53,9 @@ type Service struct {
|
||||
mu sync.RWMutex
|
||||
conns map[string]*conn
|
||||
|
||||
// engines is what every node is told to install, set once at start.
|
||||
engines map[string]string
|
||||
// enginesFor is what a node is told to install, set once at start: the
|
||||
// cluster's engines, with any override that picks the node laid over.
|
||||
enginesFor func(node inventory.Node) map[string]string
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -68,7 +69,14 @@ type Service struct {
|
||||
}
|
||||
|
||||
// SetEngines sets the engines every node installs, by name and version.
|
||||
func (s *Service) SetEngines(e map[string]string) { s.engines = maps.Clone(e) }
|
||||
func (s *Service) SetEngines(e map[string]string) {
|
||||
e = maps.Clone(e)
|
||||
s.enginesFor = func(inventory.Node) map[string]string { return e }
|
||||
}
|
||||
|
||||
// SetEnginesFor sets what each node installs, decided per node: a cluster
|
||||
// of two GPU generations needs two vLLM builds, or vLLM on only one.
|
||||
func (s *Service) SetEnginesFor(f func(node inventory.Node) map[string]string) { s.enginesFor = f }
|
||||
|
||||
// conn is one agent's open stream.
|
||||
type conn struct {
|
||||
@@ -179,15 +187,8 @@ func (s *Service) Connect(
|
||||
// What this node should have installed. Sent on every registration, so a
|
||||
// node rebuilt from nothing is told the same as one that has been here a
|
||||
// month, and a changed version reaches every node on its next connect.
|
||||
if s.engines != nil {
|
||||
p := &llmgwv1.Provision{}
|
||||
can := installable(reg)
|
||||
for _, e := range slices.Sorted(maps.Keys(s.engines)) {
|
||||
if !slices.Contains(can, e) {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: s.engines[e]})
|
||||
}
|
||||
if s.enginesFor != nil {
|
||||
p := s.provisionFor(node, reg)
|
||||
if err := stream.Send(&llmgwv1.ControlMessage{
|
||||
Msg: &llmgwv1.ControlMessage_Provision{Provision: p},
|
||||
}); err != nil {
|
||||
@@ -853,6 +854,21 @@ func rate(before, now uint64, secs float64) float64 {
|
||||
return float64(now-before) / secs
|
||||
}
|
||||
|
||||
// provisionFor is what a node is told to install: the engines decided for
|
||||
// it, those it can install.
|
||||
func (s *Service) provisionFor(node inventory.Node, reg *llmgwv1.Register) *llmgwv1.Provision {
|
||||
p := &llmgwv1.Provision{}
|
||||
can := installable(reg)
|
||||
wants := s.enginesFor(node)
|
||||
for _, e := range slices.Sorted(maps.Keys(wants)) {
|
||||
if !slices.Contains(can, e) {
|
||||
continue
|
||||
}
|
||||
p.Engines = append(p.Engines, &llmgwv1.EngineWant{Engine: e, Version: wants[e]})
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// installable are the engines a node can be asked to install. An agent older
|
||||
// than the list said nothing, and could install llama.cpp alone: asked for
|
||||
// vLLM too, it reported it unsupported, and a cluster half way through an
|
||||
|
||||
@@ -86,10 +86,12 @@ type Facts struct {
|
||||
|
||||
// What installing vLLM here would meet: the version this llmgw installs,
|
||||
// where, the space there, and whether it already has.
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
VLLMVersion string `json:"vllm_version,omitempty"`
|
||||
// DisabledEngines are those agent.yaml says this node never runs.
|
||||
DisabledEngines []string `json:"disabled_engines,omitempty"`
|
||||
EnginesDir string `json:"engines_dir,omitempty"`
|
||||
EnginesFreeGB float64 `json:"engines_free_gb,omitempty"`
|
||||
VLLMInstalled bool `json:"vllm_installed"`
|
||||
}
|
||||
|
||||
// Modules checked, and why each matters.
|
||||
@@ -238,6 +240,7 @@ func (f *Facts) readConfig(o Options) {
|
||||
_, err = net.InterfaceByName(f.MgmtIface)
|
||||
f.IfaceExists = err == nil
|
||||
f.EnginesDir = cfg.EnginesDir
|
||||
f.DisabledEngines = cfg.Engines.Disabled
|
||||
f.LaunchMode = cfg.Engines.VLLM.LaunchMode
|
||||
if f.LaunchMode == "" {
|
||||
f.LaunchMode = "mp"
|
||||
@@ -525,7 +528,9 @@ func Evaluate(f Facts) []Finding {
|
||||
}
|
||||
}
|
||||
// --- engines ---
|
||||
if f.VLLMVersion != "" {
|
||||
// An engine disabled on this node: whether it could be installed is not
|
||||
// a question anybody is asking.
|
||||
if f.VLLMVersion != "" && !slices.Contains(f.DisabledEngines, "vllm") {
|
||||
vllmFindings(f, add)
|
||||
}
|
||||
|
||||
@@ -549,15 +554,19 @@ func vllmFindings(f Facts, add func(Level, string, string, string)) {
|
||||
add(Info, "engines", "vLLM will not be installed: "+nh.Reason, "")
|
||||
return
|
||||
case err != nil:
|
||||
add(Fail, "engines", "vLLM "+v+" cannot be installed: "+err.Error(),
|
||||
"upgrade the NVIDIA driver, or set the control plane's engines.vllm to a version for this one")
|
||||
// A warning, not a failure: the node is still a node, and serves
|
||||
// llama.cpp. Refusing it here kept older GPU generations out of a
|
||||
// cluster for an engine nobody meant to run on them.
|
||||
add(Warn, "engines", "vLLM "+v+" will not run here, and the node joins without it: "+err.Error(),
|
||||
"give this node a vLLM it can run with engine_overrides in the server's configuration, "+
|
||||
"or add it with --disable-engine vllm if it is meant for llama.cpp alone")
|
||||
return
|
||||
case f.VLLMInstalled:
|
||||
add(OK, "engines", "vLLM "+v+" is installed under "+f.EnginesDir, "")
|
||||
return
|
||||
}
|
||||
if f.EnginesFreeGB > 0 && f.EnginesFreeGB < float64(b.FreeGB) {
|
||||
add(Fail, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB",
|
||||
add(Warn, "engines", fmt.Sprintf("vLLM %s needs about %d GB free under %s and there are %.1f GB; the node joins without it until there is",
|
||||
v, b.FreeGB, f.EnginesDir, f.EnginesFreeGB),
|
||||
"free space there, or point engines_dir in agent.yaml at a larger disk")
|
||||
return
|
||||
|
||||
@@ -151,8 +151,14 @@ func TestDoctorSaysWhetherVLLMCanBeInstalled(t *testing.T) {
|
||||
says string
|
||||
}{
|
||||
{"CPU node", Facts{Host: discovery.Host{Arch: "amd64"}, VLLMVersion: "0.30.0"}, Info, "has none"},
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Fail, "driver 580"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Fail, "9.0 GB"},
|
||||
// What vLLM cannot do keeps vLLM off the node, not the node out of
|
||||
// the cluster: it still serves llama.cpp.
|
||||
{"old driver", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15"}, VLLMVersion: "0.30.0"}, Warn, "driver 580"},
|
||||
{"volta", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0}, VLLMVersion: "0.30.0"},
|
||||
Warn, "before 0.26 is built for CUDA 12"},
|
||||
{"volta, a CUDA 12 vLLM", Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "550.54.15", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.25.1", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "(cu12) can be installed"},
|
||||
{"small disk", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 9}, Warn, "9.0 GB"},
|
||||
{"ready to install", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", EnginesFreeGB: 200}, OK, "can be installed"},
|
||||
{"installed", Facts{Host: gpu, VLLMVersion: "0.30.0", EnginesDir: "/opt/llmgw/engines", VLLMInstalled: true}, OK, "is installed"},
|
||||
}
|
||||
@@ -200,3 +206,15 @@ func TestDoctorSaysWhenTheCPUHidesAVX2(t *testing.T) {
|
||||
t.Errorf("ARM: %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnTheNodeIsNotChecked(t *testing.T) {
|
||||
// The field: a V100 node meant for llama.cpp, refused for a vLLM it was
|
||||
// never going to run.
|
||||
f := Facts{Host: discovery.Host{Arch: "amd64", NVIDIADriver: "580.65.06", MinComputeCapability: 7.0},
|
||||
VLLMVersion: "0.30.0", DisabledEngines: []string{"vllm"}}
|
||||
for _, fd := range Evaluate(f) {
|
||||
if fd.Area == "engines" {
|
||||
t.Errorf("a disabled vLLM was reported on: %s %q", fd.Level, fd.Message)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -18,7 +18,6 @@ import (
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -27,6 +26,8 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/doctor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
@@ -86,6 +87,15 @@ type Options struct {
|
||||
Releases string
|
||||
// IgnoreChecks goes on when doctor finds a problem.
|
||||
IgnoreChecks bool
|
||||
// DisableEngines are engines this node never installs or advertises,
|
||||
// written into its agent.yaml, and not checked by doctor: vLLM on a node
|
||||
// of cards it does not run on, meant for llama.cpp alone. Narrower than
|
||||
// IgnoreChecks, which also passes over everything else.
|
||||
DisableEngines []string
|
||||
// VLLMVersion is the vLLM the control plane has nodes install, for
|
||||
// doctor to check against; "none" when it has them install none, empty
|
||||
// for doctor's own default.
|
||||
VLLMVersion string
|
||||
// SSHWait is how long an unreachable machine is waited for.
|
||||
SSHWait time.Duration
|
||||
|
||||
@@ -390,6 +400,13 @@ func (r *Runner) check(ctx context.Context, res *Result) error {
|
||||
if r.Opts.IBRoot != "" {
|
||||
args += " --ib-root " + shellQuote(r.Opts.IBRoot)
|
||||
}
|
||||
switch {
|
||||
case slices.Contains(r.Opts.DisableEngines, "vllm"):
|
||||
args += " --vllm none"
|
||||
r.emit(StepCheck, Info, "vLLM is disabled on this node: it joins with llama.cpp alone", "")
|
||||
case r.Opts.VLLMVersion != "":
|
||||
args += " --vllm " + shellQuote(r.Opts.VLLMVersion)
|
||||
}
|
||||
out, err := r.Remote.Run(ctx, args, nil)
|
||||
var rep struct {
|
||||
Findings []doctor.Finding `json:"findings"`
|
||||
@@ -596,8 +613,11 @@ func AgentYAML(o Options, res *Result) string {
|
||||
fmt.Fprintf(&b, " ib_root: %q\n", o.IBRoot)
|
||||
}
|
||||
}
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" {
|
||||
if o.LlamaCppBin != "" || o.VLLMBin != "" || o.SGLangPython != "" || len(o.DisableEngines) > 0 {
|
||||
b.WriteString("engines:\n")
|
||||
if len(o.DisableEngines) > 0 {
|
||||
fmt.Fprintf(&b, " disabled: [%s]\n", strings.Join(o.DisableEngines, ", "))
|
||||
}
|
||||
if o.LlamaCppBin != "" {
|
||||
fmt.Fprintf(&b, " llamacpp:\n bin: %q\n", o.LlamaCppBin)
|
||||
}
|
||||
|
||||
@@ -280,3 +280,31 @@ func said(events []Event, level, s string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func TestAnEngineDisabledOnANodeIsWrittenAndNotChecked(t *testing.T) {
|
||||
m := newFake()
|
||||
r, events := runner(t, m, Options{Version: "dev", DisableEngines: []string{"vllm"}, VLLMVersion: "0.30.0"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if yaml := string(m.files["/etc/llmgw/agent.yaml"]); !strings.Contains(yaml, "engines:\n disabled: [vllm]") {
|
||||
t.Errorf("agent.yaml:\n%s", yaml)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm none") {
|
||||
t.Errorf("doctor was asked about a vLLM this node will never run: %v", m.scripts)
|
||||
}
|
||||
if !said(*events, Info, "joins with llama.cpp alone") {
|
||||
t.Error("the reduced engine set went unsaid")
|
||||
}
|
||||
|
||||
// Not disabled: doctor checks the vLLM the control plane installs, not
|
||||
// its own default.
|
||||
m = newFake()
|
||||
r, _ = runner(t, m, Options{Version: "dev", VLLMVersion: "0.25.1"})
|
||||
if _, err := r.Run(t.Context()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !m.ran("sudo llmgw doctor --json --vllm '0.25.1'") && !m.ran("sudo llmgw doctor --json --vllm 0.25.1") {
|
||||
t.Errorf("doctor was not told the control plane's vLLM: %v", m.scripts)
|
||||
}
|
||||
}
|
||||
@@ -82,9 +82,9 @@ the adapter came up: a number that grows is the one to look at.</p>
|
||||
</tr>{{end}}
|
||||
</table>
|
||||
<table class="rows compact gpu-list">
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
<tr><th>GPU</th><th>Name</th><th class="num">Compute</th><th class="num">NUMA</th><th>Nearest adapter</th><th>Multi-node NVLink</th></tr>
|
||||
{{range .Node.GPUs}}<tr>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{.Index}}</td><td>{{.Name}}</td><td class="num">{{.ComputeCapability}}</td><td class="num">{{.NUMA}}</td>
|
||||
<td>{{if .NearestNIC}}<code>{{.NearestNIC}}</code> <span class="note">{{.NICPath}}</span>{{else}}—{{end}}</td>
|
||||
<td>{{if .Domain}}<span class="state-ready">joined</span>{{if and .Bandwidth (ne .Bandwidth "Full")}} <span class="bad">{{.Bandwidth}} bandwidth</span>{{end}}{{else if .FabricState}}<span class="bad">{{.FabricState}}</span>{{else}}<span class="note">—</span>{{end}}</td>
|
||||
</tr>{{end}}
|
||||
|
||||
Reference in new issue
Block a user