CI / test (push) Failing after 1m31s
Disk was the one resource nothing reported. `statfs` was already being called on every model directory, but only to ask whether the filesystem was a network one — the free space it returns in the same struct was thrown away. So the only way to find out why a node was filling up was to ssh to it, which is what this system exists to avoid. Reported per filesystem rather than per configured directory. Free space belongs to the disk, not the path: two model directories on one disk are one pool of bytes, and counting them separately would let the control plane add 24 GB to 24 GB and place a 40 GB model on a node that has 24. Shared filesystems are summed separately from local ones and left out of a node's own total. Space on a filer belongs to everyone mounting it, so twenty nodes would otherwise each report the same free terabyte, and a scheduler would place twenty models into room for one. Nothing reported means nothing is known, everywhere: an agent too old to send storage and a node with no model directories both send nothing, and both must read as unknown rather than as a full disk. `ctl nodes` and the dashboard show a dash, the metric is absent rather than zero, and a missing directory — which is what an unmounted filer looks like — is skipped rather than reported as having no space. Shown as free out of total, because the first is useless without the second: 8 GB free is comfortable on a 10 TB filer and nearly full on a 10 GB root partition, and only the second stops a model being placed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
88 lines
3.1 KiB
Go
88 lines
3.1 KiB
Go
package discovery
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"log/slog"
|
|
)
|
|
|
|
// ErrNoNVML means this binary was built without the NVML backend.
|
|
//
|
|
// It is declared here rather than in the build-tagged file so callers — and
|
|
// tests — can refer to it in either build.
|
|
var ErrNoNVML = errors.New("built without the nvml tag; rebuild with -tags nvml for NVLink topology")
|
|
|
|
// Options configure a full inventory scan.
|
|
type Options struct {
|
|
// NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.
|
|
// Worth configuring on nodes where the driver lives outside the default
|
|
// paths, which containerized driver installs often do.
|
|
NvidiaSMI string `koanf:"nvidia_smi"`
|
|
// IBRoot is the sysfs InfiniBand directory.
|
|
IBRoot string `koanf:"ib_root"`
|
|
// PreferNVML uses the NVML backend when this binary has one. It is the only
|
|
// source of NVLink topology.
|
|
PreferNVML bool `koanf:"prefer_nvml"`
|
|
// SkipGPUs omits GPU discovery, for nodes that have none.
|
|
SkipGPUs bool `koanf:"skip_gpus"`
|
|
// ModelDirs are scanned for weights. Empty means do not look.
|
|
ModelDirs []string `koanf:"-"`
|
|
}
|
|
|
|
// Discover builds the node's inventory.
|
|
//
|
|
// When NVML is preferred but fails, it falls back to nvidia-smi and says so in
|
|
// Inventory.Source. The fallback is logged rather than silent: on a node that
|
|
// is supposed to have working NVML, a quiet downgrade hides a broken driver and
|
|
// costs the scheduler its NVLink topology.
|
|
func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, error) {
|
|
if log == nil {
|
|
log = slog.Default()
|
|
}
|
|
inv := Inventory{Source: "none"}
|
|
// First, because it has nothing to do with GPUs and every path below can
|
|
// return early. Read after them, a node with no nvidia-smi reported no
|
|
// memory at all: GPU discovery fails, Discover returns, and the fact that
|
|
// has nothing to do with cards goes with it. That is exactly the node whose
|
|
// memory matters most, since a CPU profile is what it will be asked to run.
|
|
inv.MemoryTotalMB = MemoryTotalMB()
|
|
// Also before the GPU paths, and for the same reason: which weights a node
|
|
// holds has nothing to do with its cards, and a node with none is exactly
|
|
// where a CPU profile will look for them.
|
|
inv.Weights = ScanWeights(opts.ModelDirs)
|
|
// And the disk they sit on. A node can hold a model today and be unable to
|
|
// take the next one, which the file list alone never shows.
|
|
inv.Storage = ScanStorage(opts.ModelDirs)
|
|
|
|
if !opts.SkipGPUs {
|
|
switch {
|
|
case opts.PreferNVML && nvmlAvailable:
|
|
gpus, err := DiscoverGPUsNVML(ctx)
|
|
if err == nil {
|
|
inv.GPUs, inv.Source = gpus, "nvml"
|
|
break
|
|
}
|
|
log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
|
|
"NVLink topology will be unavailable", "err", err)
|
|
fallthrough
|
|
default:
|
|
gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
|
|
if err != nil {
|
|
return inv, fmt.Errorf("gpu discovery: %w", err)
|
|
}
|
|
inv.GPUs, inv.Source = gpus, "nvidia-smi"
|
|
}
|
|
}
|
|
|
|
hcas, err := DiscoverIB(opts.IBRoot)
|
|
if err != nil {
|
|
return inv, fmt.Errorf("infiniband discovery: %w", err)
|
|
}
|
|
inv.HCAs = hcas
|
|
return inv, nil
|
|
}
|
|
|
|
// NVMLSupported reports whether this binary includes the NVML backend.
|
|
func NVMLSupported() bool { return nvmlAvailable }
|