Storage will be different at every installation — Lustre, NFS over RDMA, an image exported with SRP or NVMe-oF, local NVMe, an rsync — so llmgw observes it rather than managing it. An agent told which directories to look in reports what it can see; how the files got there is nobody's business here, and a node told nothing reports nothing and behaves exactly as every node did before this existed. That fact buys two things. Placement prefers a node that already holds the model over one that must first move tens of gigabytes, as a preference and never a filter: fetching them is a valid answer, and refusing to place would turn a slow start into no start at all. And ctl status shows what each fabric holds. Per fabric because fabrics are not connected to one another. Weights in ib-a are not weights in ib-b — the only path between them is the management network — so a model seeded into one fabric leaves the rest of the cluster unable to serve it without a slow copy, and nothing said so before now. One shared filer counts the same as twenty local copies, because "can this fabric serve it" is the question being asked. Three distinctions the tests hold to, because each is a way to get this quietly wrong. A node that was never asked to look must rank with one that looked and has nothing, or enabling the feature on a single node would drain every other. An empty directory is not weights, because a half-finished download looks exactly like one and would send placement to a node whose engine then fails to load. And local storage is never mistaken for shared: a filer read as local costs an unnecessary copy, while local read as shared would have the cluster believe a model is everywhere when it is on one node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
85 lines
3.0 KiB
Go
85 lines
3.0 KiB
Go
package discovery
|
|
|
|
import (
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"log/slog"
|
|
)
|
|
|
|
// ErrNoNVML means this binary was built without the NVML backend.
|
|
//
|
|
// It is declared here rather than in the build-tagged file so callers — and
|
|
// tests — can refer to it in either build.
|
|
var ErrNoNVML = errors.New("built without the nvml tag; rebuild with -tags nvml for NVLink topology")
|
|
|
|
// Options configure a full inventory scan.
|
|
type Options struct {
|
|
// NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.
|
|
// Worth configuring on nodes where the driver lives outside the default
|
|
// paths, which containerized driver installs often do.
|
|
NvidiaSMI string `koanf:"nvidia_smi"`
|
|
// IBRoot is the sysfs InfiniBand directory.
|
|
IBRoot string `koanf:"ib_root"`
|
|
// PreferNVML uses the NVML backend when this binary has one. It is the only
|
|
// source of NVLink topology.
|
|
PreferNVML bool `koanf:"prefer_nvml"`
|
|
// SkipGPUs omits GPU discovery, for nodes that have none.
|
|
SkipGPUs bool `koanf:"skip_gpus"`
|
|
// ModelDirs are scanned for weights. Empty means do not look.
|
|
ModelDirs []string `koanf:"-"`
|
|
}
|
|
|
|
// Discover builds the node's inventory.
|
|
//
|
|
// When NVML is preferred but fails, it falls back to nvidia-smi and says so in
|
|
// Inventory.Source. The fallback is logged rather than silent: on a node that
|
|
// is supposed to have working NVML, a quiet downgrade hides a broken driver and
|
|
// costs the scheduler its NVLink topology.
|
|
func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, error) {
|
|
if log == nil {
|
|
log = slog.Default()
|
|
}
|
|
inv := Inventory{Source: "none"}
|
|
// First, because it has nothing to do with GPUs and every path below can
|
|
// return early. Read after them, a node with no nvidia-smi reported no
|
|
// memory at all: GPU discovery fails, Discover returns, and the fact that
|
|
// has nothing to do with cards goes with it. That is exactly the node whose
|
|
// memory matters most, since a CPU profile is what it will be asked to run.
|
|
inv.MemoryTotalMB = MemoryTotalMB()
|
|
// Also before the GPU paths, and for the same reason: which weights a node
|
|
// holds has nothing to do with its cards, and a node with none is exactly
|
|
// where a CPU profile will look for them.
|
|
inv.Weights = ScanWeights(opts.ModelDirs)
|
|
|
|
if !opts.SkipGPUs {
|
|
switch {
|
|
case opts.PreferNVML && nvmlAvailable:
|
|
gpus, err := DiscoverGPUsNVML(ctx)
|
|
if err == nil {
|
|
inv.GPUs, inv.Source = gpus, "nvml"
|
|
break
|
|
}
|
|
log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
|
|
"NVLink topology will be unavailable", "err", err)
|
|
fallthrough
|
|
default:
|
|
gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
|
|
if err != nil {
|
|
return inv, fmt.Errorf("gpu discovery: %w", err)
|
|
}
|
|
inv.GPUs, inv.Source = gpus, "nvidia-smi"
|
|
}
|
|
}
|
|
|
|
hcas, err := DiscoverIB(opts.IBRoot)
|
|
if err != nil {
|
|
return inv, fmt.Errorf("infiniband discovery: %w", err)
|
|
}
|
|
inv.HCAs = hcas
|
|
return inv, nil
|
|
}
|
|
|
|
// NVMLSupported reports whether this binary includes the NVML backend.
|
|
func NVMLSupported() bool { return nvmlAvailable }
|