Files
llmgw/internal/agent/discovery/discover.go
T
Ilker ManapandClaude Opus 5.5 20deb22dfa
CI / test (push) Successful in 1m47s
Let a node describe itself instead of being described by hand
Every agent file on the cluster named its own node, its own address, the
path to llama-server and two weight directories — different on every node,
kept true by hand, and not true: node05 carried "ram: 32g" beside a measured
31 GB, looked for weights in an empty directory while fetching them into
another, and the other four never looked in theirs at all, so four nodes
reported no disk. Growing a node's RAM would have gone unnoticed, because
memory was read once, at registration.

The agent now finds these out. Its id is the hostname, which is also the
certificate's common name. Its address is the one on the route to the
control plane, which the kernel picks for a UDP dial without sending
anything. llama-server is looked for on PATH and then where release archives
are unpacked. A missing nvidia-smi is a node without NVIDIA cards rather
than a failed discovery — which also cut discovery short before InfiniBand
on every CPU node not told to skip GPUs. Fetched weights land in
/var/lib/llmgw-weights by default, and wherever they land is always looked
in.

Memory is read on every heartbeat, and installed engines are probed again
each minute and sent when they change, so a VM given more RAM or a node
given vLLM becomes a candidate without anyone restarting its agent. The
control plane logs both changes, since "the node now qualifies" is the
answer to why a model placed an hour after it was asked for.

What is written still wins. The agent file shrinks to the control plane's
addresses and the TLS files, the same on every node; onboarding writes only
the overrides it was given. What cannot be found stays configuration: which
fabric an adapter is on, because OpenSM's default prefix is the same on all
of them.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:13:42 +02:00

89 lines
3.2 KiB
Go

package discovery
import (
"context"
"errors"
"fmt"
"log/slog"
)
// ErrNoNVML means this binary was built without the NVML backend.
//
// It is declared here rather than in the build-tagged file so callers — and
// tests — can refer to it in either build.
var ErrNoNVML = errors.New("built without the nvml tag; rebuild with -tags nvml for NVLink topology")
// Options configure a full inventory scan.
type Options struct {
// NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.
// Worth configuring on nodes where the driver lives outside the default
// paths, which containerized driver installs often do.
NvidiaSMI string `koanf:"nvidia_smi"`
// IBRoot is the sysfs InfiniBand directory.
IBRoot string `koanf:"ib_root"`
// PreferNVML uses the NVML backend when this binary has one. It is the only
// source of NVLink topology.
PreferNVML bool `koanf:"prefer_nvml"`
// SkipGPUs omits GPU discovery. A node without nvidia-smi is already
// found to have none, so this is for one whose driver is broken.
SkipGPUs bool `koanf:"skip_gpus"`
// ModelDirs are scanned for weights. Empty means do not look.
ModelDirs []string `koanf:"-"`
}
// Discover builds the node's inventory.
//
// When NVML is preferred but fails, it falls back to nvidia-smi and says so in
// Inventory.Source. The fallback is logged rather than silent: on a node that
// is supposed to have working NVML, a quiet downgrade hides a broken driver and
// costs the scheduler its NVLink topology.
func Discover(ctx context.Context, opts Options, log *slog.Logger) (Inventory, error) {
if log == nil {
log = slog.Default()
}
inv := Inventory{Source: "none"}
// First, because it has nothing to do with GPUs and every path below can
// return early. Read after them, a node with no nvidia-smi reported no
// memory at all: GPU discovery fails, Discover returns, and the fact that
// has nothing to do with cards goes with it. That is exactly the node whose
// memory matters most, since a CPU profile is what it will be asked to run.
inv.MemoryTotalMB = MemoryTotalMB()
// Also before the GPU paths, and for the same reason: which weights a node
// holds has nothing to do with its cards, and a node with none is exactly
// where a CPU profile will look for them.
inv.Weights = ScanWeights(opts.ModelDirs)
// And the disk they sit on. A node can hold a model today and be unable to
// take the next one, which the file list alone never shows.
inv.Storage = ScanStorage(opts.ModelDirs)
if !opts.SkipGPUs {
switch {
case opts.PreferNVML && nvmlAvailable:
gpus, err := DiscoverGPUsNVML(ctx)
if err == nil {
inv.GPUs, inv.Source = gpus, "nvml"
break
}
log.Warn("NVML discovery failed, falling back to nvidia-smi; "+
"NVLink topology will be unavailable", "err", err)
fallthrough
default:
gpus, err := DiscoverGPUsSMI(ctx, opts.NvidiaSMI)
if err != nil {
return inv, fmt.Errorf("gpu discovery: %w", err)
}
inv.GPUs, inv.Source = gpus, "nvidia-smi"
}
}
hcas, err := DiscoverIB(opts.IBRoot)
if err != nil {
return inv, fmt.Errorf("infiniband discovery: %w", err)
}
inv.HCAs = hcas
return inv, nil
}
// NVMLSupported reports whether this binary includes the NVML backend.
func NVMLSupported() bool { return nvmlAvailable }