Merge C22 MIG
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
330a421118
39 files changed
+2587
-327
No files matched your search
+302
-182
File diff suppressed because it is too large.
Load diff
@@ -252,6 +252,25 @@ message GPU {
|
||||
string cluster_uuid = 19;
|
||||
string clique_id = 20;
|
||||
string fabric_bandwidth = 21;
|
||||
// mig is set when this is a MIG instance rather than a whole card. index
|
||||
// is then 1000 + 100 × card + device, uuid the instance's (MIG-…), and
|
||||
// memory the instance's share; utilization_percent stays zero, because no
|
||||
// driver measures it per instance.
|
||||
MIGDevice mig = 22;
|
||||
}
|
||||
|
||||
// MIGDevice is which card a MIG instance is carved from, and how.
|
||||
message MIGDevice {
|
||||
int32 parent_index = 1;
|
||||
string parent_uuid = 2;
|
||||
// parent_memory_total_mb is the whole card's: the class is named from it.
|
||||
int32 parent_memory_total_mb = 3;
|
||||
// device is the instance's number on its card, as nvidia-smi -L lists it.
|
||||
int32 device = 4;
|
||||
// profile is NVIDIA's name for the slice, like 3g.40gb.
|
||||
string profile = 5;
|
||||
int32 gpu_instance = 6;
|
||||
int32 compute_instance = 7;
|
||||
}
|
||||
|
||||
message FabricAttachment {
|
||||
|
||||
@@ -428,6 +428,71 @@ sudo llmgw doctor
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### MIG, if you want to cut a card into pieces
|
||||
|
||||
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
|
||||
GPU) instances, each with its own memory, its own share of the card's compute
|
||||
and its own UUID. llmgw places a model on an instance as it would on a card,
|
||||
which makes MIG the way to run several small models on one large card without
|
||||
them competing for it.
|
||||
|
||||
**llmgw does not create the instances.** How a card is cut is your decision,
|
||||
made once with `nvidia-smi` on the node; the agent reads what is there when
|
||||
it starts:
|
||||
|
||||
```
|
||||
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
|
||||
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
|
||||
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
|
||||
nvidia-smi -L # the instances, with their MIG- UUIDs
|
||||
sudo systemctl restart llmgw-agent # so it registers the new devices
|
||||
```
|
||||
|
||||
`-C` matters: a GPU instance with no compute instance inside it is not a
|
||||
device yet. Turning MIG on takes effect at the card's next reset
|
||||
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
|
||||
do not survive a reboot unless something recreates them — NVIDIA's
|
||||
`mig-parted` or a systemd unit of your own.
|
||||
|
||||
**Classes.** A card in MIG mode is reported as its instances instead of as
|
||||
the whole card, and each instance's class is the card's class and the
|
||||
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
|
||||
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
|
||||
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
|
||||
|
||||
```yaml
|
||||
profiles:
|
||||
- name: mig
|
||||
engine: vllm
|
||||
gpu_class: a100-80g-mig-3g.40gb
|
||||
```
|
||||
|
||||
**One instance a replica.** A process can open only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer access between them, so a replica on
|
||||
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
|
||||
MIG class asking for more is refused when the model is added, with that
|
||||
reason. A model too large for one instance wants a larger profile or a whole
|
||||
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
|
||||
instance is offered for a model it holds, and never several together.
|
||||
|
||||
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
|
||||
device 1002 (1000 + 100 × card + instance), and every view shows it as
|
||||
`0/mig2`. The engine is given the instance by its UUID —
|
||||
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
|
||||
build. llama.cpp's Vulkan build cannot see MIG instances at all.
|
||||
|
||||
**What is measured.** Each instance's memory is its own; temperature, power
|
||||
and wear are its card's, shown on each of its instances. Utilisation is not
|
||||
reported for an instance, because no driver measures it per instance: the
|
||||
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
|
||||
published for it rather than a zero.
|
||||
|
||||
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
|
||||
card in MIG mode with none created (nothing can run on it: CUDA will not open
|
||||
the whole card while MIG is on), of GPU instances with no compute instance,
|
||||
and of a mode change waiting for a reset. Such a card is also kept out of
|
||||
placement, with the reason, in `llmgw ctl nodes`.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
```
|
||||
|
||||
+28
-5
@@ -1202,18 +1202,20 @@ const (
|
||||
//
|
||||
// The interface NCCL bootstraps over is this node's, not the cluster's: the
|
||||
// control plane sends one name for every node, and a node whose management
|
||||
// network is on ens3 rather than eth0 would have NCCL bind nothing. And on
|
||||
// RoCE, which GID entry routes is a property of these adapters.
|
||||
// network is on ens3 rather than eth0 would have NCCL bind nothing. On RoCE,
|
||||
// which GID entry routes is a property of these adapters. And a MIG instance
|
||||
// is opened by its UUID, which only this node knows.
|
||||
func (a *Agent) localizeSpec(spec *engines.ShardSpec) {
|
||||
if a.cfg.MgmtInterface != "" {
|
||||
spec.MgmtInterface = a.cfg.MgmtInterface
|
||||
}
|
||||
a.mu.Lock()
|
||||
hcas, gpus := a.inv.HCAs, a.inv.GPUs
|
||||
a.mu.Unlock()
|
||||
spec.GPUUUIDs = migUUIDs(gpus, spec.GPUIndices)
|
||||
if len(spec.HCAs) == 0 {
|
||||
return
|
||||
}
|
||||
a.mu.Lock()
|
||||
hcas := a.inv.HCAs
|
||||
a.mu.Unlock()
|
||||
want := map[string]bool{}
|
||||
for _, h := range spec.HCAs {
|
||||
want[h] = true
|
||||
@@ -1232,6 +1234,27 @@ func (a *Agent) localizeSpec(spec *engines.ShardSpec) {
|
||||
}
|
||||
}
|
||||
|
||||
// migUUIDs names a shard's devices by UUID when one of them is a MIG
|
||||
// instance, and is nil otherwise: whole cards keep being given by index, as
|
||||
// they always were. A device this node does not have is left to fail where
|
||||
// it always has, in the engine, rather than guessed at here.
|
||||
func migUUIDs(gpus []discovery.GPU, indices []int) []string {
|
||||
var out []string
|
||||
mig := false
|
||||
for _, i := range indices {
|
||||
k := slices.IndexFunc(gpus, func(g discovery.GPU) bool { return g.Index == i })
|
||||
if k < 0 || gpus[k].UUID == "" {
|
||||
return nil
|
||||
}
|
||||
mig = mig || gpus[k].IsMIG()
|
||||
out = append(out, gpus[k].UUID)
|
||||
}
|
||||
if !mig {
|
||||
return nil
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ResolveManagement is the address and interface this node's agent would use,
|
||||
// found the way normalize finds them, for a tool that reports them.
|
||||
func ResolveManagement(c Config) (addr, iface string) {
|
||||
|
||||
@@ -47,6 +47,11 @@ type GPU struct {
|
||||
Fabric *GPUFabric `json:"fabric,omitempty"`
|
||||
// Health carries anything that should keep the GPU out of placement.
|
||||
Health GPUHealth `json:"health"`
|
||||
// MIG is set when this is a MIG instance rather than a whole card: its
|
||||
// Index is then MIGIndex's, its UUID the instance's (MIG-…), and its
|
||||
// memory the instance's share. Utilisation stays unreported, because no
|
||||
// driver measures it per instance.
|
||||
MIG *MIGDevice `json:"mig,omitempty"`
|
||||
}
|
||||
|
||||
// GPUHealth records conditions that make a GPU unsafe to schedule onto.
|
||||
@@ -94,8 +99,30 @@ func (g GPU) Class() string {
|
||||
}
|
||||
|
||||
// Round to the nearest GiB: vendors report 81559 MiB for an "80GB" card.
|
||||
gb := int(math.Round(float64(g.MemoryTotalMB) / 1024))
|
||||
return fmt.Sprintf("%s-%dg", model, gb)
|
||||
// A MIG instance is named from its card's memory and then its profile,
|
||||
// so two instances are one class only when both the card and the slice
|
||||
// are the same.
|
||||
mem := g.MemoryTotalMB
|
||||
if g.MIG != nil {
|
||||
mem = g.MIG.ParentMemoryTotalMB
|
||||
}
|
||||
gb := int(math.Round(float64(mem) / 1024))
|
||||
class := fmt.Sprintf("%s-%dg", model, gb)
|
||||
if g.MIG != nil {
|
||||
return migClass(class, g.MIG.Profile)
|
||||
}
|
||||
return class
|
||||
}
|
||||
|
||||
// normalizeBusID makes bus ids comparable: NVML reports them uppercase from one
|
||||
// call and lowercase from another, and sometimes with a shortened domain.
|
||||
func normalizeBusID(s string) string {
|
||||
s = strings.ToLower(strings.TrimSpace(s))
|
||||
// "0000:1b:00.0" and "00000000:1B:00.0" are the same device.
|
||||
if i := strings.Index(s, ":"); i > 0 {
|
||||
s = strings.TrimLeft(s[:i], "0") + s[i:]
|
||||
}
|
||||
return strings.TrimPrefix(s, ":")
|
||||
}
|
||||
|
||||
// MarshalJSON adds the derived class, so a consumer of the JSON does not have
|
||||
|
||||
@@ -14,7 +14,6 @@ package discovery
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/NVIDIA/go-nvml/pkg/nvml"
|
||||
)
|
||||
@@ -83,11 +82,18 @@ func DiscoverGPUsNVML(_ context.Context) ([]GPU, error) {
|
||||
}
|
||||
|
||||
assessHealth(&g)
|
||||
gpus = append(gpus, g)
|
||||
// A card in MIG mode is its instances, not itself: CUDA will not
|
||||
// open the whole card while MIG is on.
|
||||
devs, _ := migFromNVML(g, nvmlCard{dev})
|
||||
gpus = append(gpus, devs...)
|
||||
}
|
||||
|
||||
// A second pass, because a link's remote end may be a GPU enumerated later.
|
||||
// A second pass, because a link's remote end may be a GPU enumerated
|
||||
// later. MIG instances have no NVLink of their own to use.
|
||||
for i := range gpus {
|
||||
if gpus[i].IsMIG() {
|
||||
continue
|
||||
}
|
||||
dev, ret := nvml.DeviceGetHandleByIndex(gpus[i].Index)
|
||||
if ret != nvml.SUCCESS {
|
||||
continue
|
||||
@@ -136,13 +142,73 @@ func busIDString(b []int8) string {
|
||||
return string(buf)
|
||||
}
|
||||
|
||||
// normalizeBusID makes bus ids comparable: NVML reports them uppercase from one
|
||||
// call and lowercase from another, and sometimes with a shortened domain.
|
||||
func normalizeBusID(s string) string {
|
||||
s = strings.ToLower(strings.TrimSpace(s))
|
||||
// "0000:1b:00.0" and "00000000:1B:00.0" are the same device.
|
||||
if i := strings.Index(s, ":"); i > 0 {
|
||||
s = strings.TrimLeft(s[:i], "0") + s[i:]
|
||||
// nvmlCard adapts an NVML device to what MIG discovery reads of it.
|
||||
type nvmlCard struct{ dev nvml.Device }
|
||||
|
||||
func (c nvmlCard) MIGMode() (current, pending, ok bool) {
|
||||
cur, pend, ret := c.dev.GetMigMode()
|
||||
if ret != nvml.SUCCESS {
|
||||
// NOT_SUPPORTED on a card without MIG, which is most of them.
|
||||
return false, false, false
|
||||
}
|
||||
return strings.TrimPrefix(s, ":")
|
||||
return cur == nvml.DEVICE_MIG_ENABLE, pend == nvml.DEVICE_MIG_ENABLE, true
|
||||
}
|
||||
|
||||
func (c nvmlCard) MaxMIGDevices() int {
|
||||
n, ret := c.dev.GetMaxMigDeviceCount()
|
||||
if ret != nvml.SUCCESS {
|
||||
return 0
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func (c nvmlCard) MIGDevice(i int) (nvmlMIGInstance, bool) {
|
||||
h, ret := c.dev.GetMigDeviceHandleByIndex(i)
|
||||
if ret != nvml.SUCCESS {
|
||||
// NOT_FOUND for an index with no instance: the count is the most a
|
||||
// card can hold, not how many it has.
|
||||
return nil, false
|
||||
}
|
||||
return nvmlMIG{h}, true
|
||||
}
|
||||
|
||||
// nvmlMIG adapts a MIG device handle.
|
||||
type nvmlMIG struct{ dev nvml.Device }
|
||||
|
||||
func (m nvmlMIG) UUID() string {
|
||||
u, _ := m.dev.GetUUID()
|
||||
return u
|
||||
}
|
||||
|
||||
func (m nvmlMIG) Name() string {
|
||||
n, _ := m.dev.GetName()
|
||||
return n
|
||||
}
|
||||
|
||||
func (m nvmlMIG) Memory() (totalMB, usedMB int, ok bool) {
|
||||
mem, ret := m.dev.GetMemoryInfo()
|
||||
if ret != nvml.SUCCESS {
|
||||
return 0, 0, false
|
||||
}
|
||||
return int(mem.Total / (1 << 20)), int(mem.Used / (1 << 20)), true
|
||||
}
|
||||
|
||||
func (m nvmlMIG) Instances() (gpuInstance, computeInstance int) {
|
||||
gi, ret := m.dev.GetGpuInstanceId()
|
||||
if ret != nvml.SUCCESS {
|
||||
gi = -1
|
||||
}
|
||||
ci, ret := m.dev.GetComputeInstanceId()
|
||||
if ret != nvml.SUCCESS {
|
||||
ci = -1
|
||||
}
|
||||
return gi, ci
|
||||
}
|
||||
|
||||
func (m nvmlMIG) Attributes() (slices, memoryMB int, ok bool) {
|
||||
a, ret := m.dev.GetAttributes()
|
||||
if ret != nvml.SUCCESS {
|
||||
return 0, 0, false
|
||||
}
|
||||
return int(a.GpuInstanceSliceCount), int(a.MemorySizeMB), true
|
||||
}
|
||||
@@ -60,7 +60,33 @@ func DiscoverGPUsSMI(ctx context.Context, binary string) ([]GPU, error) {
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s: %w", binary, err)
|
||||
}
|
||||
return parseSMI(string(out))
|
||||
cards, err := parseSMI(string(out))
|
||||
if err != nil || len(cards) == 0 {
|
||||
return cards, err
|
||||
}
|
||||
// The query lists whole cards, MIG mode or not. A card in MIG mode is
|
||||
// replaced by its instances, which only -L names and only the full
|
||||
// report sizes; asked only when some card could be in MIG mode, since
|
||||
// the report is the slowest thing nvidia-smi does.
|
||||
if !anyMIGCapable(cards) {
|
||||
return cards, nil
|
||||
}
|
||||
listed, report := readMIG(ctx, binary)
|
||||
return expandMIG(cards, listed, report), nil
|
||||
}
|
||||
|
||||
// anyMIGCapable reports whether any card could be in MIG mode: MIG began
|
||||
// with compute capability 8.0 (the A100). The consumer cards of the same
|
||||
// generations have none, but telling them apart by name is guesswork, and
|
||||
// asking one costs only two more nvidia-smi runs at registration.
|
||||
func anyMIGCapable(cards []GPU) bool {
|
||||
for _, c := range cards {
|
||||
major, _, _ := strings.Cut(c.ComputeCapability, ".")
|
||||
if n, err := strconv.Atoi(major); err == nil && n >= 8 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func parseSMI(out string) ([]GPU, error) {
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
@@ -34,9 +35,16 @@ func GPUProcesses(ctx context.Context, binary string) (map[int][]int, error) {
|
||||
return parseGPUProcesses(out)
|
||||
}
|
||||
|
||||
// GPUMemory is one card's memory as nvidia-smi reports it at a moment: what
|
||||
// GPUMemory is one device's memory as nvidia-smi reports it at a moment: what
|
||||
// is allocated on it, and which processes hold how much of that.
|
||||
//
|
||||
// A device is what discovery reports: a whole card, or each MIG instance of a
|
||||
// card in MIG mode, in the same order, so the readings line up with the
|
||||
// inventory one for one.
|
||||
type GPUMemory struct {
|
||||
// Index is the device's number as placement knows it: the card's
|
||||
// nvidia-smi index, or MIGIndex's for an instance.
|
||||
Index int
|
||||
UsedMB int
|
||||
Procs []GPUProcess
|
||||
}
|
||||
@@ -66,48 +74,62 @@ func GPUMemoryByProcess(ctx context.Context, binary string) ([]GPUMemory, error)
|
||||
return parseGPUMemory(out)
|
||||
}
|
||||
|
||||
// smiLog is the part of nvidia-smi's XML report this reads: each GPU, in
|
||||
// index order, its memory and the processes on it.
|
||||
type smiLog struct {
|
||||
GPUs []struct {
|
||||
Used string `xml:"fb_memory_usage>used"`
|
||||
Processes []struct {
|
||||
PID string `xml:"pid"`
|
||||
Type string `xml:"type"`
|
||||
UsedMB string `xml:"used_memory"`
|
||||
} `xml:"processes>process_info"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
|
||||
// parseGPUProcesses reads the report. GPUs are listed in index order, which
|
||||
// is nvidia-smi's own numbering — the one placement assigns by.
|
||||
// parseGPUProcesses reads the report, by device as placement numbers them:
|
||||
// nvidia-smi's index for a whole card, MIGIndex's for a MIG instance.
|
||||
func parseGPUProcesses(report []byte) (map[int][]int, error) {
|
||||
mem, err := parseGPUMemory(report)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := map[int][]int{}
|
||||
for i, g := range mem {
|
||||
for _, g := range mem {
|
||||
for _, p := range g.Procs {
|
||||
out[i] = append(out[i], p.PID)
|
||||
out[g.Index] = append(out[g.Index], p.PID)
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseGPUMemory reads the report into each GPU's memory and its holders.
|
||||
// parseGPUMemory reads the report into each device's memory and its holders.
|
||||
//
|
||||
// Cards are listed in index order, which is nvidia-smi's own numbering. A
|
||||
// card in MIG mode is read as its instances, each with its own memory and
|
||||
// the processes nvidia-smi attributes to its GPU and compute instance: a
|
||||
// process on one instance is not on its neighbours, and the card's total
|
||||
// would have charged every instance with all of them.
|
||||
func parseGPUMemory(report []byte) ([]GPUMemory, error) {
|
||||
var log smiLog
|
||||
var log migLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := make([]GPUMemory, len(log.GPUs))
|
||||
out := make([]GPUMemory, 0, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
out[i].UsedMB = mib(g.Used)
|
||||
for _, p := range g.Processes {
|
||||
if pid, err := strconv.Atoi(strings.TrimSpace(p.PID)); err == nil {
|
||||
out[i].Procs = append(out[i].Procs, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
|
||||
procs := func(gi, ci int) []GPUProcess {
|
||||
var ps []GPUProcess
|
||||
for _, p := range g.Processes {
|
||||
pid, err := strconv.Atoi(strings.TrimSpace(p.PID))
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if gi >= 0 && (atoiOr(p.GPUInstance, -1) != gi || atoiOr(p.ComputeInstance, -1) != ci) {
|
||||
continue
|
||||
}
|
||||
ps = append(ps, GPUProcess{PID: pid, UsedMB: mib(p.UsedMB)})
|
||||
}
|
||||
return ps
|
||||
}
|
||||
if !enabled(g.Current) || len(g.Devices) == 0 {
|
||||
out = append(out, GPUMemory{Index: i, UsedMB: mib(g.Used), Procs: procs(-1, -1)})
|
||||
continue
|
||||
}
|
||||
devs := slices.Clone(g.Devices)
|
||||
slices.SortFunc(devs, func(a, b migLogDevice) int { return atoiOr(a.Index, -1) - atoiOr(b.Index, -1) })
|
||||
for _, d := range devs {
|
||||
out = append(out, GPUMemory{
|
||||
Index: MIGIndex(i, atoiOr(d.Index, 0)),
|
||||
UsedMB: mib(d.Used),
|
||||
Procs: procs(atoiOr(d.GPUInstance, -1), atoiOr(d.ComputeInstance, -1)),
|
||||
})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
|
||||
@@ -57,8 +57,8 @@ func TestGPUMemoryIsReadPerProcess(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []GPUMemory{
|
||||
{UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
|
||||
{},
|
||||
{Index: 0, UsedMB: 14514, Procs: []GPUProcess{{PID: 71315, UsedMB: 4932}, {PID: 4100, UsedMB: 9500}}},
|
||||
{Index: 1},
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("got %+v, want %+v", got, want)
|
||||
|
||||
@@ -0,0 +1,560 @@
|
||||
package discovery
|
||||
|
||||
// MIG — Multi-Instance GPU — carves one A100, H100 or later card into up to
|
||||
// seven instances, each with its own memory, its own share of the
|
||||
// multiprocessors and its own UUID. To CUDA an instance is a device of its
|
||||
// own, and to placement it is one too: a card in MIG mode is reported as its
|
||||
// instances instead of as the whole card, each a GPU of a class named from
|
||||
// the card's and the instance's profile ("a100-80g-mig-3g.40gb").
|
||||
//
|
||||
// llmgw does not create instances. Which split a card gets is an operator's
|
||||
// decision, made with nvidia-smi mig; discovery reads what is there.
|
||||
//
|
||||
// Two facts about MIG shape everything below. A process is given one MIG
|
||||
// device: CUDA enumerates a single instance however many are made visible.
|
||||
// And instances have no NVLink or peer-to-peer access between them. A replica
|
||||
// on MIG therefore runs on exactly one instance, never a tensor-parallel group
|
||||
// of them, and the catalog refuses a profile that asks for more.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"math"
|
||||
"os/exec"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// MIGDevice is what a MIG instance carries beyond any GPU's fields: which
|
||||
// card it is carved from, and how.
|
||||
type MIGDevice struct {
|
||||
// ParentIndex is the card's nvidia-smi index, which is what its sensors,
|
||||
// topology and wear are reported by.
|
||||
ParentIndex int `json:"parent_index"`
|
||||
ParentUUID string `json:"parent_uuid,omitempty"`
|
||||
// ParentMemoryTotalMB is the whole card's memory: the class is named from
|
||||
// the card, so an instance of an 80 GB card is not mistaken for one of a
|
||||
// 40 GB card that happens to have the same profile name.
|
||||
ParentMemoryTotalMB int `json:"parent_memory_total_mb"`
|
||||
// Device is the instance's number on its card, as nvidia-smi -L lists it.
|
||||
Device int `json:"device"`
|
||||
// Profile is NVIDIA's name for the slice, like "3g.40gb": three of the
|
||||
// card's seven compute slices and forty gigabytes of its memory.
|
||||
Profile string `json:"profile"`
|
||||
// GPUInstance and ComputeInstance are the instance's ids in nvidia-smi
|
||||
// mig -lgi and -lci, which is how nvidia-smi attributes processes to it.
|
||||
GPUInstance int `json:"gpu_instance"`
|
||||
ComputeInstance int `json:"compute_instance"`
|
||||
}
|
||||
|
||||
// migIndexBase starts the numbers MIG instances are given.
|
||||
//
|
||||
// Placement, the agent protocol and every running shard name a node's devices
|
||||
// by an integer, which for a whole card is its nvidia-smi index. An instance
|
||||
// has none of its own, so it is given 1000 + 100 × card + device: card 0's
|
||||
// instance 2 is 1002, card 3's instance 0 is 1300. Readable, stable while the
|
||||
// instances are, and never one a whole card could have.
|
||||
const migIndexBase = 1000
|
||||
|
||||
// MIGIndex is the device number placement knows a MIG instance by.
|
||||
func MIGIndex(card, device int) int { return migIndexBase + 100*card + device }
|
||||
|
||||
// IsMIGIndex reports whether a device number names a MIG instance.
|
||||
func IsMIGIndex(i int) bool { return i >= migIndexBase }
|
||||
|
||||
// DeviceLabel is how a device number is shown to people: "3" for a whole
|
||||
// card, "0/mig2" for card 0's MIG instance 2.
|
||||
func DeviceLabel(i int) string {
|
||||
if !IsMIGIndex(i) {
|
||||
return strconv.Itoa(i)
|
||||
}
|
||||
i -= migIndexBase
|
||||
return fmt.Sprintf("%d/mig%d", i/100, i%100)
|
||||
}
|
||||
|
||||
// DeviceLabels is DeviceLabel for each of a list.
|
||||
func DeviceLabels(is []int) []string {
|
||||
out := make([]string, len(is))
|
||||
for k, i := range is {
|
||||
out[k] = DeviceLabel(i)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// IsMIG reports whether this GPU is a MIG instance rather than a whole card.
|
||||
func (g GPU) IsMIG() bool { return g.MIG != nil }
|
||||
|
||||
// CardIndex is the nvidia-smi index of the physical card this GPU is on: its
|
||||
// own index for a whole card, its parent's for a MIG instance.
|
||||
func (g GPU) CardIndex() int {
|
||||
if g.MIG != nil {
|
||||
return g.MIG.ParentIndex
|
||||
}
|
||||
return g.Index
|
||||
}
|
||||
|
||||
// migClassMarker joins a card's class to an instance's profile in a MIG
|
||||
// class: "a100-80g" + "-mig-" + "3g.40gb".
|
||||
const migClassMarker = "-mig-"
|
||||
|
||||
// IsMIGClass reports whether a GPU class names MIG instances.
|
||||
func IsMIGClass(class string) bool { return strings.Contains(class, migClassMarker) }
|
||||
|
||||
// migProfileClean keeps a profile name to what a class may hold: "1g.10gb+me"
|
||||
// becomes "1g.10gb-me".
|
||||
var migProfileClean = regexp.MustCompile(`[^a-z0-9.]+`)
|
||||
|
||||
// migClass is the class of a MIG instance: the card's class and the profile.
|
||||
func migClass(parentClass, profile string) string {
|
||||
p := migProfileClean.ReplaceAllString(strings.ToLower(strings.TrimSpace(profile)), "-")
|
||||
return parentClass + migClassMarker + strings.Trim(p, "-")
|
||||
}
|
||||
|
||||
// MIGProfileMemoryGB reads the memory out of a MIG class or profile:
|
||||
// 40 from "a100-80g-mig-3g.40gb" or "3g.40gb". Zero when there is none.
|
||||
func MIGProfileMemoryGB(s string) int {
|
||||
if i := strings.LastIndex(s, migClassMarker); i >= 0 {
|
||||
s = s[i+len(migClassMarker):]
|
||||
}
|
||||
m := migProfileMemory.FindStringSubmatch(strings.ToLower(s))
|
||||
if m == nil {
|
||||
return 0
|
||||
}
|
||||
gb, _ := strconv.Atoi(m[1])
|
||||
return gb
|
||||
}
|
||||
|
||||
var migProfileMemory = regexp.MustCompile(`^\d+g\.(\d+)gb`)
|
||||
|
||||
// --- nvidia-smi -L ---
|
||||
|
||||
// listedMIG is one MIG instance as nvidia-smi -L lists it.
|
||||
type listedMIG struct {
|
||||
Device int
|
||||
Profile string
|
||||
UUID string
|
||||
}
|
||||
|
||||
var (
|
||||
// "GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-5d5ba0d6-...)"
|
||||
smiListGPU = regexp.MustCompile(`^GPU (\d+):`)
|
||||
// " MIG 3g.40gb Device 0: (UUID: MIG-c6d4f1ef-...)". Drivers before
|
||||
// R470 name the instance MIG-GPU-<card uuid>/<gi>/<ci>, which CUDA of that
|
||||
// era takes in the same place, so the UUID is read as whatever it is.
|
||||
smiListMIG = regexp.MustCompile(`^\s+MIG\s+(\S+)\s+Device\s+(\d+):\s*\(UUID:\s*([^)\s]+)\)`)
|
||||
)
|
||||
|
||||
// parseSMIList reads nvidia-smi -L into each card's MIG instances, by the
|
||||
// card's index, in device order. A card with none is absent.
|
||||
func parseSMIList(out string) map[int][]listedMIG {
|
||||
res := map[int][]listedMIG{}
|
||||
card := -1
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
if m := smiListGPU.FindStringSubmatch(line); m != nil {
|
||||
card, _ = strconv.Atoi(m[1])
|
||||
continue
|
||||
}
|
||||
m := smiListMIG.FindStringSubmatch(line)
|
||||
if m == nil || card < 0 {
|
||||
continue
|
||||
}
|
||||
dev, _ := strconv.Atoi(m[2])
|
||||
res[card] = append(res[card], listedMIG{Device: dev, Profile: m[1], UUID: m[3]})
|
||||
}
|
||||
for c := range res {
|
||||
slices.SortFunc(res[c], func(a, b listedMIG) int { return a.Device - b.Device })
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// --- nvidia-smi -q -x ---
|
||||
|
||||
// migLog is the part of nvidia-smi's XML report MIG is read from: each card's
|
||||
// mode, and each instance's ids and memory.
|
||||
type migLog struct {
|
||||
GPUs []migLogGPU `xml:"gpu"`
|
||||
}
|
||||
|
||||
type migLogGPU struct {
|
||||
ID string `xml:"id,attr"`
|
||||
Current string `xml:"mig_mode>current_mig"`
|
||||
Pending string `xml:"mig_mode>pending_mig"`
|
||||
Devices []migLogDevice `xml:"mig_devices>mig_device"`
|
||||
// Processes carry the GPU and compute instance they run in on a card
|
||||
// in MIG mode, and "N/A" for both otherwise.
|
||||
Processes []struct {
|
||||
GPUInstance string `xml:"gpu_instance_id"`
|
||||
ComputeInstance string `xml:"compute_instance_id"`
|
||||
PID string `xml:"pid"`
|
||||
UsedMB string `xml:"used_memory"`
|
||||
} `xml:"processes>process_info"`
|
||||
Used string `xml:"fb_memory_usage>used"`
|
||||
}
|
||||
|
||||
type migLogDevice struct {
|
||||
Index string `xml:"index"`
|
||||
GPUInstance string `xml:"gpu_instance_id"`
|
||||
ComputeInstance string `xml:"compute_instance_id"`
|
||||
Total string `xml:"fb_memory_usage>total"`
|
||||
Used string `xml:"fb_memory_usage>used"`
|
||||
}
|
||||
|
||||
// migState is one card's MIG mode and instances from the report.
|
||||
type migState struct {
|
||||
BusID string
|
||||
Enabled bool
|
||||
PendingEnabled bool
|
||||
Devices []migReported
|
||||
}
|
||||
|
||||
// migReported is one instance as the report describes it.
|
||||
type migReported struct {
|
||||
Device, GPUInstance, ComputeInstance int
|
||||
TotalMB, UsedMB int
|
||||
}
|
||||
|
||||
// enabled reads nvidia-smi's mode words: "Enabled", "Disabled", "N/A".
|
||||
func enabled(s string) bool { return strings.EqualFold(strings.TrimSpace(s), "Enabled") }
|
||||
|
||||
func atoiOr(s string, def int) int {
|
||||
n, err := strconv.Atoi(strings.TrimSpace(s))
|
||||
if err != nil {
|
||||
return def
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// parseMIGReport reads each card's MIG state from the XML report, in the
|
||||
// report's order, which is nvidia-smi's index order.
|
||||
func parseMIGReport(report []byte) ([]migState, error) {
|
||||
var log migLog
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
out := make([]migState, len(log.GPUs))
|
||||
for i, g := range log.GPUs {
|
||||
out[i] = migState{BusID: g.ID, Enabled: enabled(g.Current), PendingEnabled: enabled(g.Pending)}
|
||||
for _, d := range g.Devices {
|
||||
out[i].Devices = append(out[i].Devices, migReported{
|
||||
Device: atoiOr(d.Index, -1), GPUInstance: atoiOr(d.GPUInstance, -1),
|
||||
ComputeInstance: atoiOr(d.ComputeInstance, -1),
|
||||
TotalMB: mib(d.Total), UsedMB: mib(d.Used),
|
||||
})
|
||||
}
|
||||
slices.SortFunc(out[i].Devices, func(a, b migReported) int { return a.Device - b.Device })
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// --- putting it together ---
|
||||
|
||||
// migProblemNoInstances keeps a card in MIG mode with nothing carved from it
|
||||
// out of placement: CUDA cannot use the whole card while MIG is on, and
|
||||
// there is no instance to use instead.
|
||||
const migProblemNoInstances = "MIG is enabled with no instances created: nothing can run on it " +
|
||||
"until instances are created (nvidia-smi mig -cgi … -C) or MIG is turned off"
|
||||
|
||||
// migProblemUnnamed is a card whose instances exist but could not be named:
|
||||
// an engine is given an instance by its UUID, and without one it cannot be.
|
||||
const migProblemUnnamed = "MIG is enabled and has instances, but nvidia-smi -L did not name them, " +
|
||||
"so none can be given to an engine"
|
||||
|
||||
// expandMIG replaces each card in MIG mode with its instances.
|
||||
//
|
||||
// listed is nvidia-smi -L's instances by card, which names them; report is
|
||||
// the XML report's state per card, which gives their memory and ids. Either
|
||||
// may be missing: -L alone still says which instances exist, and the report
|
||||
// alone still says a card is in MIG mode, so a card it says has instances
|
||||
// that -L did not name is kept as the whole card with the reason, rather
|
||||
// than placed as a card CUDA will not open.
|
||||
func expandMIG(cards []GPU, listed map[int][]listedMIG, report []migState) []GPU {
|
||||
if len(listed) == 0 && !slices.ContainsFunc(report, func(s migState) bool { return s.Enabled }) {
|
||||
return cards
|
||||
}
|
||||
byBus := map[string]migState{}
|
||||
for _, s := range report {
|
||||
byBus[normalizeBusID(s.BusID)] = s
|
||||
}
|
||||
out := make([]GPU, 0, len(cards))
|
||||
for pos, c := range cards {
|
||||
st, ok := byBus[normalizeBusID(c.PCIBusID)]
|
||||
if !ok && pos < len(report) {
|
||||
st = report[pos]
|
||||
}
|
||||
devs := listed[c.Index]
|
||||
if !st.Enabled && len(devs) == 0 {
|
||||
out = append(out, c)
|
||||
continue
|
||||
}
|
||||
if len(devs) == 0 {
|
||||
p := migProblemNoInstances
|
||||
if len(st.Devices) > 0 {
|
||||
p = migProblemUnnamed
|
||||
}
|
||||
c.Health.Problems = append(c.Health.Problems, p)
|
||||
out = append(out, c)
|
||||
continue
|
||||
}
|
||||
for _, d := range devs {
|
||||
var rep migReported
|
||||
found := false
|
||||
for _, r := range st.Devices {
|
||||
if r.Device == d.Device {
|
||||
rep, found = r, true
|
||||
}
|
||||
}
|
||||
mig := migInstance(c, d.Device, d.Profile, d.UUID)
|
||||
if found {
|
||||
mig.MIG.GPUInstance, mig.MIG.ComputeInstance = rep.GPUInstance, rep.ComputeInstance
|
||||
mig.MemoryTotalMB, mig.MemoryUsedMB = rep.TotalMB, rep.UsedMB
|
||||
}
|
||||
if mig.MemoryTotalMB == 0 {
|
||||
// The profile says it to the gigabyte, which is close enough
|
||||
// to place by when the report could not be read.
|
||||
mig.MemoryTotalMB = MIGProfileMemoryGB(d.Profile) * 1024
|
||||
}
|
||||
assessHealth(&mig)
|
||||
out = append(out, mig)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// migInstance builds a MIG instance of card: everything about the card that
|
||||
// is shared — driver, compute capability, NUMA node, bus, temperature and
|
||||
// ECC errors — and nothing that is not. Not NVLink, which an instance cannot
|
||||
// use; not utilisation, which no driver reports per instance; and not the
|
||||
// card's memory or problems, which the instance's own replace.
|
||||
func migInstance(card GPU, device int, profile, uuid string) GPU {
|
||||
g := GPU{
|
||||
Index: MIGIndex(card.Index, device), UUID: uuid,
|
||||
Name: strings.TrimSpace(card.Name + " MIG " + profile),
|
||||
ComputeCapability: card.ComputeCapability, NUMANode: card.NUMANode,
|
||||
PCIBusID: card.PCIBusID, DriverVersion: card.DriverVersion, CUDAVersion: card.CUDAVersion,
|
||||
NICPaths: card.NICPaths, Fabric: card.Fabric,
|
||||
Health: GPUHealth{ECCErrors: card.Health.ECCErrors, TemperatureC: card.Health.TemperatureC},
|
||||
MIG: &MIGDevice{
|
||||
ParentIndex: card.Index, ParentUUID: card.UUID, ParentMemoryTotalMB: card.MemoryTotalMB,
|
||||
Device: device, Profile: profile, GPUInstance: -1, ComputeInstance: -1,
|
||||
},
|
||||
}
|
||||
return g
|
||||
}
|
||||
|
||||
// readMIG asks nvidia-smi for what expandMIG needs. A failure of either
|
||||
// command leaves its half empty, which expandMIG is written to take.
|
||||
func readMIG(ctx context.Context, binary string) (map[int][]listedMIG, []migState) {
|
||||
var listed map[int][]listedMIG
|
||||
if out, err := exec.CommandContext(ctx, binary, "-L").Output(); err == nil {
|
||||
listed = parseSMIList(string(out))
|
||||
}
|
||||
var report []migState
|
||||
if out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output(); err == nil {
|
||||
report, _ = parseMIGReport(out)
|
||||
}
|
||||
return listed, report
|
||||
}
|
||||
|
||||
// --- NVML ---
|
||||
|
||||
// nvmlMIGCard is the part of an NVML device MIG discovery reads, behind an
|
||||
// interface so the logic is tested without a driver: gpu_nvml.go adapts
|
||||
// nvml.Device to it.
|
||||
type nvmlMIGCard interface {
|
||||
// MIGMode is nvmlDeviceGetMigMode; ok is false on a card without MIG.
|
||||
MIGMode() (current, pending, ok bool)
|
||||
// MaxMIGDevices is nvmlDeviceGetMaxMigDeviceCount.
|
||||
MaxMIGDevices() int
|
||||
// MIGDevice is nvmlDeviceGetMigDeviceHandleByIndex; ok is false for an
|
||||
// index with no instance behind it, which is how NVML says so.
|
||||
MIGDevice(i int) (nvmlMIGInstance, bool)
|
||||
}
|
||||
|
||||
// nvmlMIGInstance is one MIG device handle.
|
||||
type nvmlMIGInstance interface {
|
||||
UUID() string
|
||||
// Name is "NVIDIA A100-SXM4-80GB MIG 3g.40gb" on the drivers that name
|
||||
// MIG devices, which is where the profile is read from.
|
||||
Name() string
|
||||
Memory() (totalMB, usedMB int, ok bool)
|
||||
Instances() (gpuInstance, computeInstance int)
|
||||
// Slices and MemoryMB are the instance's attributes, which name the
|
||||
// profile on a driver whose device name does not.
|
||||
Attributes() (slices, memoryMB int, ok bool)
|
||||
}
|
||||
|
||||
var nvmlMIGName = regexp.MustCompile(`MIG\s+(\S+)\s*$`)
|
||||
|
||||
// migFromNVML is what one card is to placement under NVML: its instances if
|
||||
// it is in MIG mode, itself if it is not. isMIG says which.
|
||||
func migFromNVML(card GPU, dev nvmlMIGCard) (out []GPU, isMIG bool) {
|
||||
current, _, ok := dev.MIGMode()
|
||||
if !ok || !current {
|
||||
return []GPU{card}, false
|
||||
}
|
||||
n := dev.MaxMIGDevices()
|
||||
for i := range n {
|
||||
h, ok := dev.MIGDevice(i)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
profile := ""
|
||||
if m := nvmlMIGName.FindStringSubmatch(h.Name()); m != nil {
|
||||
profile = m[1]
|
||||
} else if n, mem, ok := h.Attributes(); ok {
|
||||
// The convention NVIDIA's names follow: compute slices, then the
|
||||
// memory in whole gigabytes.
|
||||
profile = fmt.Sprintf("%dg.%dgb", n, int(math.Round(float64(mem)/1024)))
|
||||
}
|
||||
g := migInstance(card, i, profile, h.UUID())
|
||||
g.MIG.GPUInstance, g.MIG.ComputeInstance = h.Instances()
|
||||
if total, used, ok := h.Memory(); ok {
|
||||
g.MemoryTotalMB, g.MemoryUsedMB = total, used
|
||||
} else {
|
||||
g.MemoryTotalMB = MIGProfileMemoryGB(profile) * 1024
|
||||
}
|
||||
assessHealth(&g)
|
||||
out = append(out, g)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
card.Health.Problems = append(card.Health.Problems, migProblemNoInstances)
|
||||
return []GPU{card}, true
|
||||
}
|
||||
return out, true
|
||||
}
|
||||
|
||||
// --- what the doctor reports ---
|
||||
|
||||
// MIGCard is one card's MIG state, as `llmgw doctor` reports it.
|
||||
type MIGCard struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name,omitempty"`
|
||||
// Enabled is the mode in force; PendingEnabled the mode after the next
|
||||
// reset of the card, which differs from it until the card is reset.
|
||||
Enabled bool `json:"enabled"`
|
||||
PendingEnabled bool `json:"pending_enabled"`
|
||||
// Instances are the MIG devices an engine can be given, as nvidia-smi -L
|
||||
// lists them.
|
||||
Instances []MIGListed `json:"instances,omitempty"`
|
||||
// GPUInstances and ComputeInstances are nvidia-smi mig -lgi and -lci: a
|
||||
// GPU instance with no compute instance in it is not a device yet.
|
||||
GPUInstances []MIGInstance `json:"gpu_instances,omitempty"`
|
||||
ComputeInstances []MIGInstance `json:"compute_instances,omitempty"`
|
||||
}
|
||||
|
||||
// MIGListed is one instance an engine can be given.
|
||||
type MIGListed struct {
|
||||
Device int `json:"device"`
|
||||
Profile string `json:"profile"`
|
||||
UUID string `json:"uuid"`
|
||||
}
|
||||
|
||||
// MIGInstance is one row of nvidia-smi mig -lgi or -lci.
|
||||
type MIGInstance struct {
|
||||
GPU int `json:"gpu"`
|
||||
Profile string `json:"profile"`
|
||||
// ProfileID is NVIDIA's number for the profile.
|
||||
ProfileID int `json:"profile_id"`
|
||||
// GPUInstance is the GPU instance's id; ComputeInstance the compute
|
||||
// instance's within it, -1 in a -lgi row.
|
||||
GPUInstance int `json:"gpu_instance"`
|
||||
ComputeInstance int `json:"compute_instance"`
|
||||
Placement string `json:"placement,omitempty"`
|
||||
}
|
||||
|
||||
var (
|
||||
// -lgi: "| 0 MIG 3g.40gb 9 2 4:4 |"
|
||||
migLGIRow = regexp.MustCompile(`^\|\s*(\d+)\s+MIG\s+(\S+)\s+(\d+)\s+(\d+)\s+(\d+:\d+)\s*\|`)
|
||||
// -lci: "| 0 1 MIG 3g.40gb 2 0 0:3 |"
|
||||
migLCIRow = regexp.MustCompile(`^\|\s*(\d+)\s+(\d+)\s+MIG\s+(\S+)\s+(\d+)\s+(\d+)\s+(\d+:\d+)\s*\|`)
|
||||
)
|
||||
|
||||
// ParseMIGInstances reads nvidia-smi mig -lgi or -lci, whichever it is
|
||||
// given: the two tables differ by the GPU instance column -lci adds. "No GPU
|
||||
// instances found" and the like are no rows, not an error.
|
||||
func ParseMIGInstances(out string) []MIGInstance {
|
||||
var res []MIGInstance
|
||||
for _, line := range strings.Split(out, "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if m := migLCIRow.FindStringSubmatch(line); m != nil {
|
||||
res = append(res, MIGInstance{
|
||||
GPU: atoiOr(m[1], -1), GPUInstance: atoiOr(m[2], -1), Profile: m[3],
|
||||
ProfileID: atoiOr(m[4], -1), ComputeInstance: atoiOr(m[5], -1), Placement: m[6],
|
||||
})
|
||||
continue
|
||||
}
|
||||
if m := migLGIRow.FindStringSubmatch(line); m != nil {
|
||||
res = append(res, MIGInstance{
|
||||
GPU: atoiOr(m[1], -1), Profile: m[2], ProfileID: atoiOr(m[3], -1),
|
||||
GPUInstance: atoiOr(m[4], -1), ComputeInstance: -1, Placement: m[5],
|
||||
})
|
||||
}
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// DiscoverMIG reads every card's MIG state for the doctor: the mode from the
|
||||
// XML report, the devices from -L, the instances from nvidia-smi mig. Only
|
||||
// cards that support MIG are listed, and an error is returned only when the
|
||||
// report itself cannot be read.
|
||||
func DiscoverMIG(ctx context.Context, binary string) ([]MIGCard, error) {
|
||||
out, err := exec.CommandContext(ctx, binary, "-q", "-x").Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("running %s -q -x: %w", binary, err)
|
||||
}
|
||||
var list, lgi, lci string
|
||||
if b, err := exec.CommandContext(ctx, binary, "-L").Output(); err == nil {
|
||||
list = string(b)
|
||||
}
|
||||
// nvidia-smi mig exits non-zero when there is nothing to list, with the
|
||||
// table or the reason on its output either way.
|
||||
b, _ := exec.CommandContext(ctx, binary, "mig", "-lgi").CombinedOutput()
|
||||
lgi = string(b)
|
||||
b, _ = exec.CommandContext(ctx, binary, "mig", "-lci").CombinedOutput()
|
||||
lci = string(b)
|
||||
return buildMIGCards(out, list, lgi, lci)
|
||||
}
|
||||
|
||||
// buildMIGCards is DiscoverMIG from the commands' output.
|
||||
func buildMIGCards(report []byte, list, lgi, lci string) ([]MIGCard, error) {
|
||||
var log struct {
|
||||
GPUs []struct {
|
||||
Name string `xml:"product_name"`
|
||||
Current string `xml:"mig_mode>current_mig"`
|
||||
Pending string `xml:"mig_mode>pending_mig"`
|
||||
} `xml:"gpu"`
|
||||
}
|
||||
if err := xml.Unmarshal(report, &log); err != nil {
|
||||
return nil, fmt.Errorf("reading nvidia-smi's report: %w", err)
|
||||
}
|
||||
listed := parseSMIList(list)
|
||||
gis, cis := ParseMIGInstances(lgi), ParseMIGInstances(lci)
|
||||
var out []MIGCard
|
||||
for i, g := range log.GPUs {
|
||||
cur, pend := strings.TrimSpace(g.Current), strings.TrimSpace(g.Pending)
|
||||
// A card without MIG says N/A, or nothing on a driver that predates
|
||||
// it; neither is worth a line.
|
||||
if !enabled(cur) && !enabled(pend) && !strings.EqualFold(cur, "Disabled") {
|
||||
continue
|
||||
}
|
||||
c := MIGCard{Index: i, Name: strings.TrimSpace(g.Name), Enabled: enabled(cur), PendingEnabled: enabled(pend)}
|
||||
for _, l := range listed[i] {
|
||||
c.Instances = append(c.Instances, MIGListed(l))
|
||||
}
|
||||
for _, gi := range gis {
|
||||
if gi.GPU == i && gi.ComputeInstance < 0 {
|
||||
c.GPUInstances = append(c.GPUInstances, gi)
|
||||
}
|
||||
}
|
||||
for _, ci := range cis {
|
||||
if ci.GPU == i && ci.ComputeInstance >= 0 {
|
||||
c.ComputeInstances = append(c.ComputeInstances, ci)
|
||||
}
|
||||
}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// Two A100 80GB cards: card 0 in MIG mode, cut into two 3g.40gb instances,
|
||||
// card 1 whole. What nvidia-smi prints for that machine, by command. None of
|
||||
// it is from real MIG hardware, which this was written without; the shapes
|
||||
// follow NVIDIA's documentation and the output of drivers R470 to R550.
|
||||
const (
|
||||
migQuery = `0, GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77, NVIDIA A100-SXM4-80GB, 81920, 0, 8.0, 00000000:07:00.0, 550.54.15, 31, 0, [N/A]
|
||||
1, GPU-1f2e3d4c-5b6a-7980-a1b2-c3d4e5f60718, NVIDIA A100-SXM4-80GB, 81920, 0, 8.0, 00000000:0F:00.0, 550.54.15, 33, 0, 0
|
||||
`
|
||||
migList = `GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77)
|
||||
MIG 3g.40gb Device 0: (UUID: MIG-c6d4f1ef-42e4-5de3-91c7-45d71c87eb3f)
|
||||
MIG 3g.40gb Device 1: (UUID: MIG-0f7c3a2e-9d1b-5a8e-b3c4-1e2f3a4b5c6d)
|
||||
GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-1f2e3d4c-5b6a-7980-a1b2-c3d4e5f60718)
|
||||
`
|
||||
migReport = `<?xml version="1.0" ?>
|
||||
<nvidia_smi_log>
|
||||
<driver_version>550.54.15</driver_version>
|
||||
<gpu id="00000000:07:00.0">
|
||||
<product_name>NVIDIA A100-SXM4-80GB</product_name>
|
||||
<mig_mode><current_mig>Enabled</current_mig><pending_mig>Enabled</pending_mig></mig_mode>
|
||||
<mig_devices>
|
||||
<mig_device>
|
||||
<index>1</index>
|
||||
<gpu_instance_id>1</gpu_instance_id>
|
||||
<compute_instance_id>0</compute_instance_id>
|
||||
<device_attributes><shared><multiprocessor_count>42</multiprocessor_count></shared></device_attributes>
|
||||
<fb_memory_usage><total>40192 MiB</total><reserved>0 MiB</reserved><used>21000 MiB</used><free>19192 MiB</free></fb_memory_usage>
|
||||
</mig_device>
|
||||
<mig_device>
|
||||
<index>0</index>
|
||||
<gpu_instance_id>2</gpu_instance_id>
|
||||
<compute_instance_id>0</compute_instance_id>
|
||||
<fb_memory_usage><total>40192 MiB</total><reserved>0 MiB</reserved><used>37 MiB</used><free>40155 MiB</free></fb_memory_usage>
|
||||
</mig_device>
|
||||
</mig_devices>
|
||||
<fb_memory_usage><total>81920 MiB</total><used>21037 MiB</used></fb_memory_usage>
|
||||
<processes>
|
||||
<process_info>
|
||||
<gpu_instance_id>1</gpu_instance_id>
|
||||
<compute_instance_id>0</compute_instance_id>
|
||||
<pid>4100</pid><type>C</type><process_name>vllm</process_name>
|
||||
<used_memory>20900 MiB</used_memory>
|
||||
</process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
<gpu id="00000000:0F:00.0">
|
||||
<product_name>NVIDIA A100-SXM4-80GB</product_name>
|
||||
<mig_mode><current_mig>Disabled</current_mig><pending_mig>Disabled</pending_mig></mig_mode>
|
||||
<mig_devices>None</mig_devices>
|
||||
<fb_memory_usage><total>81920 MiB</total><used>300 MiB</used></fb_memory_usage>
|
||||
<processes>
|
||||
<process_info>
|
||||
<gpu_instance_id>N/A</gpu_instance_id>
|
||||
<compute_instance_id>N/A</compute_instance_id>
|
||||
<pid>5200</pid><type>C</type><used_memory>250 MiB</used_memory>
|
||||
</process_info>
|
||||
</processes>
|
||||
</gpu>
|
||||
</nvidia_smi_log>`
|
||||
migLGI = `+-------------------------------------------------------+
|
||||
| GPU instances: |
|
||||
| GPU Name Profile Instance Placement |
|
||||
| ID ID Start:Size |
|
||||
|=======================================================|
|
||||
| 0 MIG 3g.40gb 9 1 4:4 |
|
||||
+-------------------------------------------------------+
|
||||
| 0 MIG 3g.40gb 9 2 0:4 |
|
||||
+-------------------------------------------------------+
|
||||
`
|
||||
migLCI = `+--------------------------------------------------------------------+
|
||||
| Compute instances: |
|
||||
| GPU GPU Name Profile Instance Placement |
|
||||
| Instance ID ID Start:Size |
|
||||
| ID |
|
||||
|====================================================================|
|
||||
| 0 1 MIG 3g.40gb 2 0 0:3 |
|
||||
+--------------------------------------------------------------------+
|
||||
| 0 2 MIG 3g.40gb 2 0 0:3 |
|
||||
+--------------------------------------------------------------------+
|
||||
`
|
||||
)
|
||||
|
||||
func migMachine(t *testing.T) []GPU {
|
||||
t.Helper()
|
||||
cards, err := parseSMI(migQuery)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
report, err := parseMIGReport([]byte(migReport))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return expandMIG(cards, parseSMIList(migList), report)
|
||||
}
|
||||
|
||||
func TestACardInMIGModeIsReportedAsItsInstancesInsteadOfItself(t *testing.T) {
|
||||
gpus := migMachine(t)
|
||||
var got []string
|
||||
for _, g := range gpus {
|
||||
got = append(got, DeviceLabel(g.Index)+" "+g.Class())
|
||||
}
|
||||
want := []string{"0/mig0 a100-80g-mig-3g.40gb", "0/mig1 a100-80g-mig-3g.40gb", "1 a100-80g"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("devices = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
mig := gpus[1]
|
||||
if mig.Index != 1001 || mig.UUID != "MIG-0f7c3a2e-9d1b-5a8e-b3c4-1e2f3a4b5c6d" {
|
||||
t.Errorf("instance 1 is %d %s; an engine is given it by that UUID", mig.Index, mig.UUID)
|
||||
}
|
||||
if mig.MemoryTotalMB != 40192 || mig.MemoryUsedMB != 21000 {
|
||||
t.Errorf("instance 1 memory %d/%d, want its own 21000/40192 rather than the card's",
|
||||
mig.MemoryUsedMB, mig.MemoryTotalMB)
|
||||
}
|
||||
if m := mig.MIG; m == nil || m.ParentIndex != 0 || m.Device != 1 || m.Profile != "3g.40gb" ||
|
||||
m.GPUInstance != 1 || m.ComputeInstance != 0 || m.ParentUUID != "GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77" {
|
||||
t.Errorf("instance 1's parentage: %+v", mig.MIG)
|
||||
}
|
||||
if mig.ComputeCapability != "8.0" || mig.DriverVersion != "550.54.15" || mig.Health.TemperatureC != 31 {
|
||||
t.Errorf("what the card and its instances share was not carried: %+v", mig)
|
||||
}
|
||||
// Twenty gigabytes in use on an instance is a foreign tenant there, and
|
||||
// only there: instance 0 next to it is free.
|
||||
if mig.Health.OK() || !gpus[0].Health.OK() {
|
||||
t.Errorf("foreign memory judged per instance: 0 %v, 1 %v", gpus[0].Health.Problems, mig.Health.Problems)
|
||||
}
|
||||
// No driver reports utilisation per instance, and none is made up.
|
||||
if mig.UtilizationPercent != 0 {
|
||||
t.Errorf("an instance reported utilisation %d", mig.UtilizationPercent)
|
||||
}
|
||||
if gpus[2].IsMIG() || gpus[2].Index != 1 || gpus[2].CardIndex() != 1 || gpus[0].CardIndex() != 0 {
|
||||
t.Errorf("the whole card beside it changed: %+v", gpus[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMIGInstanceSurvivesTheTripToTheControlPlaneWithItsClass(t *testing.T) {
|
||||
g := migMachine(t)[0]
|
||||
b, err := json.Marshal(g)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(b), `"class":"a100-80g-mig-3g.40gb"`) {
|
||||
t.Errorf("the JSON carries no class for the instance: %s", b)
|
||||
}
|
||||
var back GPU
|
||||
if err := json.Unmarshal(b, &back); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if back.Class() != g.Class() || !reflect.DeepEqual(back.MIG, g.MIG) {
|
||||
t.Errorf("came back as %s %+v", back.Class(), back.MIG)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMIGClassesAreNamedFromTheCardAndTheProfile(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
name string
|
||||
cardMB int
|
||||
profile string
|
||||
want string
|
||||
}{
|
||||
{"NVIDIA A100-SXM4-80GB MIG 3g.40gb", 81920, "3g.40gb", "a100-80g-mig-3g.40gb"},
|
||||
// The same profile name on a 40 GB card is another class: its
|
||||
// instance is the same size, but the card is not, and neither is
|
||||
// the set of profiles that card can be cut to.
|
||||
{"NVIDIA A100-SXM4-40GB MIG 3g.20gb", 40960, "3g.20gb", "a100-40g-mig-3g.20gb"},
|
||||
{"NVIDIA H100 80GB HBM3 MIG 1g.10gb+me", 81559, "1g.10gb+me", "h100-80g-mig-1g.10gb-me"},
|
||||
{"NVIDIA H100 80GB HBM3 MIG 7g.80gb", 81559, "7g.80gb", "h100-80g-mig-7g.80gb"},
|
||||
} {
|
||||
g := GPU{Name: c.name, MemoryTotalMB: 10000, MIG: &MIGDevice{ParentMemoryTotalMB: c.cardMB, Profile: c.profile}}
|
||||
if got := g.Class(); got != c.want {
|
||||
t.Errorf("%s -> %s, want %s", c.name, got, c.want)
|
||||
}
|
||||
if !IsMIGClass(g.Class()) {
|
||||
t.Errorf("%s is not recognised as a MIG class", g.Class())
|
||||
}
|
||||
}
|
||||
if IsMIGClass("a100-80g") || IsMIGClass("6000-24g") {
|
||||
t.Error("a whole card's class was taken for a MIG one")
|
||||
}
|
||||
if MIGProfileMemoryGB("a100-80g-mig-3g.40gb") != 40 || MIGProfileMemoryGB("1g.10gb+me") != 10 || MIGProfileMemoryGB("a100-80g") != 0 {
|
||||
t.Error("the instance's memory is misread from its profile")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeviceNumbersAreShownAsCardAndInstance(t *testing.T) {
|
||||
for i, want := range map[int]string{0: "0", 7: "7", MIGIndex(0, 2): "0/mig2", MIGIndex(3, 0): "3/mig0", 1302: "3/mig2"} {
|
||||
if got := DeviceLabel(i); got != want {
|
||||
t.Errorf("DeviceLabel(%d) = %s, want %s", i, got, want)
|
||||
}
|
||||
}
|
||||
if IsMIGIndex(15) || !IsMIGIndex(MIGIndex(0, 0)) {
|
||||
t.Error("whole cards and instances are told apart wrongly")
|
||||
}
|
||||
}
|
||||
|
||||
func TestACardInMIGModeWithNoInstancesIsKeptOutOfPlacement(t *testing.T) {
|
||||
cards, _ := parseSMI(migQuery)
|
||||
report, _ := parseMIGReport([]byte(strings.Replace(migReport,
|
||||
migReport[strings.Index(migReport, "<mig_devices>"):strings.Index(migReport, "</mig_devices>")+len("</mig_devices>")],
|
||||
"<mig_devices>None</mig_devices>", 1)))
|
||||
gpus := expandMIG(cards, parseSMIList("GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-x)\nGPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-y)\n"), report)
|
||||
if len(gpus) != 2 || gpus[0].IsMIG() {
|
||||
t.Fatalf("got %+v", gpus)
|
||||
}
|
||||
// CUDA will not open the whole card while MIG is on, and there is no
|
||||
// instance to open instead: placing anything here fails at start.
|
||||
if gpus[0].Health.OK() || !strings.Contains(gpus[0].Health.Problems[0], "no instances") {
|
||||
t.Errorf("card 0's problems: %v", gpus[0].Health.Problems)
|
||||
}
|
||||
if !gpus[1].Health.OK() {
|
||||
t.Errorf("card 1, not in MIG mode, was flagged: %v", gpus[1].Health.Problems)
|
||||
}
|
||||
|
||||
// Instances the report sees but -L could not name cannot be given to an
|
||||
// engine either; the card says why.
|
||||
report, _ = parseMIGReport([]byte(migReport))
|
||||
gpus = expandMIG(cards, nil, report)
|
||||
if len(gpus) != 2 || gpus[0].Health.OK() || !strings.Contains(gpus[0].Health.Problems[0], "did not name") {
|
||||
t.Errorf("with -L failing: %+v", gpus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstancesAreFoundFromTheListAloneWhenTheReportCannotBeRead(t *testing.T) {
|
||||
cards, _ := parseSMI(migQuery)
|
||||
gpus := expandMIG(cards, parseSMIList(migList), nil)
|
||||
if len(gpus) != 3 || !gpus[0].IsMIG() || gpus[0].MemoryTotalMB != 40*1024 {
|
||||
t.Errorf("without the report: %+v", gpus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOlderDriversInstanceUUIDIsTakenAsItIs(t *testing.T) {
|
||||
// R450 and R460 name an instance by its card and ids, and CUDA of that
|
||||
// era takes exactly that in CUDA_VISIBLE_DEVICES.
|
||||
l := parseSMIList("GPU 0: A100-SXM4-40GB (UUID: GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77)\n" +
|
||||
" MIG 1g.5gb Device 0: (UUID: MIG-GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77/7/0)\n")
|
||||
if len(l[0]) != 1 || l[0][0].UUID != "MIG-GPU-5d5ba0d6-d33d-2b2c-524d-9e3d8d2b8a77/7/0" || l[0][0].Profile != "1g.5gb" {
|
||||
t.Errorf("parsed %+v", l)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessesOnAMIGCardAreChargedToTheirOwnInstance(t *testing.T) {
|
||||
mem, err := parseGPUMemory([]byte(migReport))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []GPUMemory{
|
||||
{Index: 1000, UsedMB: 37},
|
||||
{Index: 1001, UsedMB: 21000, Procs: []GPUProcess{{PID: 4100, UsedMB: 20900}}},
|
||||
{Index: 1, UsedMB: 300, Procs: []GPUProcess{{PID: 5200, UsedMB: 250}}},
|
||||
}
|
||||
if !reflect.DeepEqual(mem, want) {
|
||||
t.Errorf("got %+v\nwant %+v", mem, want)
|
||||
}
|
||||
// In the order discovery reports the devices, so a heartbeat's readings
|
||||
// line up with the inventory one for one.
|
||||
gpus := migMachine(t)
|
||||
for i := range gpus {
|
||||
if mem[i].Index != gpus[i].Index {
|
||||
t.Errorf("reading %d is for device %d, inventory has %d", i, mem[i].Index, gpus[i].Index)
|
||||
}
|
||||
}
|
||||
procs, _ := parseGPUProcesses([]byte(migReport))
|
||||
if !reflect.DeepEqual(procs, map[int][]int{1001: {4100}, 1: {5200}}) {
|
||||
t.Errorf("processes by device: %v", procs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMIGInstanceTakesItsCardsPlaceInTheTopologyButNoNVLink(t *testing.T) {
|
||||
gpus := migMachine(t)
|
||||
topo := Topology{
|
||||
NVLink: map[int]map[int]int{0: {1: 12}, 1: {0: 12}},
|
||||
NUMA: map[int]int{0: 0, 1: 1},
|
||||
NICPath: map[int]map[string]string{0: {"mlx5_0": "PIX"}},
|
||||
}
|
||||
topo.Apply(gpus)
|
||||
if len(gpus[0].NVLinkPeers) != 0 || len(gpus[1].NVLinkPeers) != 0 {
|
||||
t.Errorf("MIG instances were given NVLink peers: %v %v", gpus[0].NVLinkPeers, gpus[1].NVLinkPeers)
|
||||
}
|
||||
if gpus[1].NUMANode != 0 || gpus[1].NICPaths["mlx5_0"] != "PIX" {
|
||||
t.Errorf("instance 1 did not take its card's NUMA node and adapters: %+v", gpus[1])
|
||||
}
|
||||
if gpus[2].NUMANode != 1 || !slices.Equal(gpus[2].NVLinkPeers, []int{0}) {
|
||||
t.Errorf("the whole card's topology: %+v", gpus[2])
|
||||
}
|
||||
}
|
||||
|
||||
func TestMIGInstancesAreDiscoveredThroughNvidiaSMI(t *testing.T) {
|
||||
// The real code path against a stub binary that answers each command as
|
||||
// the machine above would.
|
||||
dir := t.TempDir()
|
||||
stub := filepath.Join(dir, "nvidia-smi")
|
||||
script := "#!/bin/sh\ncase \"$1\" in\n" +
|
||||
"-L) cat <<'EOF'\n" + migList + "EOF\n;;\n" +
|
||||
"-q) cat <<'EOF'\n" + migReport + "\nEOF\n;;\n" +
|
||||
"*) cat <<'EOF'\n" + migQuery + "EOF\n;;\nesac\n"
|
||||
if err := os.WriteFile(stub, []byte(script), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
gpus, err := DiscoverGPUsSMI(t.Context(), stub)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
inv := Inventory{GPUs: gpus}
|
||||
if !slices.Equal(inv.Classes(), []string{"a100-80g", "a100-80g-mig-3g.40gb"}) || len(gpus) != 3 {
|
||||
t.Errorf("classes %v from %d devices", inv.Classes(), len(gpus))
|
||||
}
|
||||
}
|
||||
|
||||
// fakeMIGCard is NVML's view of one card, as nvmlDeviceGetMigMode,
|
||||
// nvmlDeviceGetMaxMigDeviceCount and nvmlDeviceGetMigDeviceHandleByIndex
|
||||
// would give it.
|
||||
type fakeMIGCard struct {
|
||||
mode, pending, supported bool
|
||||
max int
|
||||
devs map[int]fakeMIGInstance
|
||||
}
|
||||
|
||||
func (c fakeMIGCard) MIGMode() (bool, bool, bool) { return c.mode, c.pending, c.supported }
|
||||
func (c fakeMIGCard) MaxMIGDevices() int { return c.max }
|
||||
func (c fakeMIGCard) MIGDevice(i int) (nvmlMIGInstance, bool) {
|
||||
d, ok := c.devs[i]
|
||||
return d, ok
|
||||
}
|
||||
|
||||
type fakeMIGInstance struct {
|
||||
uuid, name string
|
||||
total, used int
|
||||
gi, ci int
|
||||
slices, attrMem int
|
||||
}
|
||||
|
||||
func (m fakeMIGInstance) UUID() string { return m.uuid }
|
||||
func (m fakeMIGInstance) Name() string { return m.name }
|
||||
func (m fakeMIGInstance) Memory() (int, int, bool) {
|
||||
return m.total, m.used, m.total > 0
|
||||
}
|
||||
func (m fakeMIGInstance) Instances() (int, int) { return m.gi, m.ci }
|
||||
func (m fakeMIGInstance) Attributes() (int, int, bool) {
|
||||
return m.slices, m.attrMem, m.slices > 0
|
||||
}
|
||||
|
||||
func TestNVMLReportsACardInMIGModeAsItsInstances(t *testing.T) {
|
||||
card := GPU{Index: 2, UUID: "GPU-card", Name: "NVIDIA H100 80GB HBM3", MemoryTotalMB: 81559,
|
||||
ComputeCapability: "9.0", NUMANode: 1, Health: GPUHealth{TemperatureC: 40}}
|
||||
// Seven slots, three filled, one of them on a driver that does not put
|
||||
// the profile in the device's name.
|
||||
dev := fakeMIGCard{mode: true, pending: true, supported: true, max: 7, devs: map[int]fakeMIGInstance{
|
||||
0: {uuid: "MIG-aaa", name: "NVIDIA H100 80GB HBM3 MIG 3g.40gb", total: 40448, used: 10, gi: 2, ci: 0},
|
||||
1: {uuid: "MIG-bbb", name: "NVIDIA H100 80GB HBM3 MIG 3g.40gb", total: 40448, used: 10, gi: 1, ci: 0},
|
||||
4: {uuid: "MIG-ccc", name: "NVIDIA H100 80GB HBM3", total: 9984, gi: 9, ci: 0, slices: 1, attrMem: 9984},
|
||||
}}
|
||||
gpus, isMIG := migFromNVML(card, dev)
|
||||
if !isMIG || len(gpus) != 3 {
|
||||
t.Fatalf("got %d devices, MIG %v", len(gpus), isMIG)
|
||||
}
|
||||
var classes []string
|
||||
for _, g := range gpus {
|
||||
classes = append(classes, DeviceLabel(g.Index)+" "+g.UUID+" "+g.Class())
|
||||
}
|
||||
want := []string{"2/mig0 MIG-aaa h100-80g-mig-3g.40gb", "2/mig1 MIG-bbb h100-80g-mig-3g.40gb",
|
||||
"2/mig4 MIG-ccc h100-80g-mig-1g.10gb"}
|
||||
if !slices.Equal(classes, want) {
|
||||
t.Errorf("devices %q, want %q", classes, want)
|
||||
}
|
||||
if g := gpus[1]; g.MemoryTotalMB != 40448 || g.MIG.GPUInstance != 1 || g.NUMANode != 1 ||
|
||||
g.Health.TemperatureC != 40 || len(g.NVLinkPeers) != 0 {
|
||||
t.Errorf("instance 1: %+v", g)
|
||||
}
|
||||
|
||||
// MIG on and nothing carved: the card, kept out of placement.
|
||||
gpus, isMIG = migFromNVML(card, fakeMIGCard{mode: true, supported: true, max: 7})
|
||||
if !isMIG || len(gpus) != 1 || gpus[0].IsMIG() || gpus[0].Health.OK() {
|
||||
t.Errorf("an empty MIG card: %+v", gpus)
|
||||
}
|
||||
// MIG off, or a card without it: the card, as it always was.
|
||||
for _, d := range []fakeMIGCard{{supported: true, max: 7}, {}} {
|
||||
gpus, isMIG = migFromNVML(card, d)
|
||||
if isMIG || len(gpus) != 1 || !reflect.DeepEqual(gpus[0], card) {
|
||||
t.Errorf("a card not in MIG mode came back as %+v", gpus)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMIGInstanceTablesAreRead(t *testing.T) {
|
||||
gis := ParseMIGInstances(migLGI)
|
||||
if len(gis) != 2 || gis[0] != (MIGInstance{GPU: 0, Profile: "3g.40gb", ProfileID: 9, GPUInstance: 1,
|
||||
ComputeInstance: -1, Placement: "4:4"}) {
|
||||
t.Errorf("-lgi: %+v", gis)
|
||||
}
|
||||
cis := ParseMIGInstances(migLCI)
|
||||
if len(cis) != 2 || cis[1] != (MIGInstance{GPU: 0, GPUInstance: 2, Profile: "3g.40gb", ProfileID: 2,
|
||||
ComputeInstance: 0, Placement: "0:3"}) {
|
||||
t.Errorf("-lci: %+v", cis)
|
||||
}
|
||||
if got := ParseMIGInstances("No GPU instances found: Not Found\n"); len(got) != 0 {
|
||||
t.Errorf("nothing to list read as %+v", got)
|
||||
}
|
||||
|
||||
cards, err := buildMIGCards([]byte(migReport), migList, migLGI, migLCI)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(cards) != 2 || !cards[0].Enabled || len(cards[0].Instances) != 2 || len(cards[0].GPUInstances) != 2 ||
|
||||
len(cards[0].ComputeInstances) != 2 || cards[1].Enabled {
|
||||
t.Errorf("cards: %+v", cards)
|
||||
}
|
||||
}
|
||||
@@ -166,19 +166,25 @@ func splitTabs(l string) []string {
|
||||
|
||||
// Apply puts the topology on the GPUs it describes. What a GPU already knows
|
||||
// — NVLink peers from NVML, which is the more exact source — is kept.
|
||||
//
|
||||
// The matrix is of whole cards. A MIG instance takes its card's NUMA node and
|
||||
// adapters, and none of its NVLink: instances cannot reach each other or any
|
||||
// other card over it, and peers listed for one would draw a tensor-parallel
|
||||
// group onto it.
|
||||
func (t Topology) Apply(gpus []GPU) {
|
||||
for i := range gpus {
|
||||
g := &gpus[i]
|
||||
if len(g.NVLinkPeers) == 0 {
|
||||
for peer := range t.NVLink[g.Index] {
|
||||
card := g.CardIndex()
|
||||
if len(g.NVLinkPeers) == 0 && !g.IsMIG() {
|
||||
for peer := range t.NVLink[card] {
|
||||
g.NVLinkPeers = append(g.NVLinkPeers, peer)
|
||||
}
|
||||
sort.Ints(g.NVLinkPeers)
|
||||
}
|
||||
if n, ok := t.NUMA[g.Index]; ok {
|
||||
if n, ok := t.NUMA[card]; ok {
|
||||
g.NUMANode = n
|
||||
}
|
||||
if paths := t.NICPath[g.Index]; len(paths) > 0 {
|
||||
if paths := t.NICPath[card]; len(paths) > 0 {
|
||||
g.NICPaths = paths
|
||||
}
|
||||
}
|
||||
|
||||
@@ -62,6 +62,12 @@ type ShardSpec struct {
|
||||
|
||||
// GPUIndices are the local GPUs this shard uses.
|
||||
GPUIndices []int
|
||||
// GPUUUIDs name the same devices by UUID, in the same order, and are set
|
||||
// by the agent only when one of them is a MIG instance. An instance has
|
||||
// no index CUDA knows: CUDA_VISIBLE_DEVICES=MIG-<uuid> is the only way to
|
||||
// give a process one, and the indices placement uses for it (1000 and
|
||||
// up) mean nothing to CUDA.
|
||||
GPUUUIDs []string `json:"gpu_uuids,omitempty"`
|
||||
// HCAs are the adapters of the chosen fabric, and only that fabric. A shard
|
||||
// must never be given HCAs from two fabrics.
|
||||
HCAs []string
|
||||
@@ -257,7 +263,7 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
|
||||
// one instead of timing out.
|
||||
func baseEnv(s ShardSpec) map[string]string {
|
||||
env := map[string]string{
|
||||
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
|
||||
"CUDA_VISIBLE_DEVICES": visibleDevices(s),
|
||||
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
|
||||
// counts fastest first unless told otherwise, and on a node with two
|
||||
// kinds of GPU the shard would get a different card than placement
|
||||
@@ -329,6 +335,21 @@ func materializeEnv(generated map[string]string) []string {
|
||||
return out
|
||||
}
|
||||
|
||||
// visibleDevices is CUDA_VISIBLE_DEVICES for a shard: the indices placement
|
||||
// gave it, or their UUIDs when the agent named them, which it does for a MIG
|
||||
// instance.
|
||||
func visibleDevices(s ShardSpec) string {
|
||||
if len(s.GPUUUIDs) > 0 {
|
||||
return strings.Join(s.GPUUUIDs, ",")
|
||||
}
|
||||
return joinInts(s.GPUIndices)
|
||||
}
|
||||
|
||||
// OnMIG reports whether the shard was given a MIG instance.
|
||||
func (s ShardSpec) OnMIG() bool {
|
||||
return slices.ContainsFunc(s.GPUUUIDs, func(u string) bool { return strings.HasPrefix(u, "MIG-") })
|
||||
}
|
||||
|
||||
func joinInts(v []int) string {
|
||||
parts := make([]string, len(v))
|
||||
for i, n := range v {
|
||||
|
||||
@@ -178,7 +178,11 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
|
||||
// as the driver lists them, which on NVIDIA follows the same bus order
|
||||
// as nvidia-smi; were it not to, the shard would not hold memory on its
|
||||
// own GPUs, and starting it checks exactly that.
|
||||
if len(s.GPUIndices) > 0 {
|
||||
//
|
||||
// Not on a MIG instance, which Vulkan does not see at all: only CUDA
|
||||
// opens one, by the UUID CUDA_VISIBLE_DEVICES carries, and an index here
|
||||
// would name some other card.
|
||||
if len(s.GPUIndices) > 0 && !s.OnMIG() {
|
||||
env["GGML_VK_VISIBLE_DEVICES"] = joinInts(s.GPUIndices)
|
||||
}
|
||||
return []Command{{
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package engines
|
||||
|
||||
import "testing"
|
||||
|
||||
const migUUID = "MIG-c6d4f1ef-42e4-5de3-91c7-45d71c87eb3f"
|
||||
|
||||
func TestAShardOnAMIGInstanceIsGivenItByUUID(t *testing.T) {
|
||||
// Placement numbers card 0's instance 1 as 1001, which CUDA has never
|
||||
// heard of: CUDA_VISIBLE_DEVICES=1001 is no device at all, and =1 would
|
||||
// be the wrong card.
|
||||
s := singleNodeSpec()
|
||||
s.GPUIndices, s.GPUUUIDs = []int{1001}, []string{migUUID}
|
||||
s.Profile.TP = 1
|
||||
|
||||
env := envMap(buildOne(t, &VLLM{}, s).Env)
|
||||
if env["CUDA_VISIBLE_DEVICES"] != migUUID {
|
||||
t.Errorf("vLLM: CUDA_VISIBLE_DEVICES=%q, want the instance's UUID", env["CUDA_VISIBLE_DEVICES"])
|
||||
}
|
||||
|
||||
l := llamaSpec([]int{1001})
|
||||
l.GPUUUIDs = s.GPUUUIDs
|
||||
cmds, err := (&LlamaCpp{}).Commands(l)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
env = envMap(cmds[0].Env)
|
||||
if env["CUDA_VISIBLE_DEVICES"] != migUUID {
|
||||
t.Errorf("llama.cpp: CUDA_VISIBLE_DEVICES=%q, want the instance's UUID", env["CUDA_VISIBLE_DEVICES"])
|
||||
}
|
||||
// Vulkan does not see MIG instances, and an index for it would name
|
||||
// some other card.
|
||||
if v, set := env["GGML_VK_VISIBLE_DEVICES"]; set {
|
||||
t.Errorf("GGML_VK_VISIBLE_DEVICES=%q on a MIG instance", v)
|
||||
}
|
||||
if v, _ := flagValue(cmds[0].Args, "--n-gpu-layers"); v != "999" {
|
||||
t.Errorf("the model was not offloaded to the instance: --n-gpu-layers %q", v)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAShardOnWholeCardsIsStillGivenThemByIndex(t *testing.T) {
|
||||
env := envMap(buildOne(t, &VLLM{}, singleNodeSpec()).Env)
|
||||
if env["CUDA_VISIBLE_DEVICES"] != "0,1,2,3" {
|
||||
t.Errorf("CUDA_VISIBLE_DEVICES=%q", env["CUDA_VISIBLE_DEVICES"])
|
||||
}
|
||||
cmds, _ := (&LlamaCpp{}).Commands(llamaSpec([]int{2}))
|
||||
if env := envMap(cmds[0].Env); env["GGML_VK_VISIBLE_DEVICES"] != "2" {
|
||||
t.Errorf("GGML_VK_VISIBLE_DEVICES=%q", env["GGML_VK_VISIBLE_DEVICES"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
)
|
||||
|
||||
func TestAShardPlacedOnAMIGInstanceIsLaunchedWithItsUUID(t *testing.T) {
|
||||
// The control plane sends device 1001: card 0's instance 1, by the number
|
||||
// placement gives it. Only this node knows the instance's UUID, which is
|
||||
// the one thing CUDA will open it by.
|
||||
a := &Agent{}
|
||||
a.inv.GPUs = []discovery.GPU{
|
||||
{Index: 1000, UUID: "MIG-aaa", MIG: &discovery.MIGDevice{ParentIndex: 0, Device: 0, Profile: "3g.40gb"}},
|
||||
{Index: 1001, UUID: "MIG-bbb", MIG: &discovery.MIGDevice{ParentIndex: 0, Device: 1, Profile: "3g.40gb"}},
|
||||
{Index: 1, UUID: "GPU-whole"},
|
||||
}
|
||||
|
||||
spec := engines.ShardSpec{GPUIndices: []int{1001}}
|
||||
a.localizeSpec(&spec)
|
||||
if got := engines.NCCLEnv(spec)["CUDA_VISIBLE_DEVICES"]; got != "MIG-bbb" {
|
||||
t.Errorf("CUDA_VISIBLE_DEVICES=%q, want MIG-bbb", got)
|
||||
}
|
||||
|
||||
// A whole card is given by index, as it always was.
|
||||
whole := engines.ShardSpec{GPUIndices: []int{1}}
|
||||
a.localizeSpec(&whole)
|
||||
if whole.GPUUUIDs != nil || engines.NCCLEnv(whole)["CUDA_VISIBLE_DEVICES"] != "1" {
|
||||
t.Errorf("a whole card was given as %v / %q", whole.GPUUUIDs, engines.NCCLEnv(whole)["CUDA_VISIBLE_DEVICES"])
|
||||
}
|
||||
|
||||
// A device this node does not have is not guessed at.
|
||||
gone := engines.ShardSpec{GPUIndices: []int{1002}}
|
||||
a.localizeSpec(&gone)
|
||||
if gone.GPUUUIDs != nil {
|
||||
t.Errorf("an unknown device was named %v", gone.GPUUUIDs)
|
||||
}
|
||||
}
|
||||
@@ -159,13 +159,13 @@ func printInventory(inv discovery.Inventory) {
|
||||
if !g.Health.OK() {
|
||||
health = g.Health.Problems[0]
|
||||
}
|
||||
fmt.Fprintf(w, " %d\t%s\t%s\t%d MiB\t%s\t%s\n",
|
||||
g.Index, g.Class(), g.Name, g.MemoryTotalMB, g.ComputeCapability, health)
|
||||
fmt.Fprintf(w, " %s\t%s\t%s\t%d MiB\t%s\t%s\n",
|
||||
discovery.DeviceLabel(g.Index), g.Class(), g.Name, g.MemoryTotalMB, g.ComputeCapability, health)
|
||||
}
|
||||
fmt.Fprintf(w, " classes: %v\n", inv.Classes())
|
||||
for _, g := range inv.GPUs {
|
||||
if len(g.NVLinkPeers) > 0 {
|
||||
fmt.Fprintf(w, " GPU %d NVLink peers: %v\n", g.Index, g.NVLinkPeers)
|
||||
fmt.Fprintf(w, " GPU %s NVLink peers: %v\n", discovery.DeviceLabel(g.Index), g.NVLinkPeers)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+24
-2
@@ -14,6 +14,7 @@ import (
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
@@ -212,7 +213,7 @@ func newNodesCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
// at when a model will not place and the hardware looks
|
||||
// fine.
|
||||
join(n.Engines),
|
||||
join(n.Classes), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
|
||||
classesColumn(n), join(n.Fabrics), n.Replicas, n.LastSeenAgo)
|
||||
}
|
||||
_ = w.Flush()
|
||||
|
||||
@@ -410,7 +411,9 @@ func newReplicasCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
for _, r := range reps {
|
||||
var nodes []string
|
||||
for _, s := range r.Shards {
|
||||
nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, s.GPUs))
|
||||
// "[0/mig2]" for a MIG instance, not the 1002 it is
|
||||
// numbered by.
|
||||
nodes = append(nodes, fmt.Sprintf("%s:%v", s.Node, discovery.DeviceLabels(s.GPUs)))
|
||||
}
|
||||
// The state says stale rather than a column of its own: most
|
||||
// replicas are not, a column would be empty for all of them,
|
||||
@@ -540,6 +543,25 @@ func emitJSON(v any) error {
|
||||
return enc.Encode(v)
|
||||
}
|
||||
|
||||
// classesColumn is a node's GPU classes, with how many of each when it has
|
||||
// more than one or MIG instances among them: a card cut into two 3g.40gb
|
||||
// instances beside a whole card is "a100-80g-mig-3g.40gb×2,a100-80g×1", where
|
||||
// the GPUS column's 3 alone would read as three of something.
|
||||
func classesColumn(n adminapi.NodeView) string {
|
||||
counted := len(n.ClassGPUs) > 1
|
||||
for _, c := range n.ClassGPUs {
|
||||
counted = counted || c.MIG
|
||||
}
|
||||
if !counted {
|
||||
return join(n.Classes)
|
||||
}
|
||||
parts := make([]string, len(n.ClassGPUs))
|
||||
for i, c := range n.ClassGPUs {
|
||||
parts[i] = fmt.Sprintf("%s×%d", c.Class, c.Total)
|
||||
}
|
||||
return strings.Join(parts, ",")
|
||||
}
|
||||
|
||||
func join(v []string) string {
|
||||
if len(v) == 0 {
|
||||
return "-"
|
||||
|
||||
@@ -110,9 +110,10 @@ func printDoctor(out io.Writer, f doctor.Facts, findings []doctor.Finding, verbo
|
||||
|
||||
w := tabwriter.NewWriter(out, 0, 0, 2, ' ', 0)
|
||||
if len(f.GPUs) > 0 {
|
||||
fmt.Fprintln(w, "\nGPU\tNAME\tNUMA\tNVLINK PEERS\tNEAREST ADAPTER")
|
||||
fmt.Fprintln(w, "\nGPU\tNAME\tCLASS\tNUMA\tNVLINK PEERS\tNEAREST ADAPTER")
|
||||
for _, g := range f.GPUs {
|
||||
fmt.Fprintf(w, "%d\t%s\t%d\t%s\t%s\n", g.Index, g.Name, g.NUMANode,
|
||||
// A MIG instance as 0/mig2, under its card.
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%d\t%s\t%s\n", discovery.DeviceLabel(g.Index), g.Name, g.Class(), g.NUMANode,
|
||||
orDash(ints(g.NVLinkPeers)), orDash(nearest(g.NICPaths)))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
)
|
||||
|
||||
func TestAPlanSeesANodesMIGInstancesAsAClassOfTheirOwn(t *testing.T) {
|
||||
// Card 0 cut into two 3g.40gb instances, both free, and card 1 whole and
|
||||
// busy: the instances are a pool of forty-gigabyte devices, and the busy
|
||||
// card no pool at all.
|
||||
hw := hardwareFrom([]adminapi.NodeView{{ID: "a100", State: "ready", GPUTotal: 3, GPUFree: 2,
|
||||
Classes: []string{"a100-80g-mig-3g.40gb", "a100-80g"},
|
||||
ClassGPUs: []adminapi.ClassGPUs{
|
||||
{Class: "a100-80g-mig-3g.40gb", Total: 2, Free: 2, MIG: true},
|
||||
{Class: "a100-80g", Total: 1, Free: 0},
|
||||
}}})
|
||||
if len(hw) != 1 {
|
||||
t.Fatalf("pools: %+v", hw)
|
||||
}
|
||||
want := modelplan.Hardware{Class: "a100-80g-mig-3g.40gb", GPUMemGB: 40, GPUsPerNode: 2, Nodes: 1}
|
||||
if hw[0] != want {
|
||||
t.Errorf("pool %+v, want %+v", hw[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCtlNodesCountsANodesDevicesByClassWhenItHasMIG(t *testing.T) {
|
||||
mig := adminapi.NodeView{Classes: []string{"a100-80g-mig-3g.40gb", "a100-80g"}, ClassGPUs: []adminapi.ClassGPUs{
|
||||
{Class: "a100-80g-mig-3g.40gb", Total: 2, MIG: true}, {Class: "a100-80g", Total: 1}}}
|
||||
if got := classesColumn(mig); got != "a100-80g-mig-3g.40gb×2,a100-80g×1" {
|
||||
t.Errorf("MIG node: %q", got)
|
||||
}
|
||||
// A node of one kind of whole card reads as it always did.
|
||||
plain := adminapi.NodeView{Classes: []string{"h100-80g"}, ClassGPUs: []adminapi.ClassGPUs{{Class: "h100-80g", Total: 8}}}
|
||||
if got := classesColumn(plain); got != "h100-80g" {
|
||||
t.Errorf("plain node: %q", got)
|
||||
}
|
||||
}
|
||||
+35
-32
@@ -155,28 +155,41 @@ func hardwareFrom(nodes []adminapi.NodeView) []modelplan.Hardware {
|
||||
if len(n.Fabrics) > 0 {
|
||||
fabric = n.Fabrics[0]
|
||||
}
|
||||
class := ""
|
||||
if len(n.Classes) > 0 {
|
||||
class = n.Classes[0]
|
||||
// Each class of the node's devices is a pool of its own: a node with
|
||||
// a card in MIG mode has its instances' classes, and any whole card's
|
||||
// beside them. A node that does not count by class is its first
|
||||
// class, as it always was.
|
||||
kinds := n.ClassGPUs
|
||||
if len(kinds) == 0 {
|
||||
class := ""
|
||||
if len(n.Classes) > 0 {
|
||||
class = n.Classes[0]
|
||||
}
|
||||
kinds = []adminapi.ClassGPUs{{Class: class, Total: n.GPUTotal, Free: n.GPUFree}}
|
||||
}
|
||||
k := key{class, fabric}
|
||||
if pools[k] == nil {
|
||||
pools[k] = &modelplan.Hardware{Class: class, Fabric: fabric}
|
||||
}
|
||||
p := pools[k]
|
||||
p.Nodes++
|
||||
// The widest a single node could contribute, not the total: a tensor
|
||||
// group takes the same number of cards from every node it spans.
|
||||
if n.GPUFree > p.GPUsPerNode {
|
||||
p.GPUsPerNode = n.GPUFree
|
||||
}
|
||||
if n.MemoryGB > p.HostMemGB {
|
||||
p.HostMemGB = n.MemoryGB
|
||||
}
|
||||
// Card memory from the class label, which carries it by construction:
|
||||
// "h100-80g" is eighty gigabytes and says so.
|
||||
if class != "" && p.GPUMemGB == 0 {
|
||||
p.GPUMemGB = memFromClass(class)
|
||||
for _, kind := range kinds {
|
||||
class := kind.Class
|
||||
k := key{class, fabric}
|
||||
if pools[k] == nil {
|
||||
pools[k] = &modelplan.Hardware{Class: class, Fabric: fabric}
|
||||
}
|
||||
p := pools[k]
|
||||
p.Nodes++
|
||||
// The widest a single node could contribute, not the total: a
|
||||
// tensor group takes the same number of cards from every node it
|
||||
// spans.
|
||||
if kind.Free > p.GPUsPerNode {
|
||||
p.GPUsPerNode = kind.Free
|
||||
}
|
||||
if n.MemoryGB > p.HostMemGB {
|
||||
p.HostMemGB = n.MemoryGB
|
||||
}
|
||||
// Card memory from the class label, which carries it by
|
||||
// construction: "h100-80g" is eighty gigabytes and says so, and
|
||||
// "a100-80g-mig-3g.40gb" is an instance of forty.
|
||||
if class != "" && p.GPUMemGB == 0 {
|
||||
p.GPUMemGB = memFromClass(class)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -191,17 +204,7 @@ func hardwareFrom(nodes []adminapi.NodeView) []modelplan.Hardware {
|
||||
}
|
||||
|
||||
// memFromClass reads the card size out of a class label such as "h100-80g".
|
||||
func memFromClass(class string) int {
|
||||
i := strings.LastIndex(class, "-")
|
||||
if i < 0 || !strings.HasSuffix(class, "g") {
|
||||
return 0
|
||||
}
|
||||
var gb int
|
||||
if _, err := fmt.Sscanf(class[i+1:], "%dg", &gb); err != nil {
|
||||
return 0
|
||||
}
|
||||
return gb
|
||||
}
|
||||
func memFromClass(class string) int { return modelplan.MemFromClass(class) }
|
||||
|
||||
// catalogEntry renders a fit as the YAML that would serve it.
|
||||
func catalogEntry(m modelplan.Model, f modelplan.Fit, source string, context int) string {
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"github.com/charmbracelet/lipgloss"
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
@@ -1080,7 +1081,7 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
for _, r := range m.data.replicas {
|
||||
var nodes []string
|
||||
for _, s := range r.Shards {
|
||||
nodes = append(nodes, fmt.Sprintf("%s%v", s.Node, s.GPUs))
|
||||
nodes = append(nodes, fmt.Sprintf("%s%v", s.Node, discovery.DeviceLabels(s.GPUs)))
|
||||
}
|
||||
rows = append(rows, []string{r.ID, r.Model, r.State, r.Pool, strings.Join(nodes, " "),
|
||||
(time.Duration(r.AgeSec) * time.Second).Round(time.Second).String()})
|
||||
@@ -1224,14 +1225,31 @@ func (m consoleModel) gpuRows() [][]string {
|
||||
for _, n := range m.data.nodes {
|
||||
for _, g := range n.GPUs {
|
||||
busy, mem, temp, power := "—", "—", "—", "—"
|
||||
// A MIG instance under its card: its own memory, the card's heat
|
||||
// and power, and no utilisation, which no driver measures per
|
||||
// instance — said, rather than shown as the card's or as zero.
|
||||
mig := g.MIGProfile != ""
|
||||
if mig {
|
||||
busy = sDim.Render("n/a on MIG")
|
||||
if g.MemoryTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", float64(g.MemoryUsedMB)/1024, float64(g.MemoryTotalMB)/1024)
|
||||
}
|
||||
}
|
||||
card := g.Card
|
||||
if g.Label == "" {
|
||||
// A control plane older than MIG, which sends no card.
|
||||
card = g.Index
|
||||
}
|
||||
if n.Load != nil {
|
||||
for _, s := range n.Load.GPUs {
|
||||
if s.Index != g.Index {
|
||||
if s.Index != card {
|
||||
continue
|
||||
}
|
||||
busy = fmt.Sprintf("%.0f%%", s.UtilPercent)
|
||||
if s.MemoryTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024)
|
||||
if !mig {
|
||||
busy = fmt.Sprintf("%.0f%%", s.UtilPercent)
|
||||
if s.MemoryTotalMB > 0 {
|
||||
mem = fmt.Sprintf("%.1f/%.0f GB", s.MemoryUsedMB/1024, s.MemoryTotalMB/1024)
|
||||
}
|
||||
}
|
||||
if s.TempC > 0 {
|
||||
temp = fmt.Sprintf("%.0f°C", s.TempC)
|
||||
@@ -1245,7 +1263,12 @@ func (m consoleModel) gpuRows() [][]string {
|
||||
if holder == "" {
|
||||
holder = sDim.Render("free")
|
||||
}
|
||||
rows = append(rows, []string{n.ID, strconv.Itoa(g.Index), g.Name, busy, mem, temp, power, holder})
|
||||
label := g.Label
|
||||
if label == "" {
|
||||
// A control plane older than labels.
|
||||
label = strconv.Itoa(g.Index)
|
||||
}
|
||||
rows = append(rows, []string{n.ID, label, g.Name, busy, mem, temp, power, holder})
|
||||
}
|
||||
}
|
||||
return rows
|
||||
@@ -1266,6 +1289,15 @@ func (m consoleModel) nodeDetail(n adminapi.NodeView) string {
|
||||
}
|
||||
kv("disk", fmt.Sprintf("%d GB free of %d", n.DiskFreeGB, n.DiskTotalGB))
|
||||
kv("replicas", strconv.Itoa(n.Replicas))
|
||||
// By class, which on a node with a card in MIG mode is the instances'
|
||||
// profiles beside any whole card: what a profile's gpu_class can name.
|
||||
for _, c := range n.ClassGPUs {
|
||||
what := fmt.Sprintf("%d, %d free", c.Total, c.Free)
|
||||
if c.MIG {
|
||||
what += " · MIG instances, one to a replica"
|
||||
}
|
||||
kv(c.Class, what)
|
||||
}
|
||||
if len(n.Provision) > 0 {
|
||||
b.WriteString("\n " + lipgloss.NewStyle().Bold(true).Render("Engines") + "\n")
|
||||
for _, p := range n.Provision {
|
||||
|
||||
@@ -228,7 +228,12 @@ type NodeView struct {
|
||||
DiskFreeGB int `json:"disk_free_gb,omitempty"`
|
||||
DiskTotalGB int `json:"disk_total_gb,omitempty"`
|
||||
Classes []string `json:"gpu_classes,omitempty"`
|
||||
Fabrics []string `json:"fabrics,omitempty"`
|
||||
// ClassGPUs counts the node's devices by class, in the order of Classes.
|
||||
// A node with a card in MIG mode has several classes — each profile its
|
||||
// instances are cut to, and any whole card beside them — and GPUTotal
|
||||
// alone would add 3g.40gb instances to 80 GB cards.
|
||||
ClassGPUs []ClassGPUs `json:"gpus_by_class,omitempty"`
|
||||
Fabrics []string `json:"fabrics,omitempty"`
|
||||
|
||||
// Unhealthy lists GPUs kept out of placement and why, which is usually the
|
||||
// answer when a node looks idle but nothing will schedule onto it.
|
||||
@@ -255,10 +260,34 @@ type NodeView struct {
|
||||
IMEXChannels int `json:"imex_channels,omitempty"`
|
||||
}
|
||||
|
||||
// ClassGPUs is how many of a node's devices are of one class, and how many
|
||||
// of those are free or kept out of placement.
|
||||
type ClassGPUs struct {
|
||||
Class string `json:"class"`
|
||||
Total int `json:"total"`
|
||||
Free int `json:"free"`
|
||||
Unhealthy int `json:"unhealthy,omitempty"`
|
||||
// MIG marks a class of MIG instances, each of which takes a replica of
|
||||
// its own and never joins a tensor-parallel group.
|
||||
MIG bool `json:"mig,omitempty"`
|
||||
}
|
||||
|
||||
// GPUView is one card as a node's page shows it.
|
||||
type GPUView struct {
|
||||
Index int `json:"index"`
|
||||
Name string `json:"name"`
|
||||
// Label is the device as people are shown it: "3" for a whole card,
|
||||
// "0/mig2" for card 0's MIG instance 2.
|
||||
Label string `json:"label"`
|
||||
Class string `json:"class"`
|
||||
// Card is the physical card's index: the device's own, or for a MIG
|
||||
// instance its parent's, which is what the card's sensors are under.
|
||||
Card int `json:"card"`
|
||||
// MIGProfile is the instance's profile, like 3g.40gb; empty for a
|
||||
// whole card.
|
||||
MIGProfile string `json:"mig_profile,omitempty"`
|
||||
MemoryTotalMB int `json:"memory_total_mb,omitempty"`
|
||||
MemoryUsedMB int `json:"memory_used_mb,omitempty"`
|
||||
// ComputeCapability is what decides which engine builds run on the card:
|
||||
// CUDA 13 builds need 7.5, so a V100 (7.0) runs only older ones.
|
||||
ComputeCapability string `json:"compute_capability,omitempty"`
|
||||
@@ -556,15 +585,25 @@ func (a *API) Nodes() []NodeView {
|
||||
for _, w := range n.Weights {
|
||||
v.Models = append(v.Models, w.Path)
|
||||
}
|
||||
seen := map[string]struct{}{}
|
||||
seen := map[string]int{}
|
||||
free := inv.FreeGPUs(n.ID)
|
||||
for _, g := range n.GPUs {
|
||||
if _, dup := seen[g.Class()]; !dup {
|
||||
seen[g.Class()] = struct{}{}
|
||||
k, dup := seen[g.Class()]
|
||||
if !dup {
|
||||
k = len(v.ClassGPUs)
|
||||
seen[g.Class()] = k
|
||||
v.Classes = append(v.Classes, g.Class())
|
||||
v.ClassGPUs = append(v.ClassGPUs, ClassGPUs{Class: g.Class(), MIG: g.IsMIG()})
|
||||
}
|
||||
if !g.Health.OK() {
|
||||
c := &v.ClassGPUs[k]
|
||||
c.Total++
|
||||
switch {
|
||||
case !g.Health.OK():
|
||||
c.Unhealthy++
|
||||
v.Unhealthy = append(v.Unhealthy,
|
||||
fmt.Sprintf("gpu %d: %s", g.Index, g.Health.Problems[0]))
|
||||
fmt.Sprintf("gpu %s: %s", discovery.DeviceLabel(g.Index), g.Health.Problems[0]))
|
||||
case slices.Contains(free, g.Index):
|
||||
c.Free++
|
||||
}
|
||||
}
|
||||
if !n.LastSeen.IsZero() {
|
||||
@@ -574,7 +613,12 @@ func (a *API) Nodes() []NodeView {
|
||||
v.NVLinkDomain, v.IMEX, v.IMEXChannels = n.NVLinkDomain(), n.IMEX.State, n.IMEX.Channels
|
||||
for _, g := range n.GPUs {
|
||||
gv := GPUView{Index: g.Index, Name: g.Name, ComputeCapability: g.ComputeCapability,
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers}
|
||||
NUMA: g.NUMANode, NVLinkPeers: g.NVLinkPeers,
|
||||
Label: discovery.DeviceLabel(g.Index), Class: g.Class(), Card: g.CardIndex(),
|
||||
MemoryTotalMB: g.MemoryTotalMB, MemoryUsedMB: g.MemoryUsedMB}
|
||||
if g.MIG != nil {
|
||||
gv.MIGProfile = g.MIG.Profile
|
||||
}
|
||||
for hca, rel := range g.NICPaths {
|
||||
if gv.NearestNIC == "" || discovery.PathRank(rel) < discovery.PathRank(gv.NICPath) ||
|
||||
(discovery.PathRank(rel) == discovery.PathRank(gv.NICPath) && hca < gv.NearestNIC) {
|
||||
@@ -1619,6 +1663,13 @@ func (a *API) PoolNodes() []modelplan.PoolNode {
|
||||
p.Class = n.Classes[0]
|
||||
p.GPUMemGB = modelplan.MemFromClass(p.Class)
|
||||
}
|
||||
// By class, with each class's own count of devices: a node with a
|
||||
// card in MIG mode is several, and its first class alone would
|
||||
// count every instance and card as that one.
|
||||
for _, c := range n.ClassGPUs {
|
||||
p.Kinds = append(p.Kinds, modelplan.PoolKind{Class: c.Class,
|
||||
GPUMemGB: modelplan.MemFromClass(c.Class), GPUs: c.Total, FreeGPUs: c.Free})
|
||||
}
|
||||
if len(n.Fabrics) > 0 {
|
||||
p.Fabric = n.Fabrics[0]
|
||||
}
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// migNode has card 0 cut into two 3g.40gb instances and card 1 whole.
|
||||
func migNode(id string) inventory.Node {
|
||||
nd := inventory.Node{ID: id, State: inventory.Ready, AgentVersion: "v1", MemoryTotalMB: 512 * 1024}
|
||||
for dev := range 2 {
|
||||
nd.GPUs = append(nd.GPUs, discovery.GPU{
|
||||
Index: discovery.MIGIndex(0, dev), Name: "NVIDIA A100-SXM4-80GB MIG 3g.40gb", UUID: "MIG-x",
|
||||
MemoryTotalMB: 40192, MemoryUsedMB: 37,
|
||||
MIG: &discovery.MIGDevice{ParentIndex: 0, ParentMemoryTotalMB: 81920, Device: dev, Profile: "3g.40gb"},
|
||||
})
|
||||
}
|
||||
nd.GPUs = append(nd.GPUs, discovery.GPU{Index: 1, Name: "NVIDIA A100-SXM4-80GB", MemoryTotalMB: 81920})
|
||||
return nd
|
||||
}
|
||||
|
||||
func TestANodeWithACardInMIGModeCountsItsDevicesByClass(t *testing.T) {
|
||||
a, ctl := setup(t, testModels(), migNode("a100"))
|
||||
ctl.Inventory().Allocate("a100", inventory.Allocation{ShardID: "s", ReplicaID: "r", GPUIndices: []int{1000}})
|
||||
|
||||
n := a.Nodes()[0]
|
||||
want := []ClassGPUs{
|
||||
{Class: "a100-80g-mig-3g.40gb", Total: 2, Free: 1, MIG: true},
|
||||
{Class: "a100-80g", Total: 1, Free: 1},
|
||||
}
|
||||
if !reflect.DeepEqual(n.ClassGPUs, want) {
|
||||
t.Errorf("by class: %+v\nwant %+v", n.ClassGPUs, want)
|
||||
}
|
||||
// Each device as people are shown it: the instance under its card, with
|
||||
// its own memory, and the card its sensors are read under.
|
||||
g := n.GPUs[1]
|
||||
if g.Label != "0/mig1" || g.Card != 0 || g.MIGProfile != "3g.40gb" || g.MemoryTotalMB != 40192 ||
|
||||
g.Class != "a100-80g-mig-3g.40gb" {
|
||||
t.Errorf("instance view: %+v", g)
|
||||
}
|
||||
if w := n.GPUs[2]; w.Label != "1" || w.Card != 1 || w.MIGProfile != "" {
|
||||
t.Errorf("whole card view: %+v", w)
|
||||
}
|
||||
|
||||
// The fleet summary attributes each device to its own class, where it
|
||||
// once put every card of a node under the node's only class.
|
||||
s := a.Summary()
|
||||
got := map[string][2]int{}
|
||||
for _, c := range s.GPUs {
|
||||
got[c.Class] = [2]int{c.Total, c.Used}
|
||||
}
|
||||
if !reflect.DeepEqual(got, map[string][2]int{"a100-80g-mig-3g.40gb": {2, 1}, "a100-80g": {1, 0}}) {
|
||||
t.Errorf("summary by class: %v", got)
|
||||
}
|
||||
|
||||
// And a plan sees both classes, each with its own count.
|
||||
pool := a.PoolNodes()
|
||||
if len(pool) != 1 || len(pool[0].Kinds) != 2 || pool[0].Kinds[0].GPUMemGB != 40 ||
|
||||
pool[0].Kinds[0].FreeGPUs != 1 || pool[0].Kinds[1].GPUMemGB != 80 {
|
||||
t.Errorf("pool: %+v", pool)
|
||||
}
|
||||
}
|
||||
@@ -148,12 +148,14 @@ func (a *API) Summary() Summary {
|
||||
byClass[c] = &ClassUse{Class: c}
|
||||
}
|
||||
}
|
||||
// A node reports one class in practice; attribute its cards to it.
|
||||
if len(n.Classes) == 1 {
|
||||
u := byClass[n.Classes[0]]
|
||||
u.Total += n.GPUTotal
|
||||
u.Unhealthy += len(n.Unhealthy)
|
||||
u.Used += n.GPUTotal - n.GPUFree - len(n.Unhealthy)
|
||||
// Each class its own devices, which a node with a card in MIG mode
|
||||
// needs: its instances are one class or several, and any whole card
|
||||
// beside them another.
|
||||
for _, c := range n.ClassGPUs {
|
||||
u := byClass[c.Class]
|
||||
u.Total += c.Total
|
||||
u.Unhealthy += c.Unhealthy
|
||||
u.Used += c.Total - c.Free - c.Unhealthy
|
||||
}
|
||||
}
|
||||
for _, u := range byClass {
|
||||
|
||||
@@ -71,6 +71,7 @@ func NodeFromRegister(r *llmgwv1.Register) inventory.Node {
|
||||
TemperatureC: int(g.GetTemperatureC()),
|
||||
Problems: g.GetProblems(),
|
||||
},
|
||||
MIG: migFrom(g.GetMig()),
|
||||
})
|
||||
}
|
||||
n.IMEX = discovery.IMEX{State: r.GetImex().GetState(), Channels: int(r.GetImex().GetChannels())}
|
||||
@@ -131,6 +132,13 @@ func RegisterFromInventory(nodeID, hostname, mgmtAddr, agentVersion string,
|
||||
last.FabricState, last.FabricStatus, last.ClusterUuid = f.State, f.Status, f.ClusterUUID
|
||||
last.CliqueId, last.FabricBandwidth = f.CliqueID, f.Bandwidth
|
||||
}
|
||||
if m := g.MIG; m != nil {
|
||||
r.Gpus[len(r.Gpus)-1].Mig = &llmgwv1.MIGDevice{
|
||||
ParentIndex: int32(m.ParentIndex), ParentUuid: m.ParentUUID,
|
||||
ParentMemoryTotalMb: int32(m.ParentMemoryTotalMB), Device: int32(m.Device),
|
||||
Profile: m.Profile, GpuInstance: int32(m.GPUInstance), ComputeInstance: int32(m.ComputeInstance),
|
||||
}
|
||||
}
|
||||
}
|
||||
r.Imex = &llmgwv1.IMEX{State: inv.IMEX.State, Channels: int32(inv.IMEX.Channels)}
|
||||
for _, f := range fabrics {
|
||||
@@ -286,6 +294,19 @@ func int32Slice(v []int) []int32 {
|
||||
return out
|
||||
}
|
||||
|
||||
// migFrom is a MIG instance's parentage as the wire carries it, nil for a
|
||||
// whole card and from every agent older than MIG support.
|
||||
func migFrom(m *llmgwv1.MIGDevice) *discovery.MIGDevice {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
return &discovery.MIGDevice{
|
||||
ParentIndex: int(m.GetParentIndex()), ParentUUID: m.GetParentUuid(),
|
||||
ParentMemoryTotalMB: int(m.GetParentMemoryTotalMb()), Device: int(m.GetDevice()),
|
||||
Profile: m.GetProfile(), GPUInstance: int(m.GetGpuInstance()), ComputeInstance: int(m.GetComputeInstance()),
|
||||
}
|
||||
}
|
||||
|
||||
// fabricFrom is a GPU's NVLink fabric as the wire carries it, nil when the
|
||||
// driver reported none.
|
||||
func fabricFrom(state, status, cluster, clique, bandwidth string) *discovery.GPUFabric {
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
func TestAMIGInstanceKeepsItsParentageOnTheWire(t *testing.T) {
|
||||
inst := discovery.GPU{Index: 1001, UUID: "MIG-bbb", Name: "NVIDIA A100-SXM4-80GB MIG 3g.40gb", MemoryTotalMB: 40192,
|
||||
MIG: &discovery.MIGDevice{ParentIndex: 0, ParentUUID: "GPU-card", ParentMemoryTotalMB: 81920,
|
||||
Device: 1, Profile: "3g.40gb", GPUInstance: 1, ComputeInstance: 0}}
|
||||
whole := discovery.GPU{Index: 1, UUID: "GPU-whole", Name: "NVIDIA A100-SXM4-80GB", MemoryTotalMB: 81920}
|
||||
|
||||
reg := RegisterFromInventory("n1", "h", "10.0.0.1", "v", nil, discovery.Inventory{GPUs: []discovery.GPU{inst, whole}}, nil)
|
||||
n := NodeFromRegister(reg)
|
||||
if len(n.GPUs) != 2 {
|
||||
t.Fatalf("got %d GPUs", len(n.GPUs))
|
||||
}
|
||||
if !reflect.DeepEqual(n.GPUs[0].MIG, inst.MIG) || n.GPUs[0].Class() != "a100-80g-mig-3g.40gb" {
|
||||
t.Errorf("the instance arrived as %s %+v", n.GPUs[0].Class(), n.GPUs[0].MIG)
|
||||
}
|
||||
if n.GPUs[1].MIG != nil || n.GPUs[1].Class() != "a100-80g" {
|
||||
t.Errorf("the whole card arrived as %s %+v", n.GPUs[1].Class(), n.GPUs[1].MIG)
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
)
|
||||
@@ -701,6 +702,17 @@ func (m Model) validate() error {
|
||||
}
|
||||
continue
|
||||
}
|
||||
// A MIG instance is a device of its own to CUDA, but a process sees
|
||||
// only one, and instances have no NVLink or peer-to-peer between
|
||||
// them: a group of them is neither tensor- nor layer-parallel, it is
|
||||
// a replica that fails at start. One instance, one node, refused here
|
||||
// rather than placed and found out.
|
||||
if discovery.IsMIGClass(p.GPUClass) && (p.GPUsPerNode > 1 || p.Nodes > 1 || p.TP > 1 || p.PP > 1) {
|
||||
return fmt.Errorf("model %q profile %q: %s is a MIG instance, and a replica runs on exactly one: "+
|
||||
"a process can open only one MIG device, and instances have no NVLink between them. "+
|
||||
"Give it tp 1 and one GPU on one node, or use a whole-card class or a larger MIG profile",
|
||||
m.Name, p.Name, p.GPUClass)
|
||||
}
|
||||
// llama.cpp on GPUs splits a model across one node's cards by layer,
|
||||
// and how many is gpus_per_node. Not tp: llama.cpp has no tensor
|
||||
// parallelism, and a tp it was given was a width nothing placed — the
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAProfileOnAMIGClassRunsOnExactlyOneInstance(t *testing.T) {
|
||||
// One instance, by default or said outright, is a profile like any other.
|
||||
for _, p := range []Profile{
|
||||
{Name: "vllm", Engine: "vllm", GPUClass: "a100-80g-mig-3g.40gb"},
|
||||
{Name: "llama", Engine: "llamacpp", GPUClass: "a100-80g-mig-3g.40gb", GPUsPerNode: 1, TP: 1},
|
||||
} {
|
||||
c, err := New([]Model{model("m", p)})
|
||||
if err != nil {
|
||||
t.Fatalf("%s on one MIG instance: %v", p.Name, err)
|
||||
}
|
||||
m, _ := c.Resolve("m")
|
||||
if pl := m.PlacementProfiles()[0]; pl.GPUsPerNode != 1 || pl.Nodes != 1 {
|
||||
t.Errorf("%s: placement asked for %+v", p.Name, pl)
|
||||
}
|
||||
}
|
||||
// More than one is refused where the model is defined: a process opens
|
||||
// one MIG device, and instances have no NVLink between them, so the
|
||||
// replica would be placed and then fail at start.
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp2", Engine: "vllm", GPUClass: "a100-80g-mig-3g.40gb", TP: 2},
|
||||
{Name: "pp2", Engine: "vllm", GPUClass: "a100-80g-mig-3g.40gb", TP: 1, PP: 2, GPUsPerNode: 2},
|
||||
{Name: "nodes", Engine: "vllm", GPUClass: "a100-80g-mig-3g.40gb", TP: 2, Nodes: 2, GPUsPerNode: 1},
|
||||
{Name: "layers", Engine: "llamacpp", GPUClass: "a100-80g-mig-3g.40gb", GPUsPerNode: 2},
|
||||
{Name: "old-tp", Engine: "llamacpp", GPUClass: "a100-80g-mig-3g.40gb", TP: 2},
|
||||
} {
|
||||
_, err := New([]Model{model("m", p)})
|
||||
if err == nil || !strings.Contains(err.Error(), "MIG instance") {
|
||||
t.Errorf("%s: %v", p.Name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -430,8 +430,18 @@ func (c *Collector) Collect(ch chan<- prometheus.Metric) {
|
||||
byClass[class]++
|
||||
labels := []string{n.ID, strconv.Itoa(g.Index), class}
|
||||
|
||||
s, fresh := sensorFor(n.Load, g.Index, now)
|
||||
if fresh && s.Busy() {
|
||||
// The sensors are the card's. A MIG instance shares its card's
|
||||
// heat and power, but not its memory, and has no utilisation of
|
||||
// its own that any driver reports: its memory is the instance's
|
||||
// own reading, and its utilisation is left out rather than
|
||||
// published as the card's or as zero.
|
||||
s, fresh := sensorFor(n.Load, g.CardIndex(), now)
|
||||
if g.IsMIG() {
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryTotal, prometheus.GaugeValue,
|
||||
float64(g.MemoryTotalMB)*1024*1024, labels...)
|
||||
ch <- prometheus.MustNewConstMetric(c.gpuMemoryUsed, prometheus.GaugeValue,
|
||||
float64(g.MemoryUsedMB)*1024*1024, labels...)
|
||||
} else if fresh && s.Busy() {
|
||||
// What the card is doing now, sampled every heartbeat.
|
||||
c.gpuBusy(ch, s, labels)
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package fleetmetrics
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
func TestAMIGInstanceHasItsOwnMemoryItsCardsHeatAndNoUtilisation(t *testing.T) {
|
||||
// Card 0 in MIG mode: the sensor sample is the card's, 60 GB used across
|
||||
// both instances and a utilisation the driver does not give per instance.
|
||||
inst := discovery.GPU{Index: 1001, Name: "NVIDIA A100-SXM4-80GB MIG 3g.40gb", MemoryTotalMB: 40192, MemoryUsedMB: 21000,
|
||||
MIG: &discovery.MIGDevice{ParentIndex: 0, ParentMemoryTotalMB: 81920, Device: 1, Profile: "3g.40gb"}}
|
||||
n := inventory.Node{ID: "g1", State: inventory.Ready, GPUs: []discovery.GPU{inst},
|
||||
Load: &inventory.Load{At: time.Now(), GPUs: []inventory.GPUSensor{{Index: 0, TempC: 61, PowerW: 250,
|
||||
UtilPercent: 0, MemoryUsedMB: 60000, MemoryTotalMB: 81920}}}}
|
||||
out := gather(t, fakeSource{nodes: []inventory.Node{n}})
|
||||
labels := "class=a100-80g-mig-3g.40gb gpu=1001 node=g1"
|
||||
for _, want := range []string{
|
||||
"llmgw_gpu_memory_used_bytes " + labels + " -> 22020096000\n",
|
||||
"llmgw_gpu_memory_total_bytes " + labels + " -> 42144366592\n",
|
||||
"llmgw_gpu_temperature_celsius " + labels + " -> 61\n",
|
||||
"llmgw_gpu_power_watts " + labels + " -> 250\n",
|
||||
} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("missing %s in:\n%s", want, out)
|
||||
}
|
||||
}
|
||||
// Not the card's zero, not anything: absent.
|
||||
if strings.Contains(out, "llmgw_gpu_utilization_percent "+labels) {
|
||||
t.Errorf("a utilisation was published for a MIG instance:\n%s", out)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
package placement
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/inventory"
|
||||
)
|
||||
|
||||
// migNode is an A100 80GB node as discovery reports it with card 0 cut into
|
||||
// two 3g.40gb instances and card 1 whole.
|
||||
func migNode(id string) inventory.Node {
|
||||
card := discovery.GPU{Name: "NVIDIA A100-SXM4-80GB", MemoryTotalMB: 81920, ComputeCapability: "8.0"}
|
||||
nd := inventory.Node{ID: id, State: inventory.Ready}
|
||||
for dev, uuid := range []string{"MIG-c6d4f1ef", "MIG-0f7c3a2e"} {
|
||||
g := card
|
||||
g.Index, g.UUID, g.MemoryTotalMB = discovery.MIGIndex(0, dev), uuid, 40192
|
||||
g.Name += " MIG 3g.40gb"
|
||||
g.MIG = &discovery.MIGDevice{ParentIndex: 0, ParentMemoryTotalMB: 81920, Device: dev, Profile: "3g.40gb"}
|
||||
nd.GPUs = append(nd.GPUs, g)
|
||||
}
|
||||
whole := card
|
||||
whole.Index, whole.UUID = 1, "GPU-whole"
|
||||
// NVLink to the card in MIG mode, as topo -m still reports it: nothing
|
||||
// may take that for a link an instance can use.
|
||||
whole.NVLinkPeers = []int{0}
|
||||
nd.GPUs = append(nd.GPUs, whole)
|
||||
return nd
|
||||
}
|
||||
|
||||
func TestAModelIsPlacedOnAMIGInstanceOfItsClass(t *testing.T) {
|
||||
st := store(migNode("n1"))
|
||||
s := New(st)
|
||||
p := Profile{Name: "mig", GPUClass: "a100-80g-mig-3g.40gb", GPUsPerNode: 1}
|
||||
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement == nil {
|
||||
t.Fatalf("no placement:\n%s", res.Explain())
|
||||
}
|
||||
got := res.Placement.Nodes[0].GPUIndices
|
||||
if !slices.Equal(got, []int{1000}) {
|
||||
t.Fatalf("placed on %v, want MIG instance 1000", got)
|
||||
}
|
||||
if res.Placement.Pool != "none/a100-80g-mig-3g.40gb" {
|
||||
t.Errorf("pool %q", res.Placement.Pool)
|
||||
}
|
||||
|
||||
// The next replica takes the other instance, and the one after finds
|
||||
// none: the whole card beside them is another class.
|
||||
st.Allocate("n1", inventory.Allocation{ShardID: "s1", ReplicaID: "r1", GPUIndices: got})
|
||||
res = s.Place([]Profile{p}, nil)
|
||||
if res.Placement == nil || !slices.Equal(res.Placement.Nodes[0].GPUIndices, []int{1001}) {
|
||||
t.Fatalf("second replica: %+v\n%s", res.Placement, res.Explain())
|
||||
}
|
||||
st.Allocate("n1", inventory.Allocation{ShardID: "s2", ReplicaID: "r2", GPUIndices: []int{1001}})
|
||||
if res = s.Place([]Profile{p}, nil); res.Placement != nil {
|
||||
t.Errorf("a third replica was placed on %v", res.Placement.Nodes[0].GPUIndices)
|
||||
}
|
||||
|
||||
// A whole-card profile takes the whole card, never an instance.
|
||||
res = s.Place([]Profile{{Name: "whole", GPUClass: "a100-80g", GPUsPerNode: 1}}, nil)
|
||||
if res.Placement == nil || !slices.Equal(res.Placement.Nodes[0].GPUIndices, []int{1}) {
|
||||
t.Errorf("whole-card profile: %+v\n%s", res.Placement, res.Explain())
|
||||
}
|
||||
}
|
||||
|
||||
func TestMoreThanOneMIGInstanceForAReplicaIsRefused(t *testing.T) {
|
||||
// Two instances are free on one node, which is what a tp 2 profile would
|
||||
// otherwise be given — and fail at start, since a process opens only one
|
||||
// MIG device and instances have no NVLink between them.
|
||||
s := New(store(migNode("n1"), migNode("n2")))
|
||||
for _, p := range []Profile{
|
||||
{Name: "tp2", GPUClass: "a100-80g-mig-3g.40gb", GPUsPerNode: 2},
|
||||
{Name: "two-nodes", GPUClass: "a100-80g-mig-3g.40gb", GPUsPerNode: 1, Nodes: 2},
|
||||
} {
|
||||
res := s.Place([]Profile{p}, nil)
|
||||
if res.Placement != nil {
|
||||
t.Errorf("%s was placed: %+v", p.Name, res.Placement.Nodes)
|
||||
continue
|
||||
}
|
||||
if !strings.Contains(res.Explain(), "a replica runs on exactly one") {
|
||||
t.Errorf("%s refused without saying why:\n%s", p.Name, res.Explain())
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -156,6 +156,15 @@ func (s *Scheduler) Place(profiles []Profile, avoid []string) Result {
|
||||
|
||||
for _, raw := range ordered {
|
||||
p := raw.normalize()
|
||||
// The catalog refuses these; a profile that reaches here some other
|
||||
// way is refused again rather than placed on instances that cannot
|
||||
// work together. MIG instances are placed one to a replica.
|
||||
if discovery.IsMIGClass(p.GPUClass) && (p.GPUsPerNode > 1 || p.Nodes > 1) {
|
||||
res.Rejections = append(res.Rejections, Rejection{Profile: p.Name, Pool: "none/" + p.GPUClass,
|
||||
Reason: fmt.Sprintf("asks for %d MIG instances on %d nodes, but a replica runs on exactly one",
|
||||
p.GPUsPerNode*p.Nodes, p.Nodes)})
|
||||
continue
|
||||
}
|
||||
if p.Nodes == 1 {
|
||||
if pl, rej := s.placeSingleNode(p, avoid); pl != nil {
|
||||
res.Placement = pl
|
||||
|
||||
@@ -17,6 +17,7 @@ import (
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -57,6 +58,10 @@ type Facts struct {
|
||||
GPUError string `json:"gpu_error,omitempty"`
|
||||
TopoRaw string `json:"topo_raw,omitempty"`
|
||||
TopoErr string `json:"topo_error,omitempty"`
|
||||
// MIG is each MIG-capable card's mode and instances; MIGErr why it could
|
||||
// not be read.
|
||||
MIG []discovery.MIGCard `json:"mig,omitempty"`
|
||||
MIGErr string `json:"mig_error,omitempty"`
|
||||
// NVSwitch is set when the machine has NVSwitch devices, which need the
|
||||
// fabric manager running before CUDA will start.
|
||||
NVSwitch bool `json:"nvswitch"`
|
||||
@@ -150,6 +155,16 @@ func Collect(ctx context.Context, o Options) Facts {
|
||||
}
|
||||
f.GPUs = gpus
|
||||
f.IMEX = discovery.DiscoverIMEX(ctx)
|
||||
if slices.ContainsFunc(gpus, func(g discovery.GPU) bool {
|
||||
cc, _ := strconv.ParseFloat(g.ComputeCapability, 64)
|
||||
return g.IsMIG() || cc >= 8
|
||||
}) {
|
||||
if mig, err := discovery.DiscoverMIG(ctx, smi); err != nil {
|
||||
f.MIGErr = err.Error()
|
||||
} else {
|
||||
f.MIG = mig
|
||||
}
|
||||
}
|
||||
}
|
||||
if m, _ := filepath.Glob("/dev/nvidia-nvswitch*"); len(m) > 0 {
|
||||
f.NVSwitch = true
|
||||
@@ -367,17 +382,22 @@ func Evaluate(f Facts) []Finding {
|
||||
add(OK, "gpu", fmt.Sprintf("%d GPUs: %s", len(f.GPUs), gpuNames(f.GPUs)), "")
|
||||
for _, g := range f.GPUs {
|
||||
if !g.Health.OK() {
|
||||
add(Warn, "gpu", fmt.Sprintf("GPU %d is kept out of placement: %s", g.Index, strings.Join(g.Health.Problems, "; ")), "")
|
||||
add(Warn, "gpu", fmt.Sprintf("GPU %s is kept out of placement: %s",
|
||||
discovery.DeviceLabel(g.Index), strings.Join(g.Health.Problems, "; ")), "")
|
||||
}
|
||||
}
|
||||
}
|
||||
evaluateMIG(f, add)
|
||||
if f.NVSwitch && f.FabricManager != "active" {
|
||||
add(Fail, "nvlink", "this machine has NVSwitch but nvidia-fabricmanager is "+f.FabricManager+
|
||||
": CUDA will not initialise on its GPUs",
|
||||
"systemctl enable --now nvidia-fabricmanager (its version must match the driver's)")
|
||||
}
|
||||
if len(f.GPUs) > 1 {
|
||||
pairs, groups := nvlinkGroups(f.GPUs)
|
||||
// NVLink is between whole cards: a MIG instance has none to report, and
|
||||
// counting instances would call a node of MIG cards one without NVLink.
|
||||
cards := slices.DeleteFunc(slices.Clone(f.GPUs), discovery.GPU.IsMIG)
|
||||
if len(cards) > 1 {
|
||||
pairs, groups := nvlinkGroups(cards)
|
||||
switch {
|
||||
case f.TopoErr != "":
|
||||
add(Warn, "nvlink", "could not read nvidia-smi topo -m: "+f.TopoErr+
|
||||
@@ -393,7 +413,7 @@ func Evaluate(f Facts) []Finding {
|
||||
|
||||
// --- multi-node NVLink ---
|
||||
domain := ""
|
||||
for _, g := range f.GPUs {
|
||||
for _, g := range cards {
|
||||
fb := g.Fabric
|
||||
if fb == nil || strings.EqualFold(fb.State, "Not Supported") {
|
||||
continue
|
||||
@@ -662,6 +682,64 @@ func rank(l Level) int {
|
||||
return 3
|
||||
}
|
||||
|
||||
// evaluateMIG reports each MIG-capable card's mode and instances. Quiet for a
|
||||
// card with MIG off, which is every card on most machines; what it warns of
|
||||
// are the states in which a card is in MIG mode and nothing can run on it.
|
||||
func evaluateMIG(f Facts, add func(Level, string, string, string)) {
|
||||
if f.MIGErr != "" {
|
||||
add(Warn, "mig", "could not read the cards' MIG state: "+f.MIGErr, "run nvidia-smi -q by hand")
|
||||
return
|
||||
}
|
||||
for _, c := range f.MIG {
|
||||
card := fmt.Sprintf("GPU %d (%s)", c.Index, c.Name)
|
||||
if c.Enabled != c.PendingEnabled {
|
||||
want := "off"
|
||||
if c.PendingEnabled {
|
||||
want = "on"
|
||||
}
|
||||
add(Warn, "mig", fmt.Sprintf("%s has MIG turning %s at its next reset, and until then runs as before", card, want),
|
||||
fmt.Sprintf("reset the card when nothing runs on it: nvidia-smi -i %d -r (or reboot)", c.Index))
|
||||
}
|
||||
if !c.Enabled {
|
||||
continue
|
||||
}
|
||||
// A GPU instance with no compute instance is carved but not usable:
|
||||
// it is not a device until one is made inside it.
|
||||
var bare []string
|
||||
for _, gi := range c.GPUInstances {
|
||||
has := slices.ContainsFunc(c.ComputeInstances, func(ci discovery.MIGInstance) bool {
|
||||
return ci.GPUInstance == gi.GPUInstance
|
||||
})
|
||||
if !has {
|
||||
bare = append(bare, fmt.Sprintf("%d (%s)", gi.GPUInstance, gi.Profile))
|
||||
}
|
||||
}
|
||||
switch {
|
||||
case len(c.Instances) == 0 && len(c.GPUInstances) == 0:
|
||||
add(Warn, "mig", card+" is in MIG mode with no instances created: nothing can run on it",
|
||||
fmt.Sprintf("create instances, for example nvidia-smi mig -i %d -cgi 3g.40gb,3g.40gb -C "+
|
||||
"(nvidia-smi mig -i %d -lgip lists the profiles), or turn MIG off: nvidia-smi -i %d -mig 0",
|
||||
c.Index, c.Index, c.Index))
|
||||
case len(c.Instances) == 0:
|
||||
add(Warn, "mig", fmt.Sprintf("%s is in MIG mode with GPU instances %s but no compute instances: nothing can run on it",
|
||||
card, strings.Join(bare, ", ")),
|
||||
fmt.Sprintf("create them: nvidia-smi mig -i %d -cci", c.Index))
|
||||
default:
|
||||
var names []string
|
||||
for _, in := range c.Instances {
|
||||
names = append(names, fmt.Sprintf("%d: %s %s", in.Device, in.Profile, in.UUID))
|
||||
}
|
||||
add(OK, "mig", fmt.Sprintf("%s is in MIG mode with %d instance(s), each a device of its own: %s",
|
||||
card, len(c.Instances), strings.Join(names, "; ")), "")
|
||||
if len(bare) > 0 {
|
||||
add(Warn, "mig", fmt.Sprintf("%s has GPU instances %s with no compute instance, which are not devices yet",
|
||||
card, strings.Join(bare, ", ")),
|
||||
fmt.Sprintf("create them: nvidia-smi mig -i %d -cci, or destroy the GPU instances", c.Index))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func gpuNames(gpus []discovery.GPU) string {
|
||||
count := map[string]int{}
|
||||
var order []string
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
package doctor
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// migFacts is the healthy node with card 0 in MIG mode, as given.
|
||||
func migFacts(card discovery.MIGCard) Facts {
|
||||
f := healthy()
|
||||
f.MIG = []discovery.MIGCard{card}
|
||||
return f
|
||||
}
|
||||
|
||||
func TestTheDoctorListsACardsMIGInstances(t *testing.T) {
|
||||
fs := Evaluate(migFacts(discovery.MIGCard{Index: 0, Name: "NVIDIA A100-SXM4-80GB", Enabled: true, PendingEnabled: true,
|
||||
Instances: []discovery.MIGListed{{Device: 0, Profile: "3g.40gb", UUID: "MIG-aaa"}, {Device: 1, Profile: "3g.40gb", UUID: "MIG-bbb"}},
|
||||
GPUInstances: []discovery.MIGInstance{{GPU: 0, Profile: "3g.40gb", GPUInstance: 1, ComputeInstance: -1},
|
||||
{GPU: 0, Profile: "3g.40gb", GPUInstance: 2, ComputeInstance: -1}},
|
||||
ComputeInstances: []discovery.MIGInstance{{GPU: 0, Profile: "3g.40gb", GPUInstance: 1},
|
||||
{GPU: 0, Profile: "3g.40gb", GPUInstance: 2}},
|
||||
}))
|
||||
has(t, fs, OK, "GPU 0 (NVIDIA A100-SXM4-80GB) is in MIG mode with 2 instance(s)")
|
||||
has(t, fs, OK, "1: 3g.40gb MIG-bbb")
|
||||
for _, f := range fs {
|
||||
if f.Area == "mig" && f.Level != OK {
|
||||
t.Errorf("a card with its instances made was warned of: %+v", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheDoctorWarnsOfMIGEnabledWithNoInstances(t *testing.T) {
|
||||
// The state a card is left in by nvidia-smi -mig 1 alone: in MIG mode,
|
||||
// so CUDA will not open it whole, and nothing carved to open instead.
|
||||
fs := Evaluate(migFacts(discovery.MIGCard{Index: 0, Name: "NVIDIA A100-SXM4-80GB", Enabled: true, PendingEnabled: true}))
|
||||
has(t, fs, Warn, "is in MIG mode with no instances created")
|
||||
for _, f := range fs {
|
||||
if f.Area == "mig" && f.Level == Warn && !strings.Contains(f.Fix, "nvidia-smi mig -i 0 -cgi") {
|
||||
t.Errorf("the fix does not say how to create them: %q", f.Fix)
|
||||
}
|
||||
}
|
||||
|
||||
// GPU instances with no compute instance in them are not devices either.
|
||||
fs = Evaluate(migFacts(discovery.MIGCard{Index: 0, Name: "A100", Enabled: true, PendingEnabled: true,
|
||||
GPUInstances: []discovery.MIGInstance{{GPU: 0, Profile: "7g.80gb", GPUInstance: 0, ComputeInstance: -1}}}))
|
||||
has(t, fs, Warn, "GPU instances 0 (7g.80gb) but no compute instances")
|
||||
|
||||
// A mode change waiting for a reset.
|
||||
fs = Evaluate(migFacts(discovery.MIGCard{Index: 0, Name: "A100", Enabled: false, PendingEnabled: true}))
|
||||
has(t, fs, Warn, "MIG turning on at its next reset")
|
||||
}
|
||||
|
||||
func TestACardWithMIGOffIsNotMentioned(t *testing.T) {
|
||||
fs := Evaluate(migFacts(discovery.MIGCard{Index: 0, Name: "A100"}))
|
||||
for _, f := range fs {
|
||||
if f.Area == "mig" {
|
||||
t.Errorf("a card with MIG off was reported: %+v", f)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNVLinkIsJudgedBetweenWholeCardsOnly(t *testing.T) {
|
||||
// Every card of the node in MIG mode: the instances have no NVLink, and
|
||||
// that is not a node "without NVLink" to warn about.
|
||||
f := healthy()
|
||||
f.GPUs = nil
|
||||
for card := range 2 {
|
||||
for dev := range 2 {
|
||||
f.GPUs = append(f.GPUs, discovery.GPU{Index: discovery.MIGIndex(card, dev), Name: "A100 MIG 3g.40gb",
|
||||
MIG: &discovery.MIGDevice{ParentIndex: card, Device: dev, Profile: "3g.40gb"}})
|
||||
}
|
||||
}
|
||||
for _, x := range Evaluate(f) {
|
||||
if x.Area == "nvlink" && x.Level != OK {
|
||||
t.Errorf("MIG instances were judged for NVLink: %+v", x)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -132,7 +132,14 @@ func fitsOn(m Model, h Hardware, d Demand, p Precision, context int) []Fit {
|
||||
// Powers of two only. Tensor parallelism divides attention heads across
|
||||
// ranks, and an engine that cannot divide them evenly refuses the profile —
|
||||
// so offering tp=3 would produce a plan that fails at launch.
|
||||
for width := 1; width <= h.GPUsPerNode*h.Nodes; width *= 2 {
|
||||
//
|
||||
// One wide on MIG instances, however many there are: a process opens
|
||||
// only one, and they have no NVLink between them.
|
||||
widest := h.GPUsPerNode * h.Nodes
|
||||
if isMIG(h.Class) {
|
||||
widest = min(widest, 1)
|
||||
}
|
||||
for width := 1; width <= widest; width *= 2 {
|
||||
nodes := 1
|
||||
perNode := width
|
||||
if width > h.GPUsPerNode {
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
package modelplan
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const mig3g = "a100-80g-mig-3g.40gb"
|
||||
|
||||
func TestAMIGInstanceIsSizedByItsProfileNotItsCard(t *testing.T) {
|
||||
for class, want := range map[string]int{
|
||||
mig3g: 40, "h100-80g-mig-1g.10gb": 10, "h100-80g-mig-1g.10gb-me": 10, "a100-80g": 80, "a100-80g-mig-": 0,
|
||||
} {
|
||||
if got := MemFromClass(class); got != want {
|
||||
t.Errorf("MemFromClass(%q) = %d, want %d", class, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelThatFitsOneMIGInstanceIsPlannedOnOne(t *testing.T) {
|
||||
// Four instances free on each of two nodes on a fabric: a 3B fits one,
|
||||
// and a 70B, which no single instance holds, is not spread over several
|
||||
// — a replica cannot use more than one.
|
||||
hw := []Hardware{{Class: mig3g, GPUMemGB: 40, GPUsPerNode: 4, Nodes: 2, Fabric: "ib-a"}}
|
||||
fits := Fits(coder3B, hw, 32768, []Precision{BF16})
|
||||
if len(fits) == 0 {
|
||||
t.Fatal("a 3B does not fit a 40 GB MIG instance")
|
||||
}
|
||||
for _, f := range fits {
|
||||
if f.TP != 1 || f.Nodes != 1 {
|
||||
t.Errorf("planned across MIG instances: %s", f.Describe())
|
||||
}
|
||||
}
|
||||
if fits := Fits(llama70B, hw, 8192, []Precision{BF16}); len(fits) != 0 {
|
||||
t.Errorf("a 70B was planned on MIG instances: %s", fits[0].Describe())
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeWithACardInMIGModeOffersEachOfItsClasses(t *testing.T) {
|
||||
// Card 0 cut into two 3g.40gb instances, card 1 whole: two classes on
|
||||
// one node, each with its own count.
|
||||
nodes := []PoolNode{{ID: "a100", Class: mig3g, GPUs: 3, FreeGPUs: 3, HostMemGB: 512, HostFreeGB: 400,
|
||||
Kinds: []PoolKind{
|
||||
{Class: mig3g, GPUMemGB: 40, GPUs: 2, FreeGPUs: 2},
|
||||
{Class: "a100-80g", GPUMemGB: 80, GPUs: 1, FreeGPUs: 1},
|
||||
}}}
|
||||
arts := []Artifact{
|
||||
{Name: "q4.gguf", Format: "gguf", Bytes: 9 * gib},
|
||||
{Name: "f16.gguf", Format: "gguf", Bytes: 29 * gib},
|
||||
{Name: "big.gguf", Format: "gguf", Bytes: 60 * gib},
|
||||
}
|
||||
opts := Options(arts, shape14B, nodes, 16384)
|
||||
got := strings.Join(labels(opts), "; ")
|
||||
for _, want := range []string{"q4.gguf 1 × " + mig3g, "f16.gguf 1 × " + mig3g, "big.gguf 1 × a100-80g"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("missing %q in %s", want, got)
|
||||
}
|
||||
}
|
||||
// Sixty gigabytes is more than an instance holds, and two instances are
|
||||
// not one bigger device.
|
||||
if strings.Contains(got, "big.gguf 2 × "+mig3g) {
|
||||
t.Errorf("planned across two MIG instances: %s", got)
|
||||
}
|
||||
for _, o := range opts {
|
||||
if o.Class == mig3g && (o.Hosts != 1 || !o.FreeNow) {
|
||||
t.Errorf("the MIG option counts %d hosts, free %v", o.Hosts, o.FreeNow)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -46,8 +46,35 @@ type PoolNode struct {
|
||||
// HostMemGB is its RAM, and HostFreeGB what is unused now.
|
||||
HostMemGB int
|
||||
HostFreeGB int
|
||||
// Kinds are the node's devices by class, for a node with more than one:
|
||||
// a card in MIG mode is one or several classes of instance, and a whole
|
||||
// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
|
||||
// say it all.
|
||||
Kinds []PoolKind
|
||||
}
|
||||
|
||||
// PoolKind is one class of a node's devices.
|
||||
type PoolKind struct {
|
||||
Class string
|
||||
GPUMemGB int
|
||||
GPUs int
|
||||
FreeGPUs int
|
||||
}
|
||||
|
||||
// kinds is the node's devices by class.
|
||||
func (n PoolNode) kinds() []PoolKind {
|
||||
if len(n.Kinds) > 0 {
|
||||
return n.Kinds
|
||||
}
|
||||
return []PoolKind{{Class: n.Class, GPUMemGB: n.GPUMemGB, GPUs: n.GPUs, FreeGPUs: n.FreeGPUs}}
|
||||
}
|
||||
|
||||
// isMIG reports whether a class names MIG instances: "a100-80g-mig-3g.40gb".
|
||||
// A replica runs on exactly one, never a group of them, so a plan offers one
|
||||
// instance and nothing wider. Read from the class label, as the card's memory
|
||||
// is, which keeps this package free of the agent's.
|
||||
func isMIG(class string) bool { return strings.Contains(class, "-mig-") }
|
||||
|
||||
// Option is one way to run one artifact.
|
||||
type Option struct {
|
||||
Artifact string `json:"artifact"`
|
||||
@@ -149,11 +176,13 @@ func ggufOptions(a Artifact, kv int64, nodes []PoolNode, bpw float64) []Option {
|
||||
}
|
||||
o := Option{Artifact: a.Name, Format: a.Format, Engine: "llamacpp", Class: c.class,
|
||||
GPUsPerNode: width, Nodes: 1, NeedGB: GB(per), HaveGB: budget, BitsPerWeight: bpw}
|
||||
for _, n := range nodes {
|
||||
if n.Class == c.class && n.GPUs >= width {
|
||||
o.Hosts++
|
||||
if n.FreeGPUs >= width {
|
||||
o.FreeNow = true
|
||||
for _, fab := range c.fabrics {
|
||||
for _, n := range fab.nodes {
|
||||
if n.GPUs >= width {
|
||||
o.Hosts++
|
||||
if n.FreeGPUs >= width {
|
||||
o.FreeNow = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -196,7 +225,9 @@ func vllmOptions(a Artifact, kv int64, sh Shape, nodes []PoolNode, bpw float64)
|
||||
single = true
|
||||
break
|
||||
}
|
||||
if single {
|
||||
// A MIG instance does not join others, on its node or across a
|
||||
// fabric: what does not fit one does not fit the class.
|
||||
if single || isMIG(c.class) {
|
||||
continue
|
||||
}
|
||||
// Several nodes, a node's full width on each, pipeline-parallel
|
||||
@@ -251,32 +282,42 @@ type fabricPool struct {
|
||||
}
|
||||
|
||||
// classes groups GPU nodes by card, and each by fabric: a replica spans
|
||||
// nodes only within one.
|
||||
// nodes only within one. A node with devices of several classes is in each
|
||||
// class's pool, as the node it is for that class: its devices of that class
|
||||
// and no others.
|
||||
func classes(nodes []PoolNode) []classPool {
|
||||
by := map[string]*classPool{}
|
||||
var order []string
|
||||
for _, n := range nodes {
|
||||
if n.Class == "" || n.GPUs == 0 || n.GPUMemGB == 0 {
|
||||
continue
|
||||
}
|
||||
c := by[n.Class]
|
||||
if c == nil {
|
||||
c = &classPool{class: n.Class, mem: n.GPUMemGB}
|
||||
by[n.Class] = c
|
||||
order = append(order, n.Class)
|
||||
}
|
||||
c.maxGPUs = max(c.maxGPUs, n.GPUs)
|
||||
i := -1
|
||||
for j := range c.fabrics {
|
||||
if c.fabrics[j].name == n.Fabric {
|
||||
i = j
|
||||
for _, node := range nodes {
|
||||
for _, k := range node.kinds() {
|
||||
if k.Class == "" || k.GPUs == 0 || k.GPUMemGB == 0 {
|
||||
continue
|
||||
}
|
||||
n := node
|
||||
n.Class, n.GPUMemGB, n.GPUs, n.FreeGPUs, n.Kinds = k.Class, k.GPUMemGB, k.GPUs, k.FreeGPUs, nil
|
||||
if isMIG(n.Class) {
|
||||
// One instance a replica, however many the node has.
|
||||
n.GPUs, n.FreeGPUs = min(n.GPUs, 1), min(n.FreeGPUs, 1)
|
||||
}
|
||||
c := by[n.Class]
|
||||
if c == nil {
|
||||
c = &classPool{class: n.Class, mem: n.GPUMemGB}
|
||||
by[n.Class] = c
|
||||
order = append(order, n.Class)
|
||||
}
|
||||
c.maxGPUs = max(c.maxGPUs, n.GPUs)
|
||||
i := -1
|
||||
for j := range c.fabrics {
|
||||
if c.fabrics[j].name == n.Fabric {
|
||||
i = j
|
||||
}
|
||||
}
|
||||
if i < 0 {
|
||||
c.fabrics = append(c.fabrics, fabricPool{name: n.Fabric})
|
||||
i = len(c.fabrics) - 1
|
||||
}
|
||||
c.fabrics[i].nodes = append(c.fabrics[i].nodes, n)
|
||||
}
|
||||
if i < 0 {
|
||||
c.fabrics = append(c.fabrics, fabricPool{name: n.Fabric})
|
||||
i = len(c.fabrics) - 1
|
||||
}
|
||||
c.fabrics[i].nodes = append(c.fabrics[i].nodes, n)
|
||||
}
|
||||
sort.Strings(order)
|
||||
out := make([]classPool, 0, len(order))
|
||||
@@ -363,8 +404,17 @@ func QuantOf(name string) string {
|
||||
return strings.ToUpper(m[len(m)-1])
|
||||
}
|
||||
|
||||
// MemFromClass reads the card size out of a class label such as "h100-80g".
|
||||
// MemFromClass reads the card size out of a class label such as "h100-80g",
|
||||
// and a MIG instance's out of its profile: 40 for "a100-80g-mig-3g.40gb",
|
||||
// whose card is 80 but whose instance holds 40 of it.
|
||||
func MemFromClass(class string) int {
|
||||
if _, profile, ok := strings.Cut(class, "-mig-"); ok {
|
||||
var compute, gb int
|
||||
if _, err := fmt.Sscanf(profile, "%dg.%dgb", &compute, &gb); err != nil {
|
||||
return 0
|
||||
}
|
||||
return gb
|
||||
}
|
||||
i := strings.LastIndex(class, "-")
|
||||
if i < 0 || !strings.HasSuffix(class, "g") {
|
||||
return 0
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# C22: MIG
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### MIG instances as GPUs of their own class
|
||||
|
||||
- **A card in MIG mode is placed as its instances.** Discovery reports a card
|
||||
in MIG mode as its MIG instances instead of the whole card, each a device of
|
||||
its own with its UUID (`MIG-…`), its own memory, and a class named from the
|
||||
card's and the profile: two `3g.40gb` instances of an A100 80GB are two
|
||||
devices of class `a100-80g-mig-3g.40gb`. A profile names that class like
|
||||
any other, and placement puts a replica on a free instance of it. Both
|
||||
discovery paths do it: nvidia-smi (instances from `nvidia-smi -L`, their
|
||||
memory and ids from `nvidia-smi -q -x`) and NVML (`GetMigMode`,
|
||||
`GetMaxMigDeviceCount`, `GetMigDeviceHandleByIndex`). llmgw does not create
|
||||
instances; `docs/install.md` has a MIG section on making them.
|
||||
- **One instance a replica.** A process opens only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer between them, so a profile on a
|
||||
MIG class asking for more than one GPU, tp or pp above 1, or more than one
|
||||
node is refused when the model is added, with that reason; placement
|
||||
refuses it again. `ctl model plan` and `ctl model serve` plan an instance
|
||||
for a model one holds and never several together.
|
||||
- **Engines are given the instance by UUID.** Placement numbers card 0's
|
||||
instance 2 as device 1002 (1000 + 100 × card + instance); the agent turns
|
||||
that into `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp, and
|
||||
leaves `GGML_VK_VISIBLE_DEVICES` unset (Vulkan cannot see MIG). Every view
|
||||
shows the device as `0/mig2`.
|
||||
- **Memory per instance, no made-up utilisation.** Each instance's memory,
|
||||
and the processes holding it, are its own — read per GPU and compute
|
||||
instance from nvidia-smi's report, so a foreign process on one instance
|
||||
keeps that instance out of placement and not its neighbours. Temperature,
|
||||
power and wear are the card's. No driver reports utilisation per instance,
|
||||
so none is published for one: the console says `n/a on MIG`, and
|
||||
`llmgw_gpu_utilization_percent` has no series for it rather than a zero.
|
||||
- **A card in MIG mode with nothing carved is kept out of placement**, with
|
||||
the reason in `ctl nodes`, since CUDA will not open the whole card while MIG
|
||||
is on.
|
||||
- **`llmgw doctor` reports MIG**: each card's mode and instances, and warns
|
||||
of a card in MIG mode with no instances, of GPU instances with no compute
|
||||
instance, and of a mode change waiting for a reset, with the `nvidia-smi`
|
||||
command for each.
|
||||
- `ctl nodes` counts a node's devices by class when it has MIG instances or
|
||||
more than one class (`a100-80g-mig-3g.40gb×2,a100-80g×1`); the console's
|
||||
GPU list shows instances under their card with their own memory and the
|
||||
card's heat and power, and a node's page lists its classes with counts.
|
||||
`ctl status` and the plan count each device under its own class, where a
|
||||
node's cards were all put under its first class before.
|
||||
- The agent protocol's `GPU` gains `mig` (parent card, its memory, profile,
|
||||
GPU and compute instance ids); the node API's GPUs gain `label`, `class`,
|
||||
`card`, `mig_profile` and memory, and nodes gain `gpus_by_class`.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
**Done (branch `features-c2`):** MIG instances discovered on both paths (nvidia-smi `-L` + `-q -x`; NVML through an interface with a faithful fake) and reported instead of their card, each with its MIG UUID, own memory and class `<card class>-mig-<profile>`; device numbers 1000 + 100 × card + instance, shown as `card/migN`; placement on MIG classes; tp/pp/gpus/nodes above 1 on a MIG class refused in catalog validation and again in placement; `CUDA_VISIBLE_DEVICES=MIG-<uuid>` for vLLM and llama.cpp (no Vulkan device variable on MIG); per-instance memory and processes in heartbeats and the GPU check, card sensors under each instance, utilisation absent rather than zero; doctor MIG findings (`mig -lgi`/`-lci` parsed); `ctl nodes`, console, `ctl status`, `ctl model plan`/`serve` by class; docs/install.md MIG section. Written and tested against fakes only — untested on real MIG hardware. Left: run it on an A100/H100 in MIG mode (check `-L`/`-q -x`/`mig -lgi`/`-lci` and NVML MIG device names against the parsers, and that `CUDA_VISIBLE_DEVICES=MIG-…` reaches vLLM's workers through Ray); instances changed while the agent runs are only seen when it restarts (no rescan); llmgw does not create or reshape instances (no mig-parted integration); quotas count an instance as one GPU, the same as a whole card; SGLang on MIG not checked; device numbers assume fewer than 100 instances per card and that instance numbers stay put while shards run on them.
|
||||
Reference in new issue
Block a user