Find multi-node NVLink domains, place across them, and show them as racks
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its GPUs reach each other over NVLink whatever node they are in. Each GPU now reports its fabric from nvidia-smi -q - state, status, ClusterUUID, CliqueId, bandwidth - at registration and once a minute after, since a fabric manager restarting drops every card out at once; and each node reports nvidia-imex and its channels, without which the domain's GPUs cannot share memory across nodes. A single machine's NVSwitch, which reports a cluster of zeros, is no domain. A replica spanning nodes is placed inside one domain first, on nodes whose every card is joined and whose IMEX is up, and its shards are told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback. /admin/nvlink draws each domain as a rack - the NVLink Switch band on top, light running down a backplane, trays of GPU chips that glow when joined, pulse red when not, turn amber on degraded bandwidth - and lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink matrix, each card's NUMA node and nearest adapter, and its domain and IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5bbc619a31
commit
20479a2d5f
31 files changed
+1535
-131
No files matched your search
+6
-3
@@ -26,6 +26,7 @@ import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/nvlink"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -128,9 +129,11 @@ var funcs = template.FuncMap{
|
||||
// shows. Binary units because that is what `ls -lh` and every disk-full
|
||||
// message on the host will say, and a page that disagrees with the shell
|
||||
// about the size of the same file starts an argument about neither.
|
||||
"bytes": humanBytes,
|
||||
"u64": func(v uint64) int64 { return int64(min(v, 1<<62)) },
|
||||
"has": func(list []string, v string) bool { return slices.Contains(list, v) },
|
||||
"bytes": humanBytes,
|
||||
"u64": func(v uint64) int64 { return int64(min(v, 1<<62)) },
|
||||
"has": func(list []string, v string) bool { return slices.Contains(list, v) },
|
||||
"hasInt": func(list []int, v int) bool { return slices.Contains(list, v) },
|
||||
"shortDomain": nvlink.Short,
|
||||
// dict lets one chart template be called with named arguments. Go
|
||||
// templates pass a single value, and four charts differing in three
|
||||
// strings each is four copies of the same markup otherwise.
|
||||
|
||||
Reference in new issue
Block a user