Find multi-node NVLink domains, place across them, and show them as racks

A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its
GPUs reach each other over NVLink whatever node they are in. Each GPU
now reports its fabric from nvidia-smi -q - state, status, ClusterUUID,
CliqueId, bandwidth - at registration and once a minute after, since a
fabric manager restarting drops every card out at once; and each node
reports nvidia-imex and its channels, without which the domain's GPUs
cannot share memory across nodes. A single machine's NVSwitch, which
reports a cluster of zeros, is no domain.

A replica spanning nodes is placed inside one domain first, on nodes
whose every card is joined and whose IMEX is up, and its shards are
told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback.

/admin/nvlink draws each domain as a rack - the NVLink Switch band on
top, light running down a backplane, trays of GPU chips that glow when
joined, pulse red when not, turn amber on degraded bandwidth - and
lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink
matrix, each card's NUMA node and nearest adapter, and its domain and
IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics
llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 09:07:59 +02:00
1 parent 5bbc619a31
commit 20479a2d5f
31 files changed
+1535 -131

No files matched your search

+68
View File
@@ -677,3 +677,71 @@ ul.fabric-problems li.info { color: var(--muted); }
}
.mode-rails .fm-host.down circle:not(.fm-halo) { fill: var(--fm-bad); stroke: #fecdd3; }
.fm-svg.focus .fm-warn { opacity: .25; }
/* A node's GPUs: which pairs NVLink joins. */
.gpu-wiring { display: flex; flex-wrap: wrap; gap: 1.25rem; align-items: flex-start; }
table.nvl-matrix { width: auto; border-collapse: separate; border-spacing: 3px; background: none; font-size: .75rem; }
table.nvl-matrix th { font-weight: 600; color: var(--muted); padding: 0 .3rem; text-align: center; border: 0; }
table.nvl-matrix td { width: 2rem; height: 2rem; text-align: center; border-radius: 6px; padding: 0; border: 0;
font-weight: 700; font-size: .7rem; }
table.nvl-matrix td.self { background: var(--line); }
table.nvl-matrix td.nv { color: #04211c; background: linear-gradient(135deg, #5eead4, #2dd4bf);
box-shadow: 0 0 10px rgba(45, 212, 191, .45); }
table.nvl-matrix td.pcie { color: var(--muted); background: var(--card); border: 1px solid var(--line); }
table.gpu-list { width: auto; flex: 1 1 22rem; }
/* ---------------------------------------------------------------------
NVLink domains, drawn as racks: the switch band on top, a backplane
carrying light down the side, trays of GPU chips that glow when joined.
--------------------------------------------------------------------- */
.racks { display: grid; grid-template-columns: repeat(auto-fill, minmax(26rem, 1fr)); gap: 1.25rem; margin: 1rem 0 1.5rem; }
.rack { border-radius: 16px; overflow: hidden; color: #dbe4ff;
background: radial-gradient(700px 260px at 50% -20%, #1a3b3b 0%, #0b1020 60%), #0b1020;
border: 1px solid #1c2f3f; box-shadow: 0 20px 50px -28px rgba(45, 212, 191, .45); }
.rack.troubled { border-color: #4a1d2c; box-shadow: 0 20px 50px -28px rgba(251, 113, 133, .45); }
.rack-head { display: flex; justify-content: space-between; align-items: flex-start; gap: 1rem; padding: .9rem 1rem .6rem; }
.rack-head h2 { margin: 0; color: #fff; font-size: 1.15rem; letter-spacing: .02em; }
.rack-head .note, .rack-head code { color: #7c89ad; background: none; font-size: .72rem; }
.rack-stats { display: flex; gap: 1rem; text-align: right; }
.rack-stats b { display: block; font-size: 1.1rem; color: #fff; font-variant-numeric: tabular-nums; }
.rack-stats span { font-size: .62rem; letter-spacing: .1em; text-transform: uppercase; color: #7c89ad; }
.rack-body { padding: .25rem 1rem 1rem; }
.nvswitch { position: relative; height: 1.9rem; border-radius: 8px; overflow: hidden; display: flex; align-items: center;
justify-content: center; font-size: .68rem; letter-spacing: .22em; text-transform: uppercase; color: #ccfbf1;
background: linear-gradient(90deg, #0f3b3a, #115e59 50%, #0f3b3a); border: 1px solid #2dd4bf55;
box-shadow: 0 0 22px rgba(45, 212, 191, .35), inset 0 0 12px rgba(45, 212, 191, .25); }
.nvswitch span { position: relative; z-index: 1; text-shadow: 0 0 8px rgba(94, 234, 212, .8); }
.nvswitch i { position: absolute; inset: 0; background: linear-gradient(100deg, transparent 30%, rgba(255,255,255,.28) 50%, transparent 70%);
transform: translateX(-100%); animation: nv-shimmer 3.2s ease-in-out infinite; }
@keyframes nv-shimmer { to { transform: translateX(100%); } }
.rack-trays { position: relative; margin-top: .6rem; padding-left: 1.1rem; display: flex; flex-direction: column; gap: .45rem; }
.backplane { position: absolute; left: .3rem; top: -.6rem; bottom: .3rem; width: 4px; border-radius: 4px;
background: linear-gradient(180deg, #2dd4bf, #0f766e 40%, #2dd4bf 60%, #0f766e); background-size: 100% 300%;
box-shadow: 0 0 12px rgba(45, 212, 191, .6); animation: nv-flow 2.4s linear infinite; }
@keyframes nv-flow { to { background-position: 0 300%; } }
.tray { position: relative; display: flex; align-items: center; gap: .75rem; padding: .45rem .6rem; border-radius: 10px;
background: linear-gradient(180deg, rgba(255,255,255,.05), rgba(255,255,255,.015)); border: 1px solid #1f2d4a; }
.tray::before { content: ""; position: absolute; left: -.8rem; top: 50%; width: .8rem; height: 2px;
background: linear-gradient(90deg, #2dd4bf, transparent); box-shadow: 0 0 6px #2dd4bf; }
.tray.out { border-color: #5b2333; background: linear-gradient(180deg, rgba(251,113,133,.08), rgba(251,113,133,.02)); }
.tray.out::before { background: linear-gradient(90deg, #fb7185, transparent); box-shadow: 0 0 6px #fb7185; }
.tray-name { min-width: 6.5rem; font-size: .8rem; font-weight: 700; color: #e0f2fe; text-decoration: none; }
.tray-name:hover { color: #5eead4; }
.chips { display: flex; flex-wrap: wrap; gap: .35rem; flex: 1; }
.chip { width: 1.7rem; height: 1.7rem; border-radius: 6px; display: inline-flex; align-items: center; justify-content: center;
font-size: .62rem; font-weight: 800; color: #04211c; cursor: default; }
.chip.joined { background: linear-gradient(135deg, #5eead4, #14b8a6); box-shadow: 0 0 12px rgba(45, 212, 191, .55);
animation: nv-breathe 3.6s ease-in-out infinite; }
.chip.joined.slow { background: linear-gradient(135deg, #fde68a, #f59e0b); box-shadow: 0 0 12px rgba(245, 158, 11, .6); }
.chip.lost { color: #fff; background: linear-gradient(135deg, #fb7185, #be123c); box-shadow: 0 0 14px rgba(251, 113, 133, .7);
animation: nv-alarm 1.2s ease-in-out infinite; }
@keyframes nv-breathe { 50% { box-shadow: 0 0 4px rgba(45, 212, 191, .25); } }
@keyframes nv-alarm { 50% { opacity: .55; } }
.imex { font-size: .6rem; font-weight: 800; letter-spacing: .08em; padding: .15rem .4rem; border-radius: 999px;
color: #fecdd3; border: 1px solid #fb7185; }
.imex.up { color: #99f6e4; border-color: #2dd4bf; box-shadow: 0 0 8px rgba(45, 212, 191, .35); }
.rack-problems { padding: 0 1rem .9rem; margin: 0; }
.rack-problems li { color: #dbe4ff; }
@media (prefers-reduced-motion: reduce) {
.nvswitch i, .backplane, .chip.joined, .chip.lost { animation: none; }
}