A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its GPUs reach each other over NVLink whatever node they are in. Each GPU now reports its fabric from nvidia-smi -q - state, status, ClusterUUID, CliqueId, bandwidth - at registration and once a minute after, since a fabric manager restarting drops every card out at once; and each node reports nvidia-imex and its channels, without which the domain's GPUs cannot share memory across nodes. A single machine's NVSwitch, which reports a cluster of zeros, is no domain. A replica spanning nodes is placed inside one domain first, on nodes whose every card is joined and whose IMEX is up, and its shards are told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback. /admin/nvlink draws each domain as a rack - the NVLink Switch band on top, light running down a backplane, trays of GPU chips that glow when joined, pulse red when not, turn amber on degraded bandwidth - and lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink matrix, each card's NUMA node and nearest adapter, and its domain and IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
55 lines
2.8 KiB
HTML
55 lines
2.8 KiB
HTML
{{define "nvlink"}}<!doctype html>
|
|
<html lang="en"><head>
|
|
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
|
<meta http-equiv="refresh" content="30">
|
|
<title>NVLink domains · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
|
</head><body><div class="wrap wide">
|
|
<header>
|
|
<h1>NVLink domains</h1>
|
|
<p class="sub">{{len .Domains}} multi-node domain{{if ne (len .Domains) 1}}s{{end}} — racks whose GPUs reach each other over NVLink across nodes</p>
|
|
{{if .OnAdminListener}}<nav><a href="/">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
|
|
</header>
|
|
|
|
{{if not .Domains}}
|
|
<p class="empty">No GPU reports a multi-node NVLink domain. An eight-GPU machine's NVSwitch ends at its
|
|
chassis and is shown on its node's page; a domain appears here once the fabric manager has joined the
|
|
GPUs of an NVL72 rack, or an NVLink Switch system, to one.</p>
|
|
{{end}}
|
|
|
|
<div class="racks">
|
|
{{range .Domains}}
|
|
<section class="rack{{if .Problems}} troubled{{end}}">
|
|
<header class="rack-head">
|
|
<div><h2>{{.Short}}</h2><p class="note"><code>{{.ClusterUUID}}</code> · clique {{.CliqueID}}</p></div>
|
|
<div class="rack-stats">
|
|
<div><b>{{.Usable}}/{{len .Members}}</b><span>trays usable</span></div>
|
|
<div><b>{{.Joined}}/{{.GPUs}}</b><span>GPUs joined</span></div>
|
|
</div>
|
|
</header>
|
|
<div class="rack-body">
|
|
<div class="nvswitch"><span>NVLink Switch</span><i></i></div>
|
|
<div class="rack-trays">
|
|
<div class="backplane" aria-hidden="true"></div>
|
|
{{range .Members}}
|
|
<div class="tray{{if not .Usable}} out{{end}}">
|
|
<a class="tray-name" href="{{$.Nodes}}/{{.Node}}">{{.Node}}</a>
|
|
<div class="chips">{{range .GPUs}}<span class="chip{{if .Joined}} joined{{else}} lost{{end}}{{if and .Bandwidth (ne .Bandwidth "Full")}} slow{{end}}"
|
|
title="GPU {{.Index}} · {{.Name}} · {{or .State "no fabric"}}{{if .Bandwidth}} · {{.Bandwidth}} bandwidth{{end}}">{{.Index}}</span>{{end}}</div>
|
|
<span class="imex{{if and (eq .IMEX "active") .IMEXChannels}} up{{end}}" title="nvidia-imex {{or .IMEX "not installed"}}, {{.IMEXChannels}} channel(s)">IMEX</span>
|
|
</div>
|
|
{{end}}
|
|
</div>
|
|
</div>
|
|
{{if .Problems}}<ul class="fabric-problems rack-problems">{{range .Problems}}
|
|
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{.Level}}</span> {{.Message}}</li>{{end}}</ul>{{end}}
|
|
</section>
|
|
{{end}}
|
|
</div>
|
|
|
|
<p class="note">A replica that spans nodes is placed inside one of these first: tensor parallelism between its
|
|
nodes then runs over NVLink rather than InfiniBand. Only trays with every GPU joined and IMEX up are used —
|
|
a tray short of either would leave the replica a card, or its memory, short.</p>
|
|
|
|
<footer><span>llmgw {{.Version}} · refreshes every 30s</span><span>{{.Now}}</span></footer>
|
|
</div></body></html>{{end}}
|