Files
llmgw/internal/web/templates/nvlink.html
T
Ilker ManapandClaude Opus 5.5 20479a2d5f Find multi-node NVLink domains, place across them, and show them as racks
A GB200 NVL72 rack or an NVLink Switch system is a scale-up island: its
GPUs reach each other over NVLink whatever node they are in. Each GPU
now reports its fabric from nvidia-smi -q - state, status, ClusterUUID,
CliqueId, bandwidth - at registration and once a minute after, since a
fabric manager restarting drops every card out at once; and each node
reports nvidia-imex and its channels, without which the domain's GPUs
cannot share memory across nodes. A single machine's NVSwitch, which
reports a cluster of zeros, is no domain.

A replica spanning nodes is placed inside one domain first, on nodes
whose every card is joined and whose IMEX is up, and its shards are
told NCCL_MNNVL_ENABLE=1; InfiniBand is the fallback.

/admin/nvlink draws each domain as a rack - the NVLink Switch band on
top, light running down a backplane, trays of GPU chips that glow when
joined, pulse red when not, turn amber on degraded bandwidth - and
lists what keeps a tray out. A node's page shows its GPU-by-GPU NVLink
matrix, each card's NUMA node and nearest adapter, and its domain and
IMEX. doctor checks joining, bandwidth, IMEX and its channel; metrics
llmgw_gpu_nvlink_joined and llmgw_imex_up; /api/v1/nvlink.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:07:59 +02:00

55 lines
2.8 KiB
HTML

{{define "nvlink"}}<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<meta http-equiv="refresh" content="30">
<title>NVLink domains · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
</head><body><div class="wrap wide">
<header>
<h1>NVLink domains</h1>
<p class="sub">{{len .Domains}} multi-node domain{{if ne (len .Domains) 1}}s{{end}} — racks whose GPUs reach each other over NVLink across nodes</p>
{{if .OnAdminListener}}<nav><a href="/">cluster</a></nav>{{else}}{{template "nav" .Nav}}{{end}}
</header>
{{if not .Domains}}
<p class="empty">No GPU reports a multi-node NVLink domain. An eight-GPU machine's NVSwitch ends at its
chassis and is shown on its node's page; a domain appears here once the fabric manager has joined the
GPUs of an NVL72 rack, or an NVLink Switch system, to one.</p>
{{end}}
<div class="racks">
{{range .Domains}}
<section class="rack{{if .Problems}} troubled{{end}}">
<header class="rack-head">
<div><h2>{{.Short}}</h2><p class="note"><code>{{.ClusterUUID}}</code> · clique {{.CliqueID}}</p></div>
<div class="rack-stats">
<div><b>{{.Usable}}/{{len .Members}}</b><span>trays usable</span></div>
<div><b>{{.Joined}}/{{.GPUs}}</b><span>GPUs joined</span></div>
</div>
</header>
<div class="rack-body">
<div class="nvswitch"><span>NVLink Switch</span><i></i></div>
<div class="rack-trays">
<div class="backplane" aria-hidden="true"></div>
{{range .Members}}
<div class="tray{{if not .Usable}} out{{end}}">
<a class="tray-name" href="{{$.Nodes}}/{{.Node}}">{{.Node}}</a>
<div class="chips">{{range .GPUs}}<span class="chip{{if .Joined}} joined{{else}} lost{{end}}{{if and .Bandwidth (ne .Bandwidth "Full")}} slow{{end}}"
title="GPU {{.Index}} · {{.Name}} · {{or .State "no fabric"}}{{if .Bandwidth}} · {{.Bandwidth}} bandwidth{{end}}">{{.Index}}</span>{{end}}</div>
<span class="imex{{if and (eq .IMEX "active") .IMEXChannels}} up{{end}}" title="nvidia-imex {{or .IMEX "not installed"}}, {{.IMEXChannels}} channel(s)">IMEX</span>
</div>
{{end}}
</div>
</div>
{{if .Problems}}<ul class="fabric-problems rack-problems">{{range .Problems}}
<li class="{{.Level}}"><span class="tag{{if eq (printf "%s" .Level) "fail"}} bad{{end}}">{{.Level}}</span> {{.Message}}</li>{{end}}</ul>{{end}}
</section>
{{end}}
</div>
<p class="note">A replica that spans nodes is placed inside one of these first: tensor parallelism between its
nodes then runs over NVLink rather than InfiniBand. Only trays with every GPU joined and IMEX up are used —
a tray short of either would leave the replica a card, or its memory, short.</p>
<footer><span>llmgw {{.Version}} · refreshes every 30s</span><span>{{.Now}}</span></footer>
</div></body></html>{{end}}