Watch InfiniBand ports while the agent runs, not only when it starts

A port's state was read once, at registration: one that went down later
left its node on the fabric, offered multi-node work that would hang on
its first collective, until somebody restarted the agent. The agent now
re-reads every IB and RoCE port on each heartbeat - state, physical
state, rate and the error and data counters - and when the fabrics it
can carry traffic on change, says so once; the control plane replaces
the node's fabrics, which is what placement reads.

The ports are exported as llmgw_ib_port_active (with state and
phys_state, so an unplugged cable reads differently from a missing
subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total,
and a node's page lists them with the fabric each carries and any error
counter that is not zero.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-29 00:13:54 +02:00
1 parent bf2d171e89
commit 93bd16f00e
15 files changed
+654 -107

No files matched your search

+20
View File
@@ -31,6 +31,26 @@
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
{{end}}
{{if or .Node.IBPorts .Node.Fabrics}}
<h2>Fabric</h2>
<p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}}.{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
{{if .Node.IBPorts}}
<table><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
{{range .Node.IBPorts}}<tr>
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
<td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
<td class="num">{{if .RateGbps}}{{.RateGbps}} Gb/s{{else}}—{{end}}</td>
<td>{{or .Fabric "—"}}</td>
<td>{{if .Errors}}{{range $k, $v := .Errors}}<span class="tag bad">{{$k}} {{$v}}</span> {{end}}{{else}}<span class="note">none</span>{{end}}</td>
<td class="num">{{if .TxBytes}}{{bytes (u64 .TxBytes)}}{{else}}—{{end}}</td>
<td class="num">{{if .RxBytes}}{{bytes (u64 .RxBytes)}}{{else}}—{{end}}</td>
</tr>{{end}}</table>
<p class="note">Re-read on every heartbeat. A port that goes down takes its fabric off
this node within seconds; one that comes back returns it. Errors are cumulative since
the adapter came up: a number that grows is the one to look at.</p>
{{end}}
{{end}}
<h2>What runs here</h2>
{{if .Engines}}
<table><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>