Watch InfiniBand ports while the agent runs, not only when it starts
A port's state was read once, at registration: one that went down later left its node on the fabric, offered multi-node work that would hang on its first collective, until somebody restarted the agent. The agent now re-reads every IB and RoCE port on each heartbeat - state, physical state, rate and the error and data counters - and when the fabrics it can carry traffic on change, says so once; the control plane replaces the node's fabrics, which is what placement reads. The ports are exported as llmgw_ib_port_active (with state and phys_state, so an unplugged cable reads differently from a missing subnet manager), llmgw_ib_port_rate_gbps and llmgw_ib_port_counter_total, and a node's page lists them with the fabric each carries and any error counter that is not zero. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
bf2d171e89
commit
93bd16f00e
15 files changed
+654
-107
No files matched your search
@@ -31,6 +31,26 @@
|
||||
{{if .Unhealthy}}<p class="note bad">{{join .Unhealthy}}</p>{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if or .Node.IBPorts .Node.Fabrics}}
|
||||
<h2>Fabric</h2>
|
||||
<p class="sub">{{if .Node.Fabrics}}On {{join .Node.Fabrics}}.{{else}}<span class="bad">On no fabric: no port a declared fabric names can carry traffic, so no multi-node replica is placed here.</span>{{end}}</p>
|
||||
{{if .Node.IBPorts}}
|
||||
<table><tr><th>Adapter</th><th>State</th><th class="num">Rate</th><th>Fabric</th><th>Errors since boot</th><th class="num">Sent</th><th class="num">Received</th></tr>
|
||||
{{range .Node.IBPorts}}<tr>
|
||||
<td><code>{{.HCA}}/{{.Port}}</code>{{if eq .LinkLayer "Ethernet"}} <span class="tag">RoCE</span>{{end}}</td>
|
||||
<td class="{{if .Active}}state-ready{{else}}bad{{end}}">{{.State}}{{with .PhysState}} <span class="note">{{.}}</span>{{end}}</td>
|
||||
<td class="num">{{if .RateGbps}}{{.RateGbps}} Gb/s{{else}}—{{end}}</td>
|
||||
<td>{{or .Fabric "—"}}</td>
|
||||
<td>{{if .Errors}}{{range $k, $v := .Errors}}<span class="tag bad">{{$k}} {{$v}}</span> {{end}}{{else}}<span class="note">none</span>{{end}}</td>
|
||||
<td class="num">{{if .TxBytes}}{{bytes (u64 .TxBytes)}}{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .RxBytes}}{{bytes (u64 .RxBytes)}}{{else}}—{{end}}</td>
|
||||
</tr>{{end}}</table>
|
||||
<p class="note">Re-read on every heartbeat. A port that goes down takes its fabric off
|
||||
this node within seconds; one that comes back returns it. Errors are cumulative since
|
||||
the adapter came up: a number that grows is the one to look at.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
<h2>What runs here</h2>
|
||||
{{if .Engines}}
|
||||
<table><tr><th>Model</th><th>Replica</th><th>State</th><th>CPU</th><th class="num">Memory</th></tr>
|
||||
|
||||
Reference in new issue
Block a user