Measure two nodes' fabric rail by rail: llmgw ctl fabric-test
doctor checks a node alone, and every check can pass while two nodes still cannot reach each other. ctl fabric-test A B asks B's agent for an ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs them, one rail at a time, and grades each against its link's rate. It goes through the agents' streams, so there is no SSH between nodes; the agent installs perftest the first time, and names the RoCE v2 GID itself. The result is read from perftest's table by its heading, not a column number. Agents now list the requests they answer beyond the usual (Register.features), and the control plane refuses a test on one too old to answer instead of waiting minutes for it. The forwarding dispatcher carries the new capability, and its test says so: a missing forward would have made the command quietly unavailable in production. Not yet run over real InfiniBand or RoCE; this cluster has neither. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
af6f8e023e
commit
7bb360fc78
21 files changed
+1265
-62
No files matched your search
@@ -89,6 +89,13 @@ the same fabric the same name. In a script, give it instead:
|
||||
sudo llmgw node add ubuntu@10.0.0.21 --fabric ib-a:mlx5_0,mlx5_1,mlx5_2,mlx5_3
|
||||
```
|
||||
|
||||
Once two nodes on a fabric are in, check they reach each other at the link's
|
||||
rate before placing anything across them:
|
||||
|
||||
```
|
||||
llmgw ctl fabric-test gpu01 gpu02
|
||||
```
|
||||
|
||||
**If it cannot log in.** `sudo` runs SSH as root, with root's keys. To use your
|
||||
own keys instead, log in to the control plane with `ssh -A` and keep your
|
||||
agent through sudo:
|
||||
|
||||
Reference in new issue
Block a user