Measure two nodes' fabric rail by rail: llmgw ctl fabric-test
doctor checks a node alone, and every check can pass while two nodes still cannot reach each other. ctl fabric-test A B asks B's agent for an ib_write_bw server and A's for the client, adapter by adapter as NCCL pairs them, one rail at a time, and grades each against its link's rate. It goes through the agents' streams, so there is no SSH between nodes; the agent installs perftest the first time, and names the RoCE v2 GID itself. The result is read from perftest's table by its heading, not a column number. Agents now list the requests they answer beyond the usual (Register.features), and the control plane refuses a test on one too old to answer instead of waiting minutes for it. The forwarding dispatcher carries the new capability, and its test says so: a missing forward would have made the command quietly unavailable in production. Not yet run over real InfiniBand or RoCE; this cluster has neither. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
af6f8e023e
commit
7bb360fc78
21 files changed
+1265
-62
No files matched your search
@@ -25,6 +25,7 @@ message AgentMessage {
|
||||
ShardStatus shard = 3;
|
||||
LogsReply logs = 4;
|
||||
FabricScanReply fabric_scan = 5;
|
||||
FabricTestReply fabric_test = 6;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -39,6 +40,7 @@ message ControlMessage {
|
||||
Provision provision = 6;
|
||||
LogsRequest logs = 7;
|
||||
FabricScanRequest fabric_scan = 8;
|
||||
FabricTestRequest fabric_test = 9;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -125,6 +127,12 @@ message Register {
|
||||
// vLLM, it answered "unsupported", and every node of a cluster half way
|
||||
// through an upgrade showed as failing.
|
||||
repeated string installable_engines = 16;
|
||||
|
||||
// features are the requests this agent answers beyond the ones every agent
|
||||
// does: "fabric-test". An agent that does not know a request ignores it, so
|
||||
// without this the control plane would wait minutes for an answer that is
|
||||
// never coming.
|
||||
repeated string features = 17;
|
||||
}
|
||||
|
||||
// Host describes the operating system and processor of a node.
|
||||
@@ -605,3 +613,35 @@ message FabricScanReply {
|
||||
string error = 3;
|
||||
int64 took_ms = 4;
|
||||
}
|
||||
|
||||
// FabricTestRequest asks a node for one side of an RDMA bandwidth test
|
||||
// between two nodes: ib_write_bw, from perftest, over one adapter. The
|
||||
// server side is asked first and answers when the test is over; the client
|
||||
// dials it. What it proves is what a replica spanning nodes needs and nothing
|
||||
// on one node can show: that these two adapters reach each other through the
|
||||
// fabric, and how fast.
|
||||
message FabricTestRequest {
|
||||
string request_id = 1;
|
||||
// role is "server" or "client".
|
||||
string role = 2;
|
||||
string hca = 3;
|
||||
int32 port = 4;
|
||||
// peer is the server's management address, for the client. perftest
|
||||
// exchanges its connection details over TCP before any RDMA.
|
||||
string peer = 5;
|
||||
// tcp_port is that exchange's port, the same on both sides.
|
||||
int32 tcp_port = 6;
|
||||
int32 seconds = 7;
|
||||
}
|
||||
|
||||
// FabricTestReply is what one side measured.
|
||||
message FabricTestReply {
|
||||
string request_id = 1;
|
||||
// gbps is the average bandwidth, in gigabits a second.
|
||||
double gbps = 2;
|
||||
// output is the end of the tool's output, for when the number is not
|
||||
// enough.
|
||||
string output = 3;
|
||||
string error = 4;
|
||||
int64 took_ms = 5;
|
||||
}
|
||||
Reference in new issue
Block a user