Phase 4: make a multi-node replica able to start
Placement and environment generation had landed early, which made multi-node look further along than it was. A replica spanning nodes could not come up at all, and all three reasons came down to assumptions that only hold on one node. Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers; workers timed out, reported Failed and took the replica with them. Engine.ServesHTTP now says which ranks serve, and a worker is judged by staying alive. That is deliberately weaker evidence: the replica's real readiness is the leader answering, which only happens once every worker has joined. A shard is not always one process. Engine.Commands returns a sequence — every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's. vLLM under Ray is why: the leader runs a Ray head and then the server against it, while a worker runs only the join and no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist. Sidecars are killed with the shard, because a Ray head left behind holds the port the next replica needs. Start order was backwards. Workers went first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own. A command can now also require that rendezvous point to be listening before the next starts, bounded by a timeout so a head that never appears fails the shard instead of hanging it. Stop was also killing only the engine, leaving sidecars behind. None of this is verified against real engines — it is tested with stub processes. The Ray sequencing, and which launch mode a given vLLM version wants, can only be confirmed on hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
75a8e76706
commit
1d1a6aebb0
11 files changed
+647
-125
No files matched your search
@@ -411,11 +411,17 @@ func (r *Reconciler) startReplica(ctx context.Context, m catalog.Model, pl *plac
|
||||
// agent status report can mutate it concurrently, and the race detector is
|
||||
// right to object.
|
||||
//
|
||||
// Workers first, then the leader. SGLang's ranks rendezvous, and vLLM under
|
||||
// Ray needs its workers attached before the head starts serving; starting
|
||||
// rank 0 first just makes it wait.
|
||||
// The leader first, then workers ascending. Both engines put the rendezvous
|
||||
// point on rank 0 — SGLang's dist-init address and vLLM's Ray head — so a
|
||||
// worker started first has nothing to connect to and spends its startup
|
||||
// budget retrying.
|
||||
//
|
||||
// An earlier version started workers first, on the theory that the head
|
||||
// should wait for them. That is backwards: vLLM's server blocks until the
|
||||
// Ray cluster has the GPUs it needs, so the leader can be started
|
||||
// immediately and will wait on its own.
|
||||
order := slices.Clone(rep.Shards)
|
||||
slices.SortFunc(order, func(a, b Shard) int { return b.Rank - a.Rank })
|
||||
slices.SortFunc(order, func(a, b Shard) int { return a.Rank - b.Rank })
|
||||
nodeIDs := rep.NodeIDs()
|
||||
|
||||
r.mu.Lock()
|
||||
|
||||
@@ -262,9 +262,15 @@ func TestPacksWhenSpreadingIsImpossible(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestShardOrderWorkersFirst(t *testing.T) {
|
||||
// SGLang ranks rendezvous and vLLM under Ray needs its workers attached
|
||||
// before the head serves, so rank 0 goes last.
|
||||
func TestShardOrderLeaderFirst(t *testing.T) {
|
||||
// Both engines put the rendezvous point on rank 0 — SGLang's dist-init
|
||||
// address and vLLM's Ray head — so a worker started first has nothing to
|
||||
// connect to and spends its startup budget retrying.
|
||||
//
|
||||
// This test previously asserted the opposite, on the theory that the head
|
||||
// should wait for its workers. That is backwards: vLLM's server blocks
|
||||
// until the Ray cluster has the GPUs it needs, so the leader can start
|
||||
// immediately and waits on its own.
|
||||
h := newHarness(t, testModel(1, catalog.Profile{
|
||||
Name: "2node", GPUClass: "h100-80g", TP: 16, Nodes: 2, GPUsPerNode: 8,
|
||||
}), gpuNode("1", 8, "ib-a"), gpuNode("2", 8, "ib-a"))
|
||||
@@ -275,8 +281,9 @@ func TestShardOrderWorkersFirst(t *testing.T) {
|
||||
if len(started) != 2 {
|
||||
t.Fatalf("expected 2 shards, got %d", len(started))
|
||||
}
|
||||
if started[0].Spec.Rank != 1 || started[1].Spec.Rank != 0 {
|
||||
t.Errorf("start order was rank %d then %d; workers must start first",
|
||||
if started[0].Spec.Rank != 0 || started[1].Spec.Rank != 1 {
|
||||
t.Errorf("start order was rank %d then %d; the leader hosts the rendezvous "+
|
||||
"point and must start first",
|
||||
started[0].Spec.Rank, started[1].Spec.Rank)
|
||||
}
|
||||
// Every rank must be given the same rendezvous point.
|
||||
|
||||
Reference in new issue
Block a user