Survive a control plane restart without orphaning live engines
CI / test (push) Successful in 1m44s

Not persisting replica state left one hole: restarting the control plane made it
forget every replica while the engines kept running. They held GPUs the
inventory believed were free, so the next placement failed to allocate, with an
error naming the new process rather than the old one.

The fix belongs on the agent, not in a database. A row can say "running" about a
process that died; the fleet cannot.

- the supervisor writes the full shard spec, not just a pid and command line. A
  pid identifies a process; the spec is what lets it be described back to the
  control plane and matched to a replica.
- Adopt takes over an engine a previous agent left running. It is not this
  process's child, so it cannot be waited on: it is watched by polling and
  stopped by signalling its group directly. Its earlier output is gone and the
  log says so rather than presenting emptiness as silence.
- every Register reports what the agent is running, not only the first, because
  a reconnect may be a control plane restart and has to recover the same way.
- the reconciler rebuilds replicas from those reports and reclaims their GPUs.
  Anything whose model or profile left the catalog is stopped; nothing will
  route to it and it is holding cards.
- an engine the agent cannot identify is killed. A process holding GPUs that
  nothing can describe is worse than no process, because placement hands those
  GPUs out again.

Stop was also wrong for adopted shards: it checked for an exec.Cmd, which they
do not have, and reported them stopped without killing anything.

TestAgentReconnects had been passing for no reason. httptest's
CloseClientConnections does nothing to an h2c stream — prior-knowledge h2c
hijacks the connection and httptest stops tracking hijacked ones — so the agent
never reconnected, and the assertion on the connection map passed on a stale
entry. Tests now sever connections through their own listener and count opened
streams.

Plan §11 also records what multi-node still needs, since placement landing early
makes it look further along than it is: workers never become Ready because the
agent health-checks every shard while only rank 0 serves HTTP; RayCommand is
written and tested but never called; and startup is fire-and-forget.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 06:15:04 +02:00
1 parent 40427b2705
commit 75a8e76706
13 files changed
+1312 -145

No files matched your search

+42
View File
@@ -57,6 +57,43 @@ message Register {
// engine_versions decides which CLI flags and metric names apply; both drift
// between engine releases (plan §14).
map<string, string> engine_versions = 8;
// running_shards are the engines this agent is already supervising, including
// ones adopted from a previous agent.
//
// This is what lets a control plane restart recover instead of orphaning live
// engines. Without it the control plane forgets every replica, places new
// ones, and the old engines keep holding GPUs the inventory believes are free
// — so the new engines fail to allocate, with an error naming the wrong
// process (plan §6.5).
repeated RunningShard running_shards = 9;
}
// RunningShard describes an engine already running on a node.
//
// It carries enough for the control plane to rebuild the replica it belongs to,
// which is more than a shard id: the replica's identity, its pool and fabric,
// and the leader endpoint clients reach it through.
message RunningShard {
string replica_id = 1;
string shard_id = 2;
int32 rank = 3;
int32 nnodes = 4;
repeated int32 gpu_indices = 5;
repeated string hcas = 6;
int32 http_port = 7;
ShardStatus.State state = 8;
// model and profile name the catalog entry this shard was started from.
string model = 9;
string profile = 10;
// pool and fabric are echoed back from the StartShard that created it; the
// agent has no way to derive them.
string pool = 11;
string fabric = 12;
string leader_addr = 13;
int32 leader_port = 14;
}
message GPU {
@@ -142,6 +179,11 @@ message StartShard {
EngineProfile profile = 11;
int64 startup_timeout_ms = 12;
// pool and fabric are control plane concepts the agent cannot derive. They
// are sent so the agent can echo them back in RunningShard after a restart.
string pool = 13;
string fabric = 14;
}
message EngineProfile {