The previous commit's adoption fix made a deeper problem visible instead of
fixing it. Replica ids were the model name and a counter, and the counter
restarted with the process — so a control plane that came back placed a
replica called llama-1 while an engine started before the restart, still
running on a different node, also called itself llama-1.
Two nodes then claimed the same shard id. Adoption handed it back and forth
between them, every placement landed on a port an orphan was still bound to,
and the engine failed to bind, which failed the replica, which placed another
one on the same port. On the real cluster that loop ran for minutes:
couldn't bind HTTP server socket, hostname: 0.0.0.0, port: 20002
Ids now carry four random bytes, which costs nothing and makes the collision
impossible. The "already known" branch in adoption goes back to meaning what
it says: an agent reconnecting, not a name clash across restarts.
Ports a stray still holds are also reserved now. Stopping is not instant and
an engine keeps its socket until it exits, so the allocator has to know about
sockets no replica claims — otherwise the same loop reappears in the window
between telling an agent to stop something and it actually stopping.
Two of the tests written for the previous commit had quietly become vacuous
under the new ids, asserting things about a literal llama-1 that no longer
exists. They capture the id now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
