Record what failover actually did

Three seconds to move leadership, six for every agent to arrive, and every
replica adopted rather than restarted. Also what it did not do: the public
address returned 502 while the standby served correctly on its own port,
because nothing in front of the two moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-25 19:39:55 +02:00
1 parent 4ba84fa90e
commit b9c2be6cc2
1 file changed
+28
+28
View File
@@ -19,6 +19,34 @@ APIs change between minors without a deprecation period.
## Unreleased
### A second control plane, and a way to fail over to it
Leadership is a lease in the store, so two control planes against one database
cannot both place work — which they otherwise would, both seeing a model
wanting replicas and none running, leaving twice the replicas each holding GPUs
the other believes are free.
A standby also refuses agent streams. Commands travel down the stream the agent
opened, so a standby holding one is a node the leader cannot command, and the
leader's placements would fail with "no agent connected" for a node that is
plainly connected. Agents therefore take a list of addresses rather than one,
trying each in turn: a standby refusing, a stopped control plane refusing the
connection and a crashed one never answering are the same thing from a node.
Measured on the cluster. Leadership moved three seconds after the leader was
stopped — it released the lease on the way out rather than letting it lapse —
and all five agents had registered with the standby six seconds later. The
standby adopted every replica rather than restarting any: six replicas, all
ready, engines that never stopped serving.
Everything is unset-means-leader, so a single control plane and every
deployment that existed before this behave exactly as they did.
What this does not do is move client traffic. The standby served inference
correctly on its own port while the public address returned 502, because
nothing in front of them moved — that needs a load balancer or a shared
address, and it is the remaining piece.
### PostgreSQL
The store runs on PostgreSQL, chosen by what the address looks like: a path is