From b9c2be6cc226802b33fb9d63064b48ac31cafdfb Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Fri, 25 Sep 2026 19:39:55 +0200 Subject: [PATCH] Record what failover actually did Three seconds to move leadership, six for every agent to arrive, and every replica adopted rather than restarted. Also what it did not do: the public address returned 502 while the standby served correctly on its own port, because nothing in front of the two moved. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 94a64377..68769f23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,34 @@ APIs change between minors without a deprecation period. ## Unreleased +### A second control plane, and a way to fail over to it + +Leadership is a lease in the store, so two control planes against one database +cannot both place work — which they otherwise would, both seeing a model +wanting replicas and none running, leaving twice the replicas each holding GPUs +the other believes are free. + +A standby also refuses agent streams. Commands travel down the stream the agent +opened, so a standby holding one is a node the leader cannot command, and the +leader's placements would fail with "no agent connected" for a node that is +plainly connected. Agents therefore take a list of addresses rather than one, +trying each in turn: a standby refusing, a stopped control plane refusing the +connection and a crashed one never answering are the same thing from a node. + +Measured on the cluster. Leadership moved three seconds after the leader was +stopped — it released the lease on the way out rather than letting it lapse — +and all five agents had registered with the standby six seconds later. The +standby adopted every replica rather than restarting any: six replicas, all +ready, engines that never stopped serving. + +Everything is unset-means-leader, so a single control plane and every +deployment that existed before this behave exactly as they did. + +What this does not do is move client traffic. The standby served inference +correctly on its own port while the public address returned 502, because +nothing in front of them moved — that needs a load balancer or a shared +address, and it is the remaining piece. + ### PostgreSQL The store runs on PostgreSQL, chosen by what the address looks like: a path is