diff --git a/CHANGELOG.md b/CHANGELOG.md index 94a6437..68769f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,34 @@ APIs change between minors without a deprecation period. ## Unreleased +### A second control plane, and a way to fail over to it + +Leadership is a lease in the store, so two control planes against one database +cannot both place work — which they otherwise would, both seeing a model +wanting replicas and none running, leaving twice the replicas each holding GPUs +the other believes are free. + +A standby also refuses agent streams. Commands travel down the stream the agent +opened, so a standby holding one is a node the leader cannot command, and the +leader's placements would fail with "no agent connected" for a node that is +plainly connected. Agents therefore take a list of addresses rather than one, +trying each in turn: a standby refusing, a stopped control plane refusing the +connection and a crashed one never answering are the same thing from a node. + +Measured on the cluster. Leadership moved three seconds after the leader was +stopped — it released the lease on the way out rather than letting it lapse — +and all five agents had registered with the standby six seconds later. The +standby adopted every replica rather than restarting any: six replicas, all +ready, engines that never stopped serving. + +Everything is unset-means-leader, so a single control plane and every +deployment that existed before this behave exactly as they did. + +What this does not do is move client traffic. The standby served inference +correctly on its own port while the public address returned 502, because +nothing in front of them moved — that needs a load balancer or a shared +address, and it is the remaining piece. + ### PostgreSQL The store runs on PostgreSQL, chosen by what the address looks like: a path is