Three seconds to move leadership, six for every agent to arrive, and every replica adopted rather than restarted. Also what it did not do: the public address returned 502 while the standby served correctly on its own port, because nothing in front of the two moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4ba84fa90e
commit
b9c2be6cc2
1 file changed
+28
@@ -19,6 +19,34 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### A second control plane, and a way to fail over to it
|
||||
|
||||
Leadership is a lease in the store, so two control planes against one database
|
||||
cannot both place work — which they otherwise would, both seeing a model
|
||||
wanting replicas and none running, leaving twice the replicas each holding GPUs
|
||||
the other believes are free.
|
||||
|
||||
A standby also refuses agent streams. Commands travel down the stream the agent
|
||||
opened, so a standby holding one is a node the leader cannot command, and the
|
||||
leader's placements would fail with "no agent connected" for a node that is
|
||||
plainly connected. Agents therefore take a list of addresses rather than one,
|
||||
trying each in turn: a standby refusing, a stopped control plane refusing the
|
||||
connection and a crashed one never answering are the same thing from a node.
|
||||
|
||||
Measured on the cluster. Leadership moved three seconds after the leader was
|
||||
stopped — it released the lease on the way out rather than letting it lapse —
|
||||
and all five agents had registered with the standby six seconds later. The
|
||||
standby adopted every replica rather than restarting any: six replicas, all
|
||||
ready, engines that never stopped serving.
|
||||
|
||||
Everything is unset-means-leader, so a single control plane and every
|
||||
deployment that existed before this behave exactly as they did.
|
||||
|
||||
What this does not do is move client traffic. The standby served inference
|
||||
correctly on its own port while the public address returned 502, because
|
||||
nothing in front of them moved — that needs a load balancer or a shared
|
||||
address, and it is the remaining piece.
|
||||
|
||||
### PostgreSQL
|
||||
|
||||
The store runs on PostgreSQL, chosen by what the address looks like: a path is
|
||||
|
||||
Reference in new issue
Block a user