Leadership stays put while the store's primary is replaced

Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.

A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -92,6 +92,12 @@ APIs change between minors without a deprecation period.
   keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
   5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
   VMs with the primary's power cut: about 45 seconds of hanging requests before.
+- **Leadership stays put while the store's primary is replaced.** Replacing a lost primary takes longer than the
+  leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
+  took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
+  round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
+  the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
+  8 seconds (was 15), so it is on its way to the new leader as that one takes over.
 - **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
   judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
   so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A

docs/reference.md

@@ -948,7 +948,7 @@ work stands without being renewed.
 Only meaningful with a shared store, where a second control plane can
 see the claim. Long enough to survive a slow query or a brief network
 stall, short enough that a standby takes over before anyone has finished
-reading the alert. Zero takes the default.
+reading the alert. Zero takes the default, 10s.
 
 ## CustomNode
 

internal/agent/agent.go

@@ -1417,11 +1417,14 @@ func Client(cfg tlsconf.Config, timeout time.Duration) (*http.Client, error) {
 // off — is never closed: nothing sends the FIN, and the agent's heartbeats sit
 // in the socket's buffer until the kernel gives up retransmitting, fifteen
 // minutes later. Found by switching off a leader: the standby took over in 14
-// seconds and the node stayed with the dead one. An HTTP/2 ping after ten
-// quiet seconds, given five to answer, finds it in a quarter of a minute.
+// seconds and the node stayed with the dead one. An HTTP/2 ping after five
+// quiet seconds, given three to answer, finds it in eight: by the time the
+// standby leads, as it does within the lease's ten seconds, the node is on
+// its way to it. At fifteen, with the lease at fifteen too, the new leader
+// went without its nodes, and so its judges, for half a minute.
 const (
-	streamIdle = 10 * time.Second
-	streamPing = 5 * time.Second
+	streamIdle = 5 * time.Second
+	streamPing = 3 * time.Second
 )
 
 // localizeSpec fills in what only this node knows.

internal/config/config.go

@@ -550,7 +550,7 @@ type Control struct {
 	// Only meaningful with a shared store, where a second control plane can
 	// see the claim. Long enough to survive a slow query or a brief network
 	// stall, short enough that a standby takes over before anyone has finished
-	// reading the alert. Zero takes the default.
+	// reading the alert. Zero takes the default, 10s.
 	LeaseTTL time.Duration `koanf:"lease_ttl"`
 }
 

internal/control/leadership.go

@@ -3,6 +3,7 @@ package control
 import (
 	"context"
 	"crypto/rand"
+	"database/sql"
 	"encoding/hex"
 	"errors"
 	"log/slog"
@@ -59,6 +60,17 @@ type Leadership struct {
 
 	hookMu sync.Mutex
 	onLead []func()
+
+	// cut is a store this process could not reach, and back when it was
+	// reached again: Run's alone, so unlocked.
+	cut  bool
+	back time.Time
+}
+
+// LeaseReader reads who holds a lease without competing for it: a store
+// that has it lets a standby hold back after the store returns.
+type LeaseReader interface {
+	Leadership(ctx context.Context, role string) (store.Lease, error)
 }
 
 // OnLead is called each time this process becomes the leader, after
@@ -91,8 +103,9 @@ func NewLeadership(st LeaseStore, ttl time.Duration, log *slog.Logger) *Leadersh
 	if ttl <= 0 {
 		// Long enough to survive a slow database or a brief network stall,
 		// short enough that a standby takes over before anyone has finished
-		// reading the alert.
-		ttl = 15 * time.Second
+		// reading the alert. A store out of reach for longer — its primary
+		// being replaced — does not move it: see holdBack.
+		ttl = 10 * time.Second
 	}
 	l := &Leadership{
 		store: st,
@@ -146,6 +159,9 @@ func (l *Leadership) attempt(ctx context.Context) {
 	if l.stopped.Load() {
 		return
 	}
+	if l.holdBack(ctx) {
+		return
+	}
 	lease, err := l.store.AcquireLeadership(ctx, controlRole, l.owner, l.ttl)
 	switch {
 	case err == nil && l.stopped.Load():
@@ -177,9 +193,47 @@ func (l *Leadership) attempt(ctx context.Context) {
 		// stop a cluster from healing itself.
 		l.log.Warn("could not reach the store to renew leadership; keeping it for now",
 			"err", err)
+		l.cut, l.back = true, time.Time{}
 	}
 }
 
+// holdBack is a standby leaving a lapsed lease alone for one round after the
+// store returns from being out of reach.
+//
+// A store's primary that is lost takes longer to replace than the lease
+// lives, so the lease has lapsed by the time the store answers again — for
+// the leader as for the standbys, which all failed to reach it alike. The
+// first to ask then took it, and it was as often a standby as the leader:
+// leadership moved for nothing, and the new leader spent half a minute
+// learning from the agents where the judges were, refusing requests
+// meanwhile. The leader renews within a round of the store's return, so a
+// standby that waits that round takes the lease only from a leader that is
+// gone.
+func (l *Leadership) holdBack(ctx context.Context) bool {
+	if l.leader.Load() || !l.cut {
+		return false
+	}
+	r, ok := l.store.(LeaseReader)
+	if !ok {
+		l.cut = false
+		return false
+	}
+	if _, err := r.Leadership(ctx, controlRole); err != nil && !errors.Is(err, sql.ErrNoRows) {
+		l.back = time.Time{}
+		return true
+	}
+	if l.back.IsZero() {
+		l.back = time.Now()
+		l.log.Info("the store answers again; leaving the lease to its holder for a round", "owner", l.owner)
+	}
+	// A round less a tenth, for the ticker's own jitter.
+	if time.Since(l.back) < l.renew-l.renew/10 {
+		return true
+	}
+	l.cut, l.back = false, time.Time{}
+	return false
+}
+
 // StepDown gives the lease up now, for a process on its way out: called
 // before the agents are let go, so the standby they move to is already the
 // leader, and before the process exits, which would otherwise race the

internal/control/leadership_test.go

@@ -1,6 +1,8 @@
 package control
 
 import (
+	"context"
+	"errors"
 	"log/slog"
 	"path/filepath"
 	"testing"
@@ -75,3 +77,80 @@ func TestAStandbyIsToldWhenItBecomesTheLeader(t *testing.T) {
 		t.Fatal("b became the leader and was not told")
 	}
 }
+
+// cuttable is a store that can be out of reach, as one whose primary was lost.
+type cuttable struct {
+	*store.DB
+	down bool
+}
+
+func (c *cuttable) AcquireLeadership(ctx context.Context, role, owner string, ttl time.Duration) (store.Lease, error) {
+	if c.down {
+		return store.Lease{}, errors.New("dial error: timeout")
+	}
+	return c.DB.AcquireLeadership(ctx, role, owner, ttl)
+}
+
+func (c *cuttable) Leadership(ctx context.Context, role string) (store.Lease, error) {
+	if c.down {
+		return store.Lease{}, errors.New("dial error: timeout")
+	}
+	return c.DB.Leadership(ctx, role)
+}
+
+// The store out of reach for longer than the lease lives, as while Patroni
+// promotes a standby: when it answers again the lease has lapsed for all,
+// and the leader keeps it rather than a standby taking it for nothing. A
+// leader that is truly gone is still replaced, a round later.
+func TestAStoreThatReturnsLeavesTheLeaderLeading(t *testing.T) {
+	st, err := store.Open(t.Context(), filepath.Join(t.TempDir(), "s.db"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer st.Close()
+	c := &cuttable{DB: st}
+	log := slog.New(slog.DiscardHandler)
+	ctx := t.Context()
+	const ttl = 150 * time.Millisecond
+	a, b := NewLeadership(c, ttl, log), NewLeadership(c, ttl, log)
+	a.attempt(ctx)
+	b.attempt(ctx)
+	if !a.IsLeader() || b.IsLeader() {
+		t.Fatal("a should lead, b stand by")
+	}
+
+	c.down = true
+	a.attempt(ctx)
+	b.attempt(ctx)
+	time.Sleep(ttl + 20*time.Millisecond) // the lease lapses while nobody reaches the store
+	c.down = false
+
+	b.attempt(ctx) // the standby is first to ask
+	if b.IsLeader() {
+		t.Fatal("a standby took the lease the moment the store came back")
+	}
+	a.attempt(ctx)
+	if !a.IsLeader() {
+		t.Fatal("the leader did not keep the lease")
+	}
+	time.Sleep(a.renew)
+	b.attempt(ctx)
+	if b.IsLeader() {
+		t.Error("the standby took the lease from a leader renewing it")
+	}
+
+	// Now the leader is gone: the standby takes over after its round.
+	c.down = true
+	b.attempt(ctx)
+	time.Sleep(ttl + 20*time.Millisecond)
+	c.down = false
+	b.attempt(ctx)
+	if b.IsLeader() {
+		t.Fatal("took it at once")
+	}
+	time.Sleep(b.renew)
+	b.attempt(ctx)
+	if !b.IsLeader() {
+		t.Error("a leader that was gone was not replaced")
+	}
+}

internal/docgen/reference_gen.go

@@ -91,7 +91,7 @@ var Reference = []Block{
 		{Key: "timezone", Type: "string", Doc: "Timezone is where models' scaling schedules are read, an IANA name\nlike Europe/Istanbul; empty is the control plane's own.", Nested: ""},
 		{Key: "eval_dir", Type: "string", Doc: "EvalDir is where quality suites are read from by name, for `llmgw ctl\neval MODEL --suite NAME`, a model's eval gate and the console. The\nsuites the binary carries (arithmetic, json-format) are found when it\nhas none of the name; a suite here of the same name replaces one.", Nested: ""},
 		{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
-		{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
+		{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default, 10s.", Nested: ""},
 	}},
 	{Package: "catalog", Name: "CustomNode", Title: "CustomNode", Doc: "", Fields: []Field{
 		{Key: "repo", Type: "string", Doc: "Repo is the repository, https only.", Nested: ""},