A lost store machine is given up on in seconds, and a new leader waits for its judge

Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.

The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-15 20:33:13 +02:00
1 parent eeaffd764d
commit 5a4b022dcd
12 files changed
+313 -23

No files matched your search

+25
View File
@@ -0,0 +1,25 @@
package store
import (
"testing"
"time"
"github.com/jackc/pgx/v5"
)
func TestAStoreMachineThatIsGoneIsGivenUpOnInSeconds(t *testing.T) {
cfg, err := pgx.ParseConfig("postgres://u:p@10.0.0.158:5432,10.0.0.159:5432/llmgw?sslmode=disable&target_session_attrs=read-write")
if err != nil {
t.Fatal(err)
}
failFast(cfg)
if cfg.ConnectTimeout != storeConnectTimeout || cfg.DialFunc == nil {
t.Errorf("timeout %s, dialer set %v", cfg.ConnectTimeout, cfg.DialFunc != nil)
}
// One the address asks for is its own.
cfg, _ = pgx.ParseConfig("postgres://u:p@h:5432/llmgw?sslmode=disable&connect_timeout=12")
failFast(cfg)
if cfg.ConnectTimeout != 12*time.Second {
t.Errorf("the address's own timeout was replaced: %s", cfg.ConnectTimeout)
}
}