A lost store machine is given up on in seconds, and a new leader waits for its judge
CI / test (push) Failing after 30m6s

Measured on three VMs: cutting the store primary's power cost about 45 s of
hanging requests. New connections tried the dead member first with no
connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of
both cost a further 17 s of 503s while the new leader knew no judge.

The store dials each member for 3 s and closes connections to a machine that
is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with
ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing
clusters through its API. Judges that all fail at once are asked again
every 250 ms while the policy's budget lasts, and reported once.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-15 20:33:13 +02:00
1 parent eeaffd764d
commit 5a4b022dcd
12 files changed
+313 -23

No files matched your search

+10
View File
@@ -87,6 +87,16 @@ APIs change between minors without a deprecation period.
### Fixed
- **A lost store machine stops costing requests most of a minute.** The store's connections try each member for 3 seconds,
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
keepalive, and `TCP_USER_TIMEOUT` on Linux). Patroni replaces a lost primary within 20 seconds (`ttl` 20, `loop_wait`
5, `retry_timeout` 5). Running `server init --patroni` again applies this to an existing cluster. Measured on three
VMs with the primary's power cut: about 45 seconds of hanging requests before.
- **A new leader waits for the judge instead of refusing at once.** The moment a control plane takes the lead it knows no
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy's `latency_budget` lasts,
so `fail: closed` no longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
judge that is slow is not asked again, and an alarm is raised once per request, not once per try.
- Node discovery for Prometheus listed only the control plane asked and the
store, so a machine that is both was a control plane to one and a store
member to the other — scraped twice — and a standby asked alone lost every