CI / test (push) Failing after 30m6s
Measured on three VMs: cutting the store primary's power cost about 45 s of hanging requests. New connections tried the dead member first with no connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of both cost a further 17 s of 503s while the new leader knew no judge. The store dials each member for 3 s and closes connections to a machine that is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing clusters through its API. Judges that all fail at once are asked again every 250 ms while the policy's budget lasts, and reported once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
24 lines
589 B
Go
24 lines
589 B
Go
package store
|
|
|
|
import (
|
|
"syscall"
|
|
"time"
|
|
|
|
"golang.org/x/sys/unix"
|
|
)
|
|
|
|
// userTimeout sets TCP_USER_TIMEOUT: data sent and not acknowledged for that
|
|
// long closes the connection, where keepalive probes wait for it to go idle.
|
|
func userTimeout(d time.Duration) func(network, address string, c syscall.RawConn) error {
|
|
return func(_, _ string, c syscall.RawConn) error {
|
|
var serr error
|
|
err := c.Control(func(fd uintptr) {
|
|
serr = unix.SetsockoptInt(int(fd), unix.IPPROTO_TCP, unix.TCP_USER_TIMEOUT, int(d.Milliseconds()))
|
|
})
|
|
if err != nil {
|
|
return err
|
|
}
|
|
return serr
|
|
}
|
|
}
|