A lost store machine is given up on in seconds, and a new leader waits for its judge
CI / test (push) Failing after 30m6s
CI / test (push) Failing after 30m6s
Measured on three VMs: cutting the store primary's power cost about 45 s of hanging requests. New connections tried the dead member first with no connect timeout, and Patroni kept its 30 s leader key. Cutting the leader of both cost a further 17 s of 503s while the new leader knew no judge. The store dials each member for 3 s and closes connections to a machine that is gone within about 10 s (keepalive, TCP_USER_TIMEOUT). Patroni runs with ttl 20, loop_wait 5 and retry_timeout 5, applied in place to existing clusters through its API. Judges that all fail at once are asked again every 250 ms while the policy's budget lasts, and reported once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
eeaffd764d
commit
5a4b022dcd
12 files changed
+313
-23
No files matched your search
@@ -2,17 +2,46 @@ package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"net"
|
||||
"time"
|
||||
|
||||
// The pure-Go PostgreSQL driver. Pure-Go is not a preference here: the
|
||||
// whole binary is built with CGO_ENABLED=0 so that it runs on a node whose
|
||||
// libc is older than the builder's, and a cgo driver would take that away
|
||||
// from every deployment, including the ones still on SQLite.
|
||||
_ "github.com/jackc/pgx/v5/stdlib"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
// How soon a store machine that is gone is given up on. A replicated store's
|
||||
// address names every member, and a new connection tries them in order: with
|
||||
// no timeout, one to a machine that had lost its power waited for the
|
||||
// kernel's own — half a minute and more — before the next member was asked,
|
||||
// and every request whose key was not cached waited with it.
|
||||
const (
|
||||
// storeConnectTimeout is per member: the next is tried after it.
|
||||
storeConnectTimeout = 3 * time.Second
|
||||
// storeDeadAfter is how long a connection's sent data may go unanswered,
|
||||
// or an idle one's probes, before it is closed: a connection to the
|
||||
// primary that was lost fails, and the next opens to the new one.
|
||||
storeDeadAfter = 10 * time.Second
|
||||
)
|
||||
|
||||
// failFast gives a connection configuration those limits, where its address
|
||||
// gave none of its own.
|
||||
func failFast(cfg *pgx.ConnConfig) {
|
||||
if cfg.ConnectTimeout == 0 {
|
||||
cfg.ConnectTimeout = storeConnectTimeout
|
||||
}
|
||||
d := &net.Dialer{
|
||||
KeepAliveConfig: net.KeepAliveConfig{Enable: true, Idle: 5 * time.Second, Interval: time.Second,
|
||||
Count: int((storeDeadAfter - 5*time.Second) / time.Second)},
|
||||
Control: userTimeout(storeDeadAfter),
|
||||
}
|
||||
cfg.DialFunc = d.DialContext
|
||||
}
|
||||
|
||||
// OpenPostgres opens the store against PostgreSQL.
|
||||
//
|
||||
// The same queries run here as on SQLite; only the placeholders differ, and
|
||||
@@ -29,12 +58,14 @@ func OpenPostgres(ctx context.Context, dsn string) (*DB, error) {
|
||||
if dsn == "" {
|
||||
return nil, fmt.Errorf("store: no PostgreSQL address configured")
|
||||
}
|
||||
db, err := sql.Open("pgx", dsn)
|
||||
cfg, err := pgx.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
// Deliberately not wrapped with the DSN: it holds a password, and an
|
||||
// error string ends up in logs, tickets and screenshots.
|
||||
return nil, fmt.Errorf("opening the PostgreSQL store: %w", err)
|
||||
}
|
||||
failFast(cfg)
|
||||
db := stdlib.OpenDB(*cfg)
|
||||
|
||||
// Sized for a control plane rather than for a fleet of application
|
||||
// servers. The request path barely touches the store — usage is batched,
|
||||
|
||||
Reference in new issue
Block a user