Keep the store under Patroni, and back up from a store of several hosts

deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as
running on the reference cluster, and the address that follows the primary.
Backups read such an address as libpq host and port lists and prefer a
standby, rather than resolving the list as one name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -75,6 +75,20 @@ and a figure the driver does not give is left out rather than shown as zero.
   each card and whose quota it counts against, for joining the GPU series to
   models and tenants.
 
+### A store that survives losing its machine
+
+`deploy/ha/` now covers the store as well as the control planes: PostgreSQL
+under Patroni on two machines, its leader lock in a three-member etcd, and
+llmgw given every member with `target_session_attrs=read-write`. Measured: a
+planned switchover cost one failed lease renewal; switching off the primary's
+machine promoted the standby in 24 seconds with the gateway serving
+throughout. Example configurations are beside it.
+
+`llmgw server backup` reads a store address of several hosts. It took the
+list for one host name and reached the database only by falling through to
+the second; it now hands libpq the list, and dumps from a standby when there
+is one.
+
 ### People, limits and usage are one page
 
 Managing a team meant its members on the people page, its quota on the quotas

deploy/ha/README.md

@@ -85,3 +85,54 @@ auth:
 
 See [docs/install.md](../../docs/install.md) for moving an existing
 installation onto PostgreSQL without reissuing every key.
+
+## The store itself: PostgreSQL under Patroni
+
+Two control planes on one PostgreSQL have moved the single point of failure,
+not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
+other, and promotes the standby when the primary is lost. It keeps its leader
+lock in etcd, and etcd decides only with a majority — so three members, not
+two: two machines that lose sight of each other must not both decide they are
+the primary. The third member can be any machine that is up; it holds a vote,
+not data.
+
+On the reference cluster:
+
+| machine | etcd | PostgreSQL |
+|---|---|---|
+| llmgw-ctl (control plane) | member | — |
+| llmgw-pg01 | member | Patroni |
+| llmgw-ctl02 (second control plane) | member | Patroni |
+
+`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
+`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
+(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
+Start Patroni on the first node, which initialises the cluster, then on the
+second, which clones it. Create the `llmgw` role and database on the leader,
+and move an existing store into it with `pg_dump --no-owner` and
+`pg_restore --no-owner`.
+
+**Give llmgw every member, and ask for the writable one:**
+
+```
+postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
+```
+
+The driver connects to whichever member accepts writes, and reconnects to the
+new one after a failover. No floating address or HAProxy is needed for that.
+`llmgw server backup` reads the same address and dumps from a standby when
+there is one.
+
+Measured on the reference cluster, with the gateway serving throughout:
+
+- **Planned switchover** (`patronictl switchover`): the control plane logged
+  one failed lease renewal, reconnected to the new primary, and kept its
+  leadership; nothing else noticed.
+- **The primary's machine switched off**: the standby was promoted 24 seconds
+  later — the 30-second TTL in `patroni.yml` — and the control plane rode it
+  out without losing leadership or a replica. Started again, the old primary
+  rejoined as a standby by itself.
+
+Not done here, and worth doing in production: a watchdog (`softdog`) so a
+primary that has lost the lock but not died cannot keep accepting writes, TLS
+on etcd and between PostgreSQL members, and more than one standby.

deploy/ha/etcd.default.example

@@ -0,0 +1,10 @@
+# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
+ETCD_NAME=llmgw-pg01
+ETCD_DATA_DIR=/var/lib/etcd/default
+ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
+ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
+ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
+ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
+ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
+ETCD_INITIAL_CLUSTER_STATE=new
+ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni

deploy/ha/patroni.yml.example

@@ -0,0 +1,68 @@
+# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
+# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
+scope: llmgw
+namespace: /service/
+name: llmgw-pg01
+
+restapi:
+  listen: 10.0.0.149:8008
+  connect_address: 10.0.0.149:8008
+
+etcd3:
+  hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
+
+bootstrap:
+  dcs:
+    ttl: 30
+    loop_wait: 10
+    retry_timeout: 10
+    maximum_lag_on_failover: 1048576
+    postgresql:
+      use_pg_rewind: true
+      use_slots: true
+      parameters:
+        wal_level: replica
+        hot_standby: "on"
+        max_wal_senders: 10
+        max_replication_slots: 10
+        wal_keep_size: 1GB
+        wal_log_hints: "on"
+        password_encryption: scram-sha-256
+  initdb:
+    - encoding: UTF8
+    - data-checksums
+
+postgresql:
+  listen: 127.0.0.1,10.0.0.149:5432
+  connect_address: 10.0.0.149:5432
+  data_dir: /var/lib/postgresql/16/llmgw
+  bin_dir: /usr/lib/postgresql/16/bin
+  pgpass: /var/lib/postgresql/.pgpass_patroni
+  authentication:
+    superuser:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+    replication:
+      username: replicator
+      password: "<from /root/llmgw-pg>"
+    rewind:
+      username: postgres
+      password: "<from /root/llmgw-pg>"
+  parameters:
+    unix_socket_directories: /var/run/postgresql
+  pg_hba:
+    - local all all peer
+    - host all all 127.0.0.1/32 scram-sha-256
+    - host replication replicator 10.0.0.149/32 scram-sha-256
+    - host replication replicator 10.0.0.150/32 scram-sha-256
+    - host all postgres 10.0.0.149/32 scram-sha-256
+    - host all postgres 10.0.0.150/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.140/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.149/32 scram-sha-256
+    - host llmgw llmgw 10.0.0.150/32 scram-sha-256
+
+tags:
+  nofailover: false
+  noloadbalance: false
+  clonefrom: false
+  nosync: false

internal/cli/backup.go

@@ -178,30 +178,77 @@ func addStore(ctx context.Context, tw *tar.Writer, addr, scratch string) error {
 // command line: a DSN carries the password, and every user on the machine
 // can read another process's arguments.
 func pgDump(ctx context.Context, dsn, out string) error {
-	u, err := url.Parse(dsn)
+	vars, err := pgEnv(dsn)
 	if err != nil {
-		return fmt.Errorf("the store address is not a URL")
+		return err
 	}
-	env := os.Environ()
+	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
+	cmd.Env = append(os.Environ(), vars...)
+	if b, err := cmd.CombinedOutput(); err != nil {
+		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	}
+	return nil
+}
+
+// pgEnv turns a store address into libpq's environment.
+//
+// Several hosts are allowed, as pgx allows them — "h1:5432,h2:5432", the
+// members of a replicated cluster — and become libpq's own lists in PGHOST and
+// PGPORT. url.Parse does not read such a host: it took everything before the
+// last colon as one name, which libpq could not resolve, and a backup worked
+// only because it fell through to the second host.
+func pgEnv(dsn string) ([]string, error) {
+	scheme, rest, ok := strings.Cut(dsn, "://")
+	if !ok {
+		return nil, fmt.Errorf("the store address is not a URL")
+	}
+	authority, tail := rest, ""
+	if i := strings.IndexAny(rest, "/?"); i >= 0 {
+		authority, tail = rest[:i], rest[i:]
+	}
+	userinfo, hostlist := "", authority
+	if i := strings.LastIndexByte(authority, '@'); i >= 0 {
+		userinfo, hostlist = authority[:i+1], authority[i+1:]
+	}
+	var hosts, ports []string
+	for _, hp := range strings.Split(hostlist, ",") {
+		u, err := url.Parse("x://" + hp)
+		if err != nil || u.Hostname() == "" {
+			return nil, fmt.Errorf("the store address has a host that is not one: %q", hp)
+		}
+		port := u.Port()
+		if port == "" {
+			port = "5432"
+		}
+		hosts, ports = append(hosts, u.Hostname()), append(ports, port)
+	}
+	// The rest of the address with one placeholder host, which url.Parse reads.
+	u, err := url.Parse(scheme + "://" + userinfo + "host" + tail)
+	if err != nil {
+		return nil, fmt.Errorf("the store address is not a URL")
+	}
+	var env []string
 	set := func(k, v string) {
 		if v != "" {
 			env = append(env, k+"="+v)
 		}
 	}
-	set("PGHOST", u.Hostname())
-	set("PGPORT", u.Port())
+	set("PGHOST", strings.Join(hosts, ","))
+	set("PGPORT", strings.Join(ports, ","))
 	set("PGUSER", u.User.Username())
 	if pw, ok := u.User.Password(); ok {
 		set("PGPASSWORD", pw)
 	}
 	set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
 	set("PGSSLMODE", u.Query().Get("sslmode"))
-	cmd := exec.CommandContext(ctx, "pg_dump", "--format=custom", "--no-owner", "--file="+out) //nolint:gosec // fixed binary, our own file
-	cmd.Env = env
-	if b, err := cmd.CombinedOutput(); err != nil {
-		return fmt.Errorf("pg_dump: %v: %s", err, strings.TrimSpace(string(b)))
+	// A standby if there is one, else whichever answers — not the address's
+	// read-write: a dump only reads, it need not load the primary, and one
+	// that insists on the primary fails exactly while the cluster is failing
+	// over.
+	if len(hosts) > 1 {
+		set("PGTARGETSESSIONATTRS", "prefer-standby")
 	}
-	return nil
+	return env, nil
 }
 
 func addTree(tw *tar.Writer, root, as string) error {

internal/cli/backup_test.go

@@ -139,3 +139,26 @@ func TestNoScratchCopyOfTheStoreIsLeftBehind(t *testing.T) {
 		}
 	}
 }
+
+func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
+	// The address of a Patroni cluster: url.Parse read its host as one name,
+	// and pg_dump reached the database only by falling through to the second.
+	env, err := pgEnv("postgres://llmgw:s3cret@10.0.0.149:5432,10.0.0.150/llmgw?sslmode=disable&target_session_attrs=read-write")
+	if err != nil {
+		t.Fatal(err)
+	}
+	got := strings.Join(env, " ")
+	for _, want := range []string{"PGHOST=10.0.0.149,10.0.0.150", "PGPORT=5432,5432", "PGUSER=llmgw",
+		"PGPASSWORD=s3cret", "PGDATABASE=llmgw", "PGSSLMODE=disable", "PGTARGETSESSIONATTRS=prefer-standby"} {
+		if !strings.Contains(got, want) {
+			t.Errorf("%s is not in %s", want, got)
+		}
+	}
+	one, err := pgEnv("postgres://u:p@db.internal:6432/llmgw")
+	if err != nil || !strings.Contains(strings.Join(one, " "), "PGHOST=db.internal PGPORT=6432") {
+		t.Errorf("one host: %v, %v", one, err)
+	}
+	if _, err := pgEnv("postgres://u:p@a:5432,:5432/x"); err == nil {
+		t.Error("an empty host in the list was accepted")
+	}
+}