Keep the store under Patroni, and back up from a store of several hosts
deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as running on the reference cluster, and the address that follows the primary. Backups read such an address as libpq host and port lists and prefer a standby, rather than resolving the list as one name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
febcf251e4
commit
7a562ac1e7
6 files changed
+223
-10
No files matched your search
@@ -85,3 +85,54 @@ auth:
|
||||
|
||||
See [docs/install.md](../../docs/install.md) for moving an existing
|
||||
installation onto PostgreSQL without reissuing every key.
|
||||
|
||||
## The store itself: PostgreSQL under Patroni
|
||||
|
||||
Two control planes on one PostgreSQL have moved the single point of failure,
|
||||
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
|
||||
other, and promotes the standby when the primary is lost. It keeps its leader
|
||||
lock in etcd, and etcd decides only with a majority — so three members, not
|
||||
two: two machines that lose sight of each other must not both decide they are
|
||||
the primary. The third member can be any machine that is up; it holds a vote,
|
||||
not data.
|
||||
|
||||
On the reference cluster:
|
||||
|
||||
| machine | etcd | PostgreSQL |
|
||||
|---|---|---|
|
||||
| llmgw-ctl (control plane) | member | — |
|
||||
| llmgw-pg01 | member | Patroni |
|
||||
| llmgw-ctl02 (second control plane) | member | Patroni |
|
||||
|
||||
`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
|
||||
`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
|
||||
(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
|
||||
Start Patroni on the first node, which initialises the cluster, then on the
|
||||
second, which clones it. Create the `llmgw` role and database on the leader,
|
||||
and move an existing store into it with `pg_dump --no-owner` and
|
||||
`pg_restore --no-owner`.
|
||||
|
||||
**Give llmgw every member, and ask for the writable one:**
|
||||
|
||||
```
|
||||
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
|
||||
```
|
||||
|
||||
The driver connects to whichever member accepts writes, and reconnects to the
|
||||
new one after a failover. No floating address or HAProxy is needed for that.
|
||||
`llmgw server backup` reads the same address and dumps from a standby when
|
||||
there is one.
|
||||
|
||||
Measured on the reference cluster, with the gateway serving throughout:
|
||||
|
||||
- **Planned switchover** (`patronictl switchover`): the control plane logged
|
||||
one failed lease renewal, reconnected to the new primary, and kept its
|
||||
leadership; nothing else noticed.
|
||||
- **The primary's machine switched off**: the standby was promoted 24 seconds
|
||||
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
|
||||
out without losing leadership or a replica. Started again, the old primary
|
||||
rejoined as a standby by itself.
|
||||
|
||||
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
||||
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
||||
on etcd and between PostgreSQL members, and more than one standby.
|
||||
@@ -0,0 +1,10 @@
|
||||
# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
|
||||
ETCD_NAME=llmgw-pg01
|
||||
ETCD_DATA_DIR=/var/lib/etcd/default
|
||||
ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
|
||||
ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
|
||||
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
|
||||
ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
|
||||
ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
|
||||
ETCD_INITIAL_CLUSTER_STATE=new
|
||||
ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni
|
||||
@@ -0,0 +1,68 @@
|
||||
# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
|
||||
# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
|
||||
scope: llmgw
|
||||
namespace: /service/
|
||||
name: llmgw-pg01
|
||||
|
||||
restapi:
|
||||
listen: 10.0.0.149:8008
|
||||
connect_address: 10.0.0.149:8008
|
||||
|
||||
etcd3:
|
||||
hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
|
||||
|
||||
bootstrap:
|
||||
dcs:
|
||||
ttl: 30
|
||||
loop_wait: 10
|
||||
retry_timeout: 10
|
||||
maximum_lag_on_failover: 1048576
|
||||
postgresql:
|
||||
use_pg_rewind: true
|
||||
use_slots: true
|
||||
parameters:
|
||||
wal_level: replica
|
||||
hot_standby: "on"
|
||||
max_wal_senders: 10
|
||||
max_replication_slots: 10
|
||||
wal_keep_size: 1GB
|
||||
wal_log_hints: "on"
|
||||
password_encryption: scram-sha-256
|
||||
initdb:
|
||||
- encoding: UTF8
|
||||
- data-checksums
|
||||
|
||||
postgresql:
|
||||
listen: 127.0.0.1,10.0.0.149:5432
|
||||
connect_address: 10.0.0.149:5432
|
||||
data_dir: /var/lib/postgresql/16/llmgw
|
||||
bin_dir: /usr/lib/postgresql/16/bin
|
||||
pgpass: /var/lib/postgresql/.pgpass_patroni
|
||||
authentication:
|
||||
superuser:
|
||||
username: postgres
|
||||
password: "<from /root/llmgw-pg>"
|
||||
replication:
|
||||
username: replicator
|
||||
password: "<from /root/llmgw-pg>"
|
||||
rewind:
|
||||
username: postgres
|
||||
password: "<from /root/llmgw-pg>"
|
||||
parameters:
|
||||
unix_socket_directories: /var/run/postgresql
|
||||
pg_hba:
|
||||
- local all all peer
|
||||
- host all all 127.0.0.1/32 scram-sha-256
|
||||
- host replication replicator 10.0.0.149/32 scram-sha-256
|
||||
- host replication replicator 10.0.0.150/32 scram-sha-256
|
||||
- host all postgres 10.0.0.149/32 scram-sha-256
|
||||
- host all postgres 10.0.0.150/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.140/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.149/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.150/32 scram-sha-256
|
||||
|
||||
tags:
|
||||
nofailover: false
|
||||
noloadbalance: false
|
||||
clonefrom: false
|
||||
nosync: false
|
||||
Reference in new issue
Block a user