Keep the store under Patroni, and back up from a store of several hosts
deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as running on the reference cluster, and the address that follows the primary. Backups read such an address as libpq host and port lists and prefer a standby, rather than resolving the list as one name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
febcf251e4
commit
7a562ac1e7
6 files changed
+223
-10
No files matched your search
@@ -85,3 +85,54 @@ auth:
|
||||
|
||||
See [docs/install.md](../../docs/install.md) for moving an existing
|
||||
installation onto PostgreSQL without reissuing every key.
|
||||
|
||||
## The store itself: PostgreSQL under Patroni
|
||||
|
||||
Two control planes on one PostgreSQL have moved the single point of failure,
|
||||
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
|
||||
other, and promotes the standby when the primary is lost. It keeps its leader
|
||||
lock in etcd, and etcd decides only with a majority — so three members, not
|
||||
two: two machines that lose sight of each other must not both decide they are
|
||||
the primary. The third member can be any machine that is up; it holds a vote,
|
||||
not data.
|
||||
|
||||
On the reference cluster:
|
||||
|
||||
| machine | etcd | PostgreSQL |
|
||||
|---|---|---|
|
||||
| llmgw-ctl (control plane) | member | — |
|
||||
| llmgw-pg01 | member | Patroni |
|
||||
| llmgw-ctl02 (second control plane) | member | Patroni |
|
||||
|
||||
`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
|
||||
`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
|
||||
(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
|
||||
Start Patroni on the first node, which initialises the cluster, then on the
|
||||
second, which clones it. Create the `llmgw` role and database on the leader,
|
||||
and move an existing store into it with `pg_dump --no-owner` and
|
||||
`pg_restore --no-owner`.
|
||||
|
||||
**Give llmgw every member, and ask for the writable one:**
|
||||
|
||||
```
|
||||
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
|
||||
```
|
||||
|
||||
The driver connects to whichever member accepts writes, and reconnects to the
|
||||
new one after a failover. No floating address or HAProxy is needed for that.
|
||||
`llmgw server backup` reads the same address and dumps from a standby when
|
||||
there is one.
|
||||
|
||||
Measured on the reference cluster, with the gateway serving throughout:
|
||||
|
||||
- **Planned switchover** (`patronictl switchover`): the control plane logged
|
||||
one failed lease renewal, reconnected to the new primary, and kept its
|
||||
leadership; nothing else noticed.
|
||||
- **The primary's machine switched off**: the standby was promoted 24 seconds
|
||||
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
|
||||
out without losing leadership or a replica. Started again, the old primary
|
||||
rejoined as a standby by itself.
|
||||
|
||||
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
||||
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
||||
on etcd and between PostgreSQL members, and more than one standby.
|
||||
Reference in new issue
Block a user