Keep the store under Patroni, and back up from a store of several hosts

deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as
running on the reference cluster, and the address that follows the primary.
Backups read such an address as libpq host and port lists and prefer a
standby, rather than resolving the list as one name.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 20:31:19 +02:00
1 parent febcf251e4
commit 7a562ac1e7
6 files changed
+223 -10

No files matched your search

+51
View File
@@ -85,3 +85,54 @@ auth:
See [docs/install.md](../../docs/install.md) for moving an existing
installation onto PostgreSQL without reissuing every key.
## The store itself: PostgreSQL under Patroni
Two control planes on one PostgreSQL have moved the single point of failure,
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
other, and promotes the standby when the primary is lost. It keeps its leader
lock in etcd, and etcd decides only with a majority — so three members, not
two: two machines that lose sight of each other must not both decide they are
the primary. The third member can be any machine that is up; it holds a vote,
not data.
On the reference cluster:
| machine | etcd | PostgreSQL |
|---|---|---|
| llmgw-ctl (control plane) | member | — |
| llmgw-pg01 | member | Patroni |
| llmgw-ctl02 (second control plane) | member | Patroni |
`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
Start Patroni on the first node, which initialises the cluster, then on the
second, which clones it. Create the `llmgw` role and database on the leader,
and move an existing store into it with `pg_dump --no-owner` and
`pg_restore --no-owner`.
**Give llmgw every member, and ask for the writable one:**
```
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
```
The driver connects to whichever member accepts writes, and reconnects to the
new one after a failover. No floating address or HAProxy is needed for that.
`llmgw server backup` reads the same address and dumps from a standby when
there is one.
Measured on the reference cluster, with the gateway serving throughout:
- **Planned switchover** (`patronictl switchover`): the control plane logged
one failed lease renewal, reconnected to the new primary, and kept its
leadership; nothing else noticed.
- **The primary's machine switched off**: the standby was promoted 24 seconds
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
out without losing leadership or a replica. Started again, the old primary
rejoined as a standby by itself.
Not done here, and worth doing in production: a watchdog (`softdog`) so a
primary that has lost the lock but not died cannot keep accepting writes, TLS
on etcd and between PostgreSQL members, and more than one standby.