Set up a replicated store with server init --patroni, and watch it
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the given machines over ssh, with this one as the third etcd vote, makes the role and database, and moves the store in use into the cluster. Each step checks first, so it changes nothing on a cluster that is up. For a store of several hosts the control plane publishes each member's role, state and lag from Patroni, a stopped member as missing, prints it in ctl status, and alerts on it. Making an administrator on PostgreSQL sent a bool to an integer column; it sends an integer now. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7a562ac1e7
commit
975c93994f
22 files changed
+1495
-43
No files matched your search
+16
-7
@@ -104,13 +104,14 @@ On the reference cluster:
|
||||
| llmgw-pg01 | member | Patroni |
|
||||
| llmgw-ctl02 (second control plane) | member | Patroni |
|
||||
|
||||
`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
|
||||
`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
|
||||
(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
|
||||
Start Patroni on the first node, which initialises the cluster, then on the
|
||||
second, which clones it. Create the `llmgw` role and database on the leader,
|
||||
and move an existing store into it with `pg_dump --no-owner` and
|
||||
`pg_restore --no-owner`.
|
||||
`llmgw server init --patroni user@host --patroni user@host` sets all of it up
|
||||
over ssh and moves the store in — see [install.md](../../docs/install.md).
|
||||
It looks before it acts at every step, so it is also how an existing cluster
|
||||
is checked: run against one that is up, it changes nothing. What it writes
|
||||
is in `etcd.default.example` (one member's `/etc/default/etcd`) and
|
||||
`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
|
||||
doing it by hand; passwords are generated into `/root/llmgw-pg` on each
|
||||
database machine.
|
||||
|
||||
**Give llmgw every member, and ask for the writable one:**
|
||||
|
||||
@@ -133,6 +134,14 @@ Measured on the reference cluster, with the gateway serving throughout:
|
||||
out without losing leadership or a replica. Started again, the old primary
|
||||
rejoined as a standby by itself.
|
||||
|
||||
**Watching it.** For a store of several hosts the control plane reads the
|
||||
cluster from each member's Patroni (port 8008) and publishes
|
||||
`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
|
||||
and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
|
||||
lists is a member in state `missing`. `llmgw ctl status` prints it as one
|
||||
line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
|
||||
not streaming or missing, and a standby too far behind to be promoted.
|
||||
|
||||
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
||||
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
||||
on etcd and between PostgreSQL members, and more than one standby.
|
||||
@@ -350,3 +350,51 @@ groups:
|
||||
Working as designed when the cluster is full: the lowest band is
|
||||
held first. Worth knowing about because the callers being refused
|
||||
do not know why.
|
||||
|
||||
# The store, when it is a replicated cluster under Patroni (llmgw server
|
||||
# init --patroni). Published only for a store of several hosts.
|
||||
- name: llmgw-store
|
||||
rules:
|
||||
- alert: StoreHasNoLeader
|
||||
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "The store's cluster has no primary"
|
||||
description: >-
|
||||
Every write fails — keys, usage, budgets, leadership. Patroni
|
||||
promotes a standby within its 30-second TTL; a minute without one
|
||||
means it cannot: etcd has lost its majority, or no standby is fit.
|
||||
`patronictl -c /etc/patroni/config.yml list` on a member says which.
|
||||
|
||||
- alert: StorePatroniUnreachable
|
||||
expr: llmgw_store_patroni_up == 0
|
||||
for: 2m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "No member of the store's cluster answers on its Patroni port"
|
||||
description: >-
|
||||
The control plane reads the cluster's state from port 8008 on each
|
||||
member. The database may still be serving; nothing here can tell.
|
||||
|
||||
- alert: StoreReplicaNotStreaming
|
||||
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
|
||||
description: >-
|
||||
A standby that is not streaming is not a standby: if the primary
|
||||
is lost now, there is nothing to promote. "missing" is a host of
|
||||
the store address the cluster no longer lists — usually a machine
|
||||
that is down.
|
||||
|
||||
- alert: StoreReplicaBehind
|
||||
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
|
||||
description: >-
|
||||
Patroni will not promote a standby more than 1 MB behind, so a
|
||||
failover now would find none fit. -1 is a lag Patroni cannot read.
|
||||
Reference in new issue
Block a user