Set up a replicated store with server init --patroni, and watch it

init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the
given machines over ssh, with this one as the third etcd vote, makes the role
and database, and moves the store in use into the cluster. Each step checks
first, so it changes nothing on a cluster that is up. For a store of several
hosts the control plane publishes each member's role, state and lag from
Patroni, a stopped member as missing, prints it in ctl status, and alerts on
it. Making an administrator on PostgreSQL sent a bool to an integer column;
it sends an integer now.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 20:57:41 +02:00
1 parent 7a562ac1e7
commit 975c93994f
22 files changed
+1495 -43

No files matched your search

+48
View File
@@ -350,3 +350,51 @@ groups:
Working as designed when the cluster is full: the lowest band is
held first. Worth knowing about because the callers being refused
do not know why.
# The store, when it is a replicated cluster under Patroni (llmgw server
# init --patroni). Published only for a store of several hosts.
- name: llmgw-store
rules:
- alert: StoreHasNoLeader
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
for: 1m
labels: { severity: critical }
annotations:
summary: "The store's cluster has no primary"
description: >-
Every write fails — keys, usage, budgets, leadership. Patroni
promotes a standby within its 30-second TTL; a minute without one
means it cannot: etcd has lost its majority, or no standby is fit.
`patronictl -c /etc/patroni/config.yml list` on a member says which.
- alert: StorePatroniUnreachable
expr: llmgw_store_patroni_up == 0
for: 2m
labels: { severity: warning }
annotations:
summary: "No member of the store's cluster answers on its Patroni port"
description: >-
The control plane reads the cluster's state from port 8008 on each
member. The database may still be serving; nothing here can tell.
- alert: StoreReplicaNotStreaming
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
description: >-
A standby that is not streaming is not a standby: if the primary
is lost now, there is nothing to promote. "missing" is a host of
the store address the cluster no longer lists — usually a machine
that is down.
- alert: StoreReplicaBehind
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
description: >-
Patroni will not promote a standby more than 1 MB behind, so a
failover now would find none fit. -1 is a lag Patroni cannot read.