Merge ha: replicated store, second control plane, join codes, setup and console screens
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
36cdbce125
67 files changed
+8574
-490
No files matched your search
@@ -85,3 +85,63 @@ auth:
|
||||
|
||||
See [docs/install.md](../../docs/install.md) for moving an existing
|
||||
installation onto PostgreSQL without reissuing every key.
|
||||
|
||||
## The store itself: PostgreSQL under Patroni
|
||||
|
||||
Two control planes on one PostgreSQL have moved the single point of failure,
|
||||
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
|
||||
other, and promotes the standby when the primary is lost. It keeps its leader
|
||||
lock in etcd, and etcd decides only with a majority — so three members, not
|
||||
two: two machines that lose sight of each other must not both decide they are
|
||||
the primary. The third member can be any machine that is up; it holds a vote,
|
||||
not data.
|
||||
|
||||
On the reference cluster:
|
||||
|
||||
| machine | etcd | PostgreSQL |
|
||||
|---|---|---|
|
||||
| llmgw-ctl (control plane) | member | — |
|
||||
| llmgw-pg01 | member | Patroni |
|
||||
| llmgw-ctl02 (second control plane) | member | Patroni |
|
||||
|
||||
`llmgw server init --patroni user@host --patroni user@host` sets all of it up
|
||||
over ssh and moves the store in — see [install.md](../../docs/install.md).
|
||||
It looks before it acts at every step, so it is also how an existing cluster
|
||||
is checked: run against one that is up, it changes nothing. What it writes
|
||||
is in `etcd.default.example` (one member's `/etc/default/etcd`) and
|
||||
`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
|
||||
doing it by hand; passwords are generated into `/root/llmgw-pg` on each
|
||||
database machine.
|
||||
|
||||
**Give llmgw every member, and ask for the writable one:**
|
||||
|
||||
```
|
||||
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
|
||||
```
|
||||
|
||||
The driver connects to whichever member accepts writes, and reconnects to the
|
||||
new one after a failover. No floating address or HAProxy is needed for that.
|
||||
`llmgw server backup` reads the same address and dumps from a standby when
|
||||
there is one.
|
||||
|
||||
Measured on the reference cluster, with the gateway serving throughout:
|
||||
|
||||
- **Planned switchover** (`patronictl switchover`): the control plane logged
|
||||
one failed lease renewal, reconnected to the new primary, and kept its
|
||||
leadership; nothing else noticed.
|
||||
- **The primary's machine switched off**: the standby was promoted 24 seconds
|
||||
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
|
||||
out without losing leadership or a replica. Started again, the old primary
|
||||
rejoined as a standby by itself.
|
||||
|
||||
**Watching it.** For a store of several hosts the control plane reads the
|
||||
cluster from each member's Patroni (port 8008) and publishes
|
||||
`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
|
||||
and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
|
||||
lists is a member in state `missing`. `llmgw ctl status` prints it as one
|
||||
line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
|
||||
not streaming or missing, and a standby too far behind to be promoted.
|
||||
|
||||
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
||||
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
||||
on etcd and between PostgreSQL members, and more than one standby.
|
||||
@@ -0,0 +1,10 @@
|
||||
# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
|
||||
ETCD_NAME=llmgw-pg01
|
||||
ETCD_DATA_DIR=/var/lib/etcd/default
|
||||
ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
|
||||
ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
|
||||
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
|
||||
ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
|
||||
ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
|
||||
ETCD_INITIAL_CLUSTER_STATE=new
|
||||
ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni
|
||||
@@ -0,0 +1,68 @@
|
||||
# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
|
||||
# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
|
||||
scope: llmgw
|
||||
namespace: /service/
|
||||
name: llmgw-pg01
|
||||
|
||||
restapi:
|
||||
listen: 10.0.0.149:8008
|
||||
connect_address: 10.0.0.149:8008
|
||||
|
||||
etcd3:
|
||||
hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
|
||||
|
||||
bootstrap:
|
||||
dcs:
|
||||
ttl: 30
|
||||
loop_wait: 10
|
||||
retry_timeout: 10
|
||||
maximum_lag_on_failover: 1048576
|
||||
postgresql:
|
||||
use_pg_rewind: true
|
||||
use_slots: true
|
||||
parameters:
|
||||
wal_level: replica
|
||||
hot_standby: "on"
|
||||
max_wal_senders: 10
|
||||
max_replication_slots: 10
|
||||
wal_keep_size: 1GB
|
||||
wal_log_hints: "on"
|
||||
password_encryption: scram-sha-256
|
||||
initdb:
|
||||
- encoding: UTF8
|
||||
- data-checksums
|
||||
|
||||
postgresql:
|
||||
listen: 127.0.0.1,10.0.0.149:5432
|
||||
connect_address: 10.0.0.149:5432
|
||||
data_dir: /var/lib/postgresql/16/llmgw
|
||||
bin_dir: /usr/lib/postgresql/16/bin
|
||||
pgpass: /var/lib/postgresql/.pgpass_patroni
|
||||
authentication:
|
||||
superuser:
|
||||
username: postgres
|
||||
password: "<from /root/llmgw-pg>"
|
||||
replication:
|
||||
username: replicator
|
||||
password: "<from /root/llmgw-pg>"
|
||||
rewind:
|
||||
username: postgres
|
||||
password: "<from /root/llmgw-pg>"
|
||||
parameters:
|
||||
unix_socket_directories: /var/run/postgresql
|
||||
pg_hba:
|
||||
- local all all peer
|
||||
- host all all 127.0.0.1/32 scram-sha-256
|
||||
- host replication replicator 10.0.0.149/32 scram-sha-256
|
||||
- host replication replicator 10.0.0.150/32 scram-sha-256
|
||||
- host all postgres 10.0.0.149/32 scram-sha-256
|
||||
- host all postgres 10.0.0.150/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.140/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.149/32 scram-sha-256
|
||||
- host llmgw llmgw 10.0.0.150/32 scram-sha-256
|
||||
|
||||
tags:
|
||||
nofailover: false
|
||||
noloadbalance: false
|
||||
clonefrom: false
|
||||
nosync: false
|
||||
@@ -430,3 +430,50 @@ groups:
|
||||
What their people feel. Compare with SlowFirstToken per model: if
|
||||
only this tenant is slow, it is the models it uses or its queue
|
||||
band, not the cluster.
|
||||
# The store, when it is a replicated cluster under Patroni (llmgw server
|
||||
# init --patroni). Published only for a store of several hosts.
|
||||
- name: llmgw-store
|
||||
rules:
|
||||
- alert: StoreHasNoLeader
|
||||
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
|
||||
for: 1m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "The store's cluster has no primary"
|
||||
description: >-
|
||||
Every write fails — keys, usage, budgets, leadership. Patroni
|
||||
promotes a standby within its 30-second TTL; a minute without one
|
||||
means it cannot: etcd has lost its majority, or no standby is fit.
|
||||
`patronictl -c /etc/patroni/config.yml list` on a member says which.
|
||||
|
||||
- alert: StorePatroniUnreachable
|
||||
expr: llmgw_store_patroni_up == 0
|
||||
for: 2m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "No member of the store's cluster answers on its Patroni port"
|
||||
description: >-
|
||||
The control plane reads the cluster's state from port 8008 on each
|
||||
member. The database may still be serving; nothing here can tell.
|
||||
|
||||
- alert: StoreReplicaNotStreaming
|
||||
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
|
||||
description: >-
|
||||
A standby that is not streaming is not a standby: if the primary
|
||||
is lost now, there is nothing to promote. "missing" is a host of
|
||||
the store address the cluster no longer lists — usually a machine
|
||||
that is down.
|
||||
|
||||
- alert: StoreReplicaBehind
|
||||
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
|
||||
description: >-
|
||||
Patroni will not promote a standby more than 1 MB behind, so a
|
||||
failover now would find none fit. -1 is a lag Patroni cannot read.
|
||||
Reference in new issue
Block a user