Merge ha: replicated store, second control plane, join codes, setup and console screens
Release / build (push) Successful in 4m26s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 5m14s

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-03 22:59:49 +02:00
commit 36cdbce125
67 files changed
+8574 -490

No files matched your search

+60
View File
@@ -85,3 +85,63 @@ auth:
See [docs/install.md](../../docs/install.md) for moving an existing
installation onto PostgreSQL without reissuing every key.
## The store itself: PostgreSQL under Patroni
Two control planes on one PostgreSQL have moved the single point of failure,
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
other, and promotes the standby when the primary is lost. It keeps its leader
lock in etcd, and etcd decides only with a majority — so three members, not
two: two machines that lose sight of each other must not both decide they are
the primary. The third member can be any machine that is up; it holds a vote,
not data.
On the reference cluster:
| machine | etcd | PostgreSQL |
|---|---|---|
| llmgw-ctl (control plane) | member | — |
| llmgw-pg01 | member | Patroni |
| llmgw-ctl02 (second control plane) | member | Patroni |
`llmgw server init --patroni user@host --patroni user@host` sets all of it up
over ssh and moves the store in — see [install.md](../../docs/install.md).
It looks before it acts at every step, so it is also how an existing cluster
is checked: run against one that is up, it changes nothing. What it writes
is in `etcd.default.example` (one member's `/etc/default/etcd`) and
`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
doing it by hand; passwords are generated into `/root/llmgw-pg` on each
database machine.
**Give llmgw every member, and ask for the writable one:**
```
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
```
The driver connects to whichever member accepts writes, and reconnects to the
new one after a failover. No floating address or HAProxy is needed for that.
`llmgw server backup` reads the same address and dumps from a standby when
there is one.
Measured on the reference cluster, with the gateway serving throughout:
- **Planned switchover** (`patronictl switchover`): the control plane logged
one failed lease renewal, reconnected to the new primary, and kept its
leadership; nothing else noticed.
- **The primary's machine switched off**: the standby was promoted 24 seconds
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
out without losing leadership or a replica. Started again, the old primary
rejoined as a standby by itself.
**Watching it.** For a store of several hosts the control plane reads the
cluster from each member's Patroni (port 8008) and publishes
`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
lists is a member in state `missing`. `llmgw ctl status` prints it as one
line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
not streaming or missing, and a standby too far behind to be promoted.
Not done here, and worth doing in production: a watchdog (`softdog`) so a
primary that has lost the lock but not died cannot keep accepting writes, TLS
on etcd and between PostgreSQL members, and more than one standby.
+10
View File
@@ -0,0 +1,10 @@
# llmgw: one member of the three that hold Patroni's leader lock (140, 149, 150).
ETCD_NAME=llmgw-pg01
ETCD_DATA_DIR=/var/lib/etcd/default
ETCD_LISTEN_PEER_URLS=http://10.0.0.149:2380
ETCD_LISTEN_CLIENT_URLS=http://10.0.0.149:2379,http://127.0.0.1:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=http://10.0.0.149:2380
ETCD_ADVERTISE_CLIENT_URLS=http://10.0.0.149:2379
ETCD_INITIAL_CLUSTER=llmgw-ctl=http://10.0.0.140:2380,llmgw-pg01=http://10.0.0.149:2380,llmgw-ctl02=http://10.0.0.150:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=llmgw-patroni
+68
View File
@@ -0,0 +1,68 @@
# llmgw: the control planes' store, kept by Patroni on llmgw-pg01 and
# llmgw-ctl02, with the leader lock in etcd on 140, 149 and 150.
scope: llmgw
namespace: /service/
name: llmgw-pg01
restapi:
listen: 10.0.0.149:8008
connect_address: 10.0.0.149:8008
etcd3:
hosts: 10.0.0.140:2379,10.0.0.149:2379,10.0.0.150:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
postgresql:
use_pg_rewind: true
use_slots: true
parameters:
wal_level: replica
hot_standby: "on"
max_wal_senders: 10
max_replication_slots: 10
wal_keep_size: 1GB
wal_log_hints: "on"
password_encryption: scram-sha-256
initdb:
- encoding: UTF8
- data-checksums
postgresql:
listen: 127.0.0.1,10.0.0.149:5432
connect_address: 10.0.0.149:5432
data_dir: /var/lib/postgresql/16/llmgw
bin_dir: /usr/lib/postgresql/16/bin
pgpass: /var/lib/postgresql/.pgpass_patroni
authentication:
superuser:
username: postgres
password: "<from /root/llmgw-pg>"
replication:
username: replicator
password: "<from /root/llmgw-pg>"
rewind:
username: postgres
password: "<from /root/llmgw-pg>"
parameters:
unix_socket_directories: /var/run/postgresql
pg_hba:
- local all all peer
- host all all 127.0.0.1/32 scram-sha-256
- host replication replicator 10.0.0.149/32 scram-sha-256
- host replication replicator 10.0.0.150/32 scram-sha-256
- host all postgres 10.0.0.149/32 scram-sha-256
- host all postgres 10.0.0.150/32 scram-sha-256
- host llmgw llmgw 10.0.0.140/32 scram-sha-256
- host llmgw llmgw 10.0.0.149/32 scram-sha-256
- host llmgw llmgw 10.0.0.150/32 scram-sha-256
tags:
nofailover: false
noloadbalance: false
clonefrom: false
nosync: false
+47
View File
@@ -430,3 +430,50 @@ groups:
What their people feel. Compare with SlowFirstToken per model: if
only this tenant is slow, it is the models it uses or its queue
band, not the cluster.
# The store, when it is a replicated cluster under Patroni (llmgw server
# init --patroni). Published only for a store of several hosts.
- name: llmgw-store
rules:
- alert: StoreHasNoLeader
expr: llmgw_store_patroni_up == 1 and on() count(llmgw_store_member{role="leader"}) == 0
for: 1m
labels: { severity: critical }
annotations:
summary: "The store's cluster has no primary"
description: >-
Every write fails — keys, usage, budgets, leadership. Patroni
promotes a standby within its 30-second TTL; a minute without one
means it cannot: etcd has lost its majority, or no standby is fit.
`patronictl -c /etc/patroni/config.yml list` on a member says which.
- alert: StorePatroniUnreachable
expr: llmgw_store_patroni_up == 0
for: 2m
labels: { severity: warning }
annotations:
summary: "No member of the store's cluster answers on its Patroni port"
description: >-
The control plane reads the cluster's state from port 8008 on each
member. The database may still be serving; nothing here can tell.
- alert: StoreReplicaNotStreaming
expr: llmgw_store_member{role!="leader", state!="streaming"} == 1
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.member }} is {{ $labels.state }}, not streaming from the primary"
description: >-
A standby that is not streaming is not a standby: if the primary
is lost now, there is nothing to promote. "missing" is a host of
the store address the cluster no longer lists — usually a machine
that is down.
- alert: StoreReplicaBehind
expr: llmgw_store_replica_lag_bytes > 64 * 1024 * 1024 or llmgw_store_replica_lag_bytes == -1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.member }} is {{ $value | humanize1024 }}B behind the primary"
description: >-
Patroni will not promote a standby more than 1 MB behind, so a
failover now would find none fit. -1 is a lag Patroni cannot read.