deploy/ha documents PostgreSQL under Patroni with a three-member etcd, as running on the reference cluster, and the address that follows the primary. Backups read such an address as libpq host and port lists and prefer a standby, rather than resolving the list as one name. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
139 lines
5.8 KiB
Markdown
139 lines
5.8 KiB
Markdown
# Two control planes
|
|
|
|
llmgw handles the hard half of this itself: only one control plane places work,
|
|
and agents find whichever one is leading. What it does not do is move client
|
|
traffic, because that happens in front of both and llmgw is not in front of
|
|
anything.
|
|
|
|
This directory is the missing half. It is a pattern rather than a product:
|
|
substitute your own load balancer if you have one.
|
|
|
|
## What llmgw does on its own
|
|
|
|
**One leader.** Leadership is a lease in the shared store. A second control
|
|
plane starts, sees the lease, and stands by — it places nothing, evicts
|
|
nothing and drains nothing. Start one by accident and your cluster is
|
|
unharmed.
|
|
|
|
**Agents follow.** A standby refuses agent streams, so agents move to the
|
|
leader by themselves. Give each agent both addresses:
|
|
|
|
```yaml
|
|
servers:
|
|
- "https://ctl-1.internal:7443"
|
|
- "https://ctl-2.internal:7443"
|
|
```
|
|
|
|
Measured on a cluster: leadership moved three seconds after the leader was
|
|
stopped, every agent had arrived six seconds later, and every replica was
|
|
adopted rather than restarted — the engines never stopped serving.
|
|
|
|
## What it does not do
|
|
|
|
Client traffic. During that same failover the standby served inference
|
|
correctly on its own port while the public address returned 502, because
|
|
nothing in front of the two had moved.
|
|
|
|
So something has to. The options, in the order most people should consider
|
|
them:
|
|
|
|
**A load balancer you already run.** Health-check `/ready` on each control
|
|
plane and send traffic to whichever answers. This is the right answer if you
|
|
have one, and it needs nothing else from this directory.
|
|
|
|
**A floating address.** `keepalived.conf.example` here moves one IP between
|
|
the two machines. Fewer moving parts than a load balancer and no extra hop,
|
|
at the cost of one machine carrying all the traffic.
|
|
|
|
**DNS.** Works, eventually. A record with a short TTL is a failover measured
|
|
in whatever your resolvers actually honour rather than what you set, which on
|
|
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
|
one an agent is driving.
|
|
|
|
## Check `/ready`, not `/health`
|
|
|
|
This is the one thing to get right, and the obvious choice is the wrong one.
|
|
|
|
`/health` says the process is up. It says nothing about whether it can serve,
|
|
and a standby is exactly that case: alive, answering, and holding no endpoints
|
|
at all — its reconciler placed nothing and it has no agent streams to adopt
|
|
from. A load balancer checking `/health` would send it half the traffic, and
|
|
half the traffic would be a 503.
|
|
|
|
`/ready` says this process has somewhere to send a request. It is 503 on a
|
|
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
|
something to serve — which is the question a load balancer is actually asking.
|
|
Both are open, because a load balancer has no key.
|
|
|
|
During a failover neither passes for about the length of the lease handover:
|
|
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
|
is not. That gap is real and there is no way to have it both ways — traffic
|
|
cannot go to a control plane that has nothing to route to.
|
|
|
|
Not the admin port, for a different reason: it serves pprof, and pprof can
|
|
dump the process heap, where API keys sit in plaintext.
|
|
|
|
## Both control planes need the same store
|
|
|
|
Leadership lives in it, and so do keys, budgets and conversations. Two control
|
|
planes on two SQLite files are two clusters that happen to share a name.
|
|
|
|
```yaml
|
|
auth:
|
|
store: "@/etc/llmgw/secrets/store-dsn"
|
|
```
|
|
|
|
See [docs/install.md](../../docs/install.md) for moving an existing
|
|
installation onto PostgreSQL without reissuing every key.
|
|
|
|
## The store itself: PostgreSQL under Patroni
|
|
|
|
Two control planes on one PostgreSQL have moved the single point of failure,
|
|
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
|
|
other, and promotes the standby when the primary is lost. It keeps its leader
|
|
lock in etcd, and etcd decides only with a majority — so three members, not
|
|
two: two machines that lose sight of each other must not both decide they are
|
|
the primary. The third member can be any machine that is up; it holds a vote,
|
|
not data.
|
|
|
|
On the reference cluster:
|
|
|
|
| machine | etcd | PostgreSQL |
|
|
|---|---|---|
|
|
| llmgw-ctl (control plane) | member | — |
|
|
| llmgw-pg01 | member | Patroni |
|
|
| llmgw-ctl02 (second control plane) | member | Patroni |
|
|
|
|
`etcd.default.example` is one member's `/etc/default/etcd` (Ubuntu's
|
|
`etcd-server`), and `patroni.yml.example` one node's `/etc/patroni/config.yml`
|
|
(Ubuntu's `patroni`); the other node differs in `name` and its addresses.
|
|
Start Patroni on the first node, which initialises the cluster, then on the
|
|
second, which clones it. Create the `llmgw` role and database on the leader,
|
|
and move an existing store into it with `pg_dump --no-owner` and
|
|
`pg_restore --no-owner`.
|
|
|
|
**Give llmgw every member, and ask for the writable one:**
|
|
|
|
```
|
|
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
|
|
```
|
|
|
|
The driver connects to whichever member accepts writes, and reconnects to the
|
|
new one after a failover. No floating address or HAProxy is needed for that.
|
|
`llmgw server backup` reads the same address and dumps from a standby when
|
|
there is one.
|
|
|
|
Measured on the reference cluster, with the gateway serving throughout:
|
|
|
|
- **Planned switchover** (`patronictl switchover`): the control plane logged
|
|
one failed lease renewal, reconnected to the new primary, and kept its
|
|
leadership; nothing else noticed.
|
|
- **The primary's machine switched off**: the standby was promoted 24 seconds
|
|
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
|
|
out without losing leadership or a replica. Started again, the old primary
|
|
rejoined as a standby by itself.
|
|
|
|
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
|
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
|
on etcd and between PostgreSQL members, and more than one standby.
|