Ilker ManapandClaude Opus 5.5
975c93994f
init --patroni user@host ... installs PostgreSQL, Patroni and etcd on the given machines over ssh, with this one as the third etcd vote, makes the role and database, and moves the store in use into the cluster. Each step checks first, so it changes nothing on a cluster that is up. For a store of several hosts the control plane publishes each member's role, state and lag from Patroni, a stopped member as missing, prints it in ctl status, and alerts on it. Making an administrator on PostgreSQL sent a bool to an integer column; it sends an integer now. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
148 lines
6.4 KiB
Markdown
148 lines
6.4 KiB
Markdown
# Two control planes
|
|
|
|
llmgw handles the hard half of this itself: only one control plane places work,
|
|
and agents find whichever one is leading. What it does not do is move client
|
|
traffic, because that happens in front of both and llmgw is not in front of
|
|
anything.
|
|
|
|
This directory is the missing half. It is a pattern rather than a product:
|
|
substitute your own load balancer if you have one.
|
|
|
|
## What llmgw does on its own
|
|
|
|
**One leader.** Leadership is a lease in the shared store. A second control
|
|
plane starts, sees the lease, and stands by — it places nothing, evicts
|
|
nothing and drains nothing. Start one by accident and your cluster is
|
|
unharmed.
|
|
|
|
**Agents follow.** A standby refuses agent streams, so agents move to the
|
|
leader by themselves. Give each agent both addresses:
|
|
|
|
```yaml
|
|
servers:
|
|
- "https://ctl-1.internal:7443"
|
|
- "https://ctl-2.internal:7443"
|
|
```
|
|
|
|
Measured on a cluster: leadership moved three seconds after the leader was
|
|
stopped, every agent had arrived six seconds later, and every replica was
|
|
adopted rather than restarted — the engines never stopped serving.
|
|
|
|
## What it does not do
|
|
|
|
Client traffic. During that same failover the standby served inference
|
|
correctly on its own port while the public address returned 502, because
|
|
nothing in front of the two had moved.
|
|
|
|
So something has to. The options, in the order most people should consider
|
|
them:
|
|
|
|
**A load balancer you already run.** Health-check `/ready` on each control
|
|
plane and send traffic to whichever answers. This is the right answer if you
|
|
have one, and it needs nothing else from this directory.
|
|
|
|
**A floating address.** `keepalived.conf.example` here moves one IP between
|
|
the two machines. Fewer moving parts than a load balancer and no extra hop,
|
|
at the cost of one machine carrying all the traffic.
|
|
|
|
**DNS.** Works, eventually. A record with a short TTL is a failover measured
|
|
in whatever your resolvers actually honour rather than what you set, which on
|
|
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
|
one an agent is driving.
|
|
|
|
## Check `/ready`, not `/health`
|
|
|
|
This is the one thing to get right, and the obvious choice is the wrong one.
|
|
|
|
`/health` says the process is up. It says nothing about whether it can serve,
|
|
and a standby is exactly that case: alive, answering, and holding no endpoints
|
|
at all — its reconciler placed nothing and it has no agent streams to adopt
|
|
from. A load balancer checking `/health` would send it half the traffic, and
|
|
half the traffic would be a 503.
|
|
|
|
`/ready` says this process has somewhere to send a request. It is 503 on a
|
|
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
|
something to serve — which is the question a load balancer is actually asking.
|
|
Both are open, because a load balancer has no key.
|
|
|
|
During a failover neither passes for about the length of the lease handover:
|
|
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
|
is not. That gap is real and there is no way to have it both ways — traffic
|
|
cannot go to a control plane that has nothing to route to.
|
|
|
|
Not the admin port, for a different reason: it serves pprof, and pprof can
|
|
dump the process heap, where API keys sit in plaintext.
|
|
|
|
## Both control planes need the same store
|
|
|
|
Leadership lives in it, and so do keys, budgets and conversations. Two control
|
|
planes on two SQLite files are two clusters that happen to share a name.
|
|
|
|
```yaml
|
|
auth:
|
|
store: "@/etc/llmgw/secrets/store-dsn"
|
|
```
|
|
|
|
See [docs/install.md](../../docs/install.md) for moving an existing
|
|
installation onto PostgreSQL without reissuing every key.
|
|
|
|
## The store itself: PostgreSQL under Patroni
|
|
|
|
Two control planes on one PostgreSQL have moved the single point of failure,
|
|
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
|
|
other, and promotes the standby when the primary is lost. It keeps its leader
|
|
lock in etcd, and etcd decides only with a majority — so three members, not
|
|
two: two machines that lose sight of each other must not both decide they are
|
|
the primary. The third member can be any machine that is up; it holds a vote,
|
|
not data.
|
|
|
|
On the reference cluster:
|
|
|
|
| machine | etcd | PostgreSQL |
|
|
|---|---|---|
|
|
| llmgw-ctl (control plane) | member | — |
|
|
| llmgw-pg01 | member | Patroni |
|
|
| llmgw-ctl02 (second control plane) | member | Patroni |
|
|
|
|
`llmgw server init --patroni user@host --patroni user@host` sets all of it up
|
|
over ssh and moves the store in — see [install.md](../../docs/install.md).
|
|
It looks before it acts at every step, so it is also how an existing cluster
|
|
is checked: run against one that is up, it changes nothing. What it writes
|
|
is in `etcd.default.example` (one member's `/etc/default/etcd`) and
|
|
`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
|
|
doing it by hand; passwords are generated into `/root/llmgw-pg` on each
|
|
database machine.
|
|
|
|
**Give llmgw every member, and ask for the writable one:**
|
|
|
|
```
|
|
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
|
|
```
|
|
|
|
The driver connects to whichever member accepts writes, and reconnects to the
|
|
new one after a failover. No floating address or HAProxy is needed for that.
|
|
`llmgw server backup` reads the same address and dumps from a standby when
|
|
there is one.
|
|
|
|
Measured on the reference cluster, with the gateway serving throughout:
|
|
|
|
- **Planned switchover** (`patronictl switchover`): the control plane logged
|
|
one failed lease renewal, reconnected to the new primary, and kept its
|
|
leadership; nothing else noticed.
|
|
- **The primary's machine switched off**: the standby was promoted 24 seconds
|
|
later — the 30-second TTL in `patroni.yml` — and the control plane rode it
|
|
out without losing leadership or a replica. Started again, the old primary
|
|
rejoined as a standby by itself.
|
|
|
|
**Watching it.** For a store of several hosts the control plane reads the
|
|
cluster from each member's Patroni (port 8008) and publishes
|
|
`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
|
|
and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
|
|
lists is a member in state `missing`. `llmgw ctl status` prints it as one
|
|
line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
|
|
not streaming or missing, and a standby too far behind to be promoted.
|
|
|
|
Not done here, and worth doing in production: a watchdog (`softdog`) so a
|
|
primary that has lost the lock but not died cannot keep accepting writes, TLS
|
|
on etcd and between PostgreSQL members, and more than one standby.
|