llmgw/deploy/ha/README.md

# More than one control plane

llmgw handles the hard half of this itself: only one control plane places work,
and agents find whichever one is leading. What it does not do is move client
traffic, because that happens in front of both and llmgw is not in front of
anything.

This directory is the missing half. It is a pattern rather than a product:
substitute your own load balancer if you have one.

## What llmgw does on its own

**One leader.** Leadership is a lease in the shared store. A second control
plane starts, sees the lease, and stands by — it places nothing, evicts
nothing and drains nothing. Start one by accident and your cluster is
unharmed.

**Agents follow.** A standby refuses agent streams, so agents move to the
leader by themselves. Give each agent both addresses:

```yaml
servers:
  - "https://ctl-1.internal:7443"
  - "https://ctl-2.internal:7443"
```

Measured on a cluster: leadership moved three seconds after the leader was
stopped, every agent had arrived six seconds later, and every replica was
adopted rather than restarted — the engines never stopped serving.

## What it does not do

Client traffic. During that same failover the standby served inference
correctly on its own port while the public address returned 502, because
nothing in front of the two had moved.

So something has to. The options, in the order most people should consider
them:

**A load balancer you already run.** Health-check `/leader` on each control
plane and send traffic to whichever answers. This is the right answer if you
have one, and it needs nothing else from this directory.

**A floating address.** `keepalived.conf.example` here moves one IP between
the two machines. Fewer moving parts than a load balancer and no extra hop,
at the cost of one machine carrying all the traffic.

**DNS.** Works, eventually. A record with a short TTL is a failover measured
in whatever your resolvers actually honour rather than what you set, which on
a bad day is minutes. Fine for a cluster people use interactively, wrong for
one an agent is driving.

## Check `/leader`, not `/health`

This is the one thing to get right, and the obvious choice is the wrong one.

`/health` says the process is up. It says nothing about whether it leads,
and a standby is exactly that case: alive, answering, and holding no endpoints
at all — its reconciler placed nothing and it has no agent streams to adopt
from. A standby passes client requests on to the leader, so traffic sent to
one still works — until the leader is the machine that was lost, and there
is nobody to pass them to.

`/leader` is 200 on the control plane that leads and 503 on a standby. It is
what keepalived follows for the floating address, and what a load balancer
should send traffic by: to the one machine that answers 200.

`/ready` is narrower: 200 only when this process has somewhere to send a
request, so 503 on a standby and also on a leader with no model running yet.
Checking it takes the portal and the admin pages away with the models — on a
new cluster with no node yet, nobody can sign in to add one. Use it to stop
sending inference only, if your load balancer can tell the paths apart.

Both are open, because a load balancer has no key.

During a failover nothing passes for about the length of the lease handover:
three seconds when the leader is stopped cleanly, up to the lease TTL when it
is not. A proxy that waits that long for a healthy machine rather than
failing at once (Caddy's `lb_try_duration`) hides it from clients.

**Caddy**, in front of three control planes — the end of
[deploy/proxy/Caddyfile](../proxy/Caddyfile) has it with the rest of the
settings:

```
reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
	lb_policy first
	health_uri /leader
	health_interval 2s
	lb_try_duration 15s
	flush_interval -1
}
```

With a floating address the address already follows the leader: give Caddy
that one address instead.

Not the admin port, for a different reason: it serves pprof, and pprof can
dump the process heap, where API keys sit in plaintext.

## Both control planes need the same store

Leadership lives in it, and so do keys, budgets and conversations. Two control
planes on two SQLite files are two clusters that happen to share a name.

```yaml
auth:
  store: "@/etc/llmgw/secrets/store-dsn"
```

See [docs/install.md](../../docs/install.md) for moving an existing
installation onto PostgreSQL without reissuing every key.

## The store itself: PostgreSQL under Patroni

Two control planes on one PostgreSQL have moved the single point of failure,
not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the
other, and promotes the standby when the primary is lost. It keeps its leader
lock in etcd, and etcd decides only with a majority — so three members, not
two: two machines that lose sight of each other must not both decide they are
the primary. The third member can be any machine that is up; it holds a vote,
not data.

On the reference cluster:

| machine | etcd | PostgreSQL |
|---|---|---|
| llmgw-ctl (control plane) | member | — |
| llmgw-pg01 | member | Patroni |
| llmgw-ctl02 (second control plane) | member | Patroni |

`llmgw server init --patroni user@host --patroni user@host` sets all of it up
over ssh and moves the store in — see [install.md](../../docs/install.md).
It looks before it acts at every step, so it is also how an existing cluster
is checked: run against one that is up, it changes nothing. What it writes
is in `etcd.default.example` (one member's `/etc/default/etcd`) and
`patroni.yml.example` (one node's `/etc/patroni/config.yml`), for anyone
doing it by hand; passwords are generated into `/root/llmgw-pg` on each
database machine.

**Give llmgw every member, and ask for the writable one:**

```
postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write
```

The driver connects to whichever member accepts writes, and reconnects to the
new one after a failover. No floating address or HAProxy is needed for that.
`llmgw server backup` reads the same address and dumps from a standby when
there is one.

Measured on the reference cluster, with the gateway serving throughout:

- **Planned switchover** (`patronictl switchover`): the control plane logged
  one failed lease renewal, reconnected to the new primary, and kept its
  leadership; nothing else noticed.
- **The primary's machine switched off**: the standby was promoted 24 seconds
  later — the 30-second TTL in `patroni.yml` — and the control plane rode it
  out without losing leadership or a replica. Started again, the old primary
  rejoined as a standby by itself.

**Watching it.** For a store of several hosts the control plane reads the
cluster from each member's Patroni (port 8008) and publishes
`llmgw_store_member{member,host,role,state}`, `llmgw_store_replica_lag_bytes`
and `llmgw_store_patroni_up`; a host of the address that the cluster no longer
lists is a member in state `missing`. `llmgw ctl status` prints it as one
line, and `deploy/monitoring/llmgw-rules.yml` alerts on no leader, a member
not streaming or missing, and a standby too far behind to be promoted.

Not done here, and worth doing in production: a watchdog (`softdog`) so a
primary that has lost the lock but not died cannot keep accepting writes, TLS
on etcd and between PostgreSQL members, and more than one standby.