llmgw/deploy/ha/README.md

More than one control plane

llmgw handles the hard half of this itself: only one control plane places work, and agents find whichever one is leading. What it does not do is move client traffic, because that happens in front of both and llmgw is not in front of anything.

This directory is the missing half. It is a pattern rather than a product: substitute your own load balancer if you have one.

What llmgw does on its own

One leader. Leadership is a lease in the shared store. A second control plane starts, sees the lease, and stands by — it places nothing, evicts nothing and drains nothing. Start one by accident and your cluster is unharmed.

Agents follow. A standby refuses agent streams, so agents move to the leader by themselves. Give each agent both addresses:

servers:
  - "https://ctl-1.internal:7443"
  - "https://ctl-2.internal:7443"

Measured on a cluster: leadership moved three seconds after the leader was stopped, every agent had arrived six seconds later, and every replica was adopted rather than restarted — the engines never stopped serving.

What it does not do

Client traffic. During that same failover the standby served inference correctly on its own port while the public address returned 502, because nothing in front of the two had moved.

So something has to. The options, in the order most people should consider them:

A load balancer you already run. Health-check /leader on each control plane and send traffic to whichever answers. This is the right answer if you have one, and it needs nothing else from this directory.

A floating address. keepalived.conf.example here moves one IP between the two machines. Fewer moving parts than a load balancer and no extra hop, at the cost of one machine carrying all the traffic.

DNS. Works, eventually. A record with a short TTL is a failover measured in whatever your resolvers actually honour rather than what you set, which on a bad day is minutes. Fine for a cluster people use interactively, wrong for one an agent is driving.

Check /leader, not /health

This is the one thing to get right, and the obvious choice is the wrong one.

/health says the process is up. It says nothing about whether it leads, and a standby is exactly that case: alive, answering, and holding no endpoints at all — its reconciler placed nothing and it has no agent streams to adopt from. A standby passes client requests on to the leader, so traffic sent to one still works — until the leader is the machine that was lost, and there is nobody to pass them to.

/leader is 200 on the control plane that leads and 503 on a standby. It is what keepalived follows for the floating address, and what a load balancer should send traffic by: to the one machine that answers 200.

/ready is narrower: 200 only when this process has somewhere to send a request, so 503 on a standby and also on a leader with no model running yet. Checking it takes the portal and the admin pages away with the models — on a new cluster with no node yet, nobody can sign in to add one. Use it to stop sending inference only, if your load balancer can tell the paths apart.

Both are open, because a load balancer has no key.

During a failover nothing passes for about the length of the lease handover: three seconds when the leader is stopped cleanly, up to the lease TTL when it is not. A proxy that waits that long for a healthy machine rather than failing at once (Caddy's lb_try_duration) hides it from clients.

Caddy, in front of three control planes — the end of deploy/proxy/Caddyfile has it with the rest of the settings:

reverse_proxy 10.0.0.157:8080 10.0.0.158:8080 10.0.0.159:8080 {
	lb_policy first
	health_uri /leader
	health_interval 2s
	lb_try_duration 15s
	flush_interval -1
}

With a floating address the address already follows the leader: give Caddy that one address instead.

Not the admin port, for a different reason: it serves pprof, and pprof can dump the process heap, where API keys sit in plaintext.

Both control planes need the same store

Leadership lives in it, and so do keys, budgets and conversations. Two control planes on two SQLite files are two clusters that happen to share a name.

auth:
  store: "@/etc/llmgw/secrets/store-dsn"

See docs/install.md for moving an existing installation onto PostgreSQL without reissuing every key.

The store itself: PostgreSQL under Patroni

Two control planes on one PostgreSQL have moved the single point of failure, not removed it. Patroni keeps PostgreSQL on two machines, one streaming to the other, and promotes the standby when the primary is lost. It keeps its leader lock in etcd, and etcd decides only with a majority — so three members, not two: two machines that lose sight of each other must not both decide they are the primary. The third member can be any machine that is up; it holds a vote, not data.

On the reference cluster:

machine	etcd	PostgreSQL
llmgw-ctl (control plane)	member	—
llmgw-pg01	member	Patroni
llmgw-ctl02 (second control plane)	member	Patroni

llmgw server init --patroni user@host --patroni user@host sets all of it up over ssh and moves the store in — see install.md. It looks before it acts at every step, so it is also how an existing cluster is checked: run against one that is up, it changes nothing. What it writes is in etcd.default.example (one member's /etc/default/etcd) and patroni.yml.example (one node's /etc/patroni/config.yml), for anyone doing it by hand; passwords are generated into /root/llmgw-pg on each database machine.

Give llmgw every member, and ask for the writable one:

postgres://llmgw:…@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=disable&target_session_attrs=read-write

The driver connects to whichever member accepts writes, and reconnects to the new one after a failover. No floating address or HAProxy is needed for that. llmgw server backup reads the same address and dumps from a standby when there is one.

Measured on the reference cluster, with the gateway serving throughout:

• Planned switchover (patronictl switchover): the control plane logged one failed lease renewal, reconnected to the new primary, and kept its leadership; nothing else noticed.
• The primary's machine switched off: the standby was promoted 24 seconds later — the 30-second TTL in patroni.yml — and the control plane rode it out without losing leadership or a replica. Started again, the old primary rejoined as a standby by itself.

Watching it. For a store of several hosts the control plane reads the cluster from each member's Patroni (port 8008) and publishes llmgw_store_member{member,host,role,state}, llmgw_store_replica_lag_bytes and llmgw_store_patroni_up; a host of the address that the cluster no longer lists is a member in state missing. llmgw ctl status prints it as one line, and deploy/monitoring/llmgw-rules.yml alerts on no leader, a member not streaming or missing, and a standby too far behind to be promoted.

Not done here, and worth doing in production: a watchdog (softdog) so a primary that has lost the lock but not died cannot keep accepting writes, TLS on etcd and between PostgreSQL members, and more than one standby.