Ilker ManapandClaude Opus 5
2bf7de1a43
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.5 KiB
Two control planes
llmgw handles the hard half of this itself: only one control plane places work, and agents find whichever one is leading. What it does not do is move client traffic, because that happens in front of both and llmgw is not in front of anything.
This directory is the missing half. It is a pattern rather than a product: substitute your own load balancer if you have one.
What llmgw does on its own
One leader. Leadership is a lease in the shared store. A second control plane starts, sees the lease, and stands by — it places nothing, evicts nothing and drains nothing. Start one by accident and your cluster is unharmed.
Agents follow. A standby refuses agent streams, so agents move to the leader by themselves. Give each agent both addresses:
servers:
- "https://ctl-1.internal:7443"
- "https://ctl-2.internal:7443"
Measured on a cluster: leadership moved three seconds after the leader was stopped, every agent had arrived six seconds later, and every replica was adopted rather than restarted — the engines never stopped serving.
What it does not do
Client traffic. During that same failover the standby served inference correctly on its own port while the public address returned 502, because nothing in front of the two had moved.
So something has to. The options, in the order most people should consider them:
A load balancer you already run. Health-check /ready on each control
plane and send traffic to whichever answers. This is the right answer if you
have one, and it needs nothing else from this directory.
A floating address. keepalived.conf.example here moves one IP between
the two machines. Fewer moving parts than a load balancer and no extra hop,
at the cost of one machine carrying all the traffic.
DNS. Works, eventually. A record with a short TTL is a failover measured in whatever your resolvers actually honour rather than what you set, which on a bad day is minutes. Fine for a cluster people use interactively, wrong for one an agent is driving.
Check /ready, not /health
This is the one thing to get right, and the obvious choice is the wrong one.
/health says the process is up. It says nothing about whether it can serve,
and a standby is exactly that case: alive, answering, and holding no endpoints
at all — its reconciler placed nothing and it has no agent streams to adopt
from. A load balancer checking /health would send it half the traffic, and
half the traffic would be a 503.
/ready says this process has somewhere to send a request. It is 503 on a
standby, 503 on a leader whose replicas have all failed, and 200 when there is
something to serve — which is the question a load balancer is actually asking.
Both are open, because a load balancer has no key.
During a failover neither passes for about the length of the lease handover: three seconds when the leader is stopped cleanly, up to the lease TTL when it is not. That gap is real and there is no way to have it both ways — traffic cannot go to a control plane that has nothing to route to.
Not the admin port, for a different reason: it serves pprof, and pprof can dump the process heap, where API keys sit in plaintext.
Both control planes need the same store
Leadership lives in it, and so do keys, budgets and conversations. Two control planes on two SQLite files are two clusters that happen to share a name.
auth:
store: "@/etc/llmgw/secrets/store-dsn"
See docs/install.md for moving an existing installation onto PostgreSQL without reissuing every key.