Ilker ManapandClaude Opus 5
2bf7de1a43
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
88 lines
3.5 KiB
Markdown
88 lines
3.5 KiB
Markdown
# Two control planes
|
|
|
|
llmgw handles the hard half of this itself: only one control plane places work,
|
|
and agents find whichever one is leading. What it does not do is move client
|
|
traffic, because that happens in front of both and llmgw is not in front of
|
|
anything.
|
|
|
|
This directory is the missing half. It is a pattern rather than a product:
|
|
substitute your own load balancer if you have one.
|
|
|
|
## What llmgw does on its own
|
|
|
|
**One leader.** Leadership is a lease in the shared store. A second control
|
|
plane starts, sees the lease, and stands by — it places nothing, evicts
|
|
nothing and drains nothing. Start one by accident and your cluster is
|
|
unharmed.
|
|
|
|
**Agents follow.** A standby refuses agent streams, so agents move to the
|
|
leader by themselves. Give each agent both addresses:
|
|
|
|
```yaml
|
|
servers:
|
|
- "https://ctl-1.internal:7443"
|
|
- "https://ctl-2.internal:7443"
|
|
```
|
|
|
|
Measured on a cluster: leadership moved three seconds after the leader was
|
|
stopped, every agent had arrived six seconds later, and every replica was
|
|
adopted rather than restarted — the engines never stopped serving.
|
|
|
|
## What it does not do
|
|
|
|
Client traffic. During that same failover the standby served inference
|
|
correctly on its own port while the public address returned 502, because
|
|
nothing in front of the two had moved.
|
|
|
|
So something has to. The options, in the order most people should consider
|
|
them:
|
|
|
|
**A load balancer you already run.** Health-check `/ready` on each control
|
|
plane and send traffic to whichever answers. This is the right answer if you
|
|
have one, and it needs nothing else from this directory.
|
|
|
|
**A floating address.** `keepalived.conf.example` here moves one IP between
|
|
the two machines. Fewer moving parts than a load balancer and no extra hop,
|
|
at the cost of one machine carrying all the traffic.
|
|
|
|
**DNS.** Works, eventually. A record with a short TTL is a failover measured
|
|
in whatever your resolvers actually honour rather than what you set, which on
|
|
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
|
one an agent is driving.
|
|
|
|
## Check `/ready`, not `/health`
|
|
|
|
This is the one thing to get right, and the obvious choice is the wrong one.
|
|
|
|
`/health` says the process is up. It says nothing about whether it can serve,
|
|
and a standby is exactly that case: alive, answering, and holding no endpoints
|
|
at all — its reconciler placed nothing and it has no agent streams to adopt
|
|
from. A load balancer checking `/health` would send it half the traffic, and
|
|
half the traffic would be a 503.
|
|
|
|
`/ready` says this process has somewhere to send a request. It is 503 on a
|
|
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
|
something to serve — which is the question a load balancer is actually asking.
|
|
Both are open, because a load balancer has no key.
|
|
|
|
During a failover neither passes for about the length of the lease handover:
|
|
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
|
is not. That gap is real and there is no way to have it both ways — traffic
|
|
cannot go to a control plane that has nothing to route to.
|
|
|
|
Not the admin port, for a different reason: it serves pprof, and pprof can
|
|
dump the process heap, where API keys sit in plaintext.
|
|
|
|
## Both control planes need the same store
|
|
|
|
Leadership lives in it, and so do keys, budgets and conversations. Two control
|
|
planes on two SQLite files are two clusters that happen to share a name.
|
|
|
|
```yaml
|
|
auth:
|
|
store: "@/etc/llmgw/secrets/store-dsn"
|
|
```
|
|
|
|
See [docs/install.md](../../docs/install.md) for moving an existing
|
|
installation onto PostgreSQL without reissuing every key.
|