Give two control planes a shared address, and a probe that means something
CI / test (push) Successful in 1m36s
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a35e8796d2
commit
2bf7de1a43
6 files changed
+257
-1
No files matched your search
@@ -0,0 +1,87 @@
|
||||
# Two control planes
|
||||
|
||||
llmgw handles the hard half of this itself: only one control plane places work,
|
||||
and agents find whichever one is leading. What it does not do is move client
|
||||
traffic, because that happens in front of both and llmgw is not in front of
|
||||
anything.
|
||||
|
||||
This directory is the missing half. It is a pattern rather than a product:
|
||||
substitute your own load balancer if you have one.
|
||||
|
||||
## What llmgw does on its own
|
||||
|
||||
**One leader.** Leadership is a lease in the shared store. A second control
|
||||
plane starts, sees the lease, and stands by — it places nothing, evicts
|
||||
nothing and drains nothing. Start one by accident and your cluster is
|
||||
unharmed.
|
||||
|
||||
**Agents follow.** A standby refuses agent streams, so agents move to the
|
||||
leader by themselves. Give each agent both addresses:
|
||||
|
||||
```yaml
|
||||
servers:
|
||||
- "https://ctl-1.internal:7443"
|
||||
- "https://ctl-2.internal:7443"
|
||||
```
|
||||
|
||||
Measured on a cluster: leadership moved three seconds after the leader was
|
||||
stopped, every agent had arrived six seconds later, and every replica was
|
||||
adopted rather than restarted — the engines never stopped serving.
|
||||
|
||||
## What it does not do
|
||||
|
||||
Client traffic. During that same failover the standby served inference
|
||||
correctly on its own port while the public address returned 502, because
|
||||
nothing in front of the two had moved.
|
||||
|
||||
So something has to. The options, in the order most people should consider
|
||||
them:
|
||||
|
||||
**A load balancer you already run.** Health-check `/ready` on each control
|
||||
plane and send traffic to whichever answers. This is the right answer if you
|
||||
have one, and it needs nothing else from this directory.
|
||||
|
||||
**A floating address.** `keepalived.conf.example` here moves one IP between
|
||||
the two machines. Fewer moving parts than a load balancer and no extra hop,
|
||||
at the cost of one machine carrying all the traffic.
|
||||
|
||||
**DNS.** Works, eventually. A record with a short TTL is a failover measured
|
||||
in whatever your resolvers actually honour rather than what you set, which on
|
||||
a bad day is minutes. Fine for a cluster people use interactively, wrong for
|
||||
one an agent is driving.
|
||||
|
||||
## Check `/ready`, not `/health`
|
||||
|
||||
This is the one thing to get right, and the obvious choice is the wrong one.
|
||||
|
||||
`/health` says the process is up. It says nothing about whether it can serve,
|
||||
and a standby is exactly that case: alive, answering, and holding no endpoints
|
||||
at all — its reconciler placed nothing and it has no agent streams to adopt
|
||||
from. A load balancer checking `/health` would send it half the traffic, and
|
||||
half the traffic would be a 503.
|
||||
|
||||
`/ready` says this process has somewhere to send a request. It is 503 on a
|
||||
standby, 503 on a leader whose replicas have all failed, and 200 when there is
|
||||
something to serve — which is the question a load balancer is actually asking.
|
||||
Both are open, because a load balancer has no key.
|
||||
|
||||
During a failover neither passes for about the length of the lease handover:
|
||||
three seconds when the leader is stopped cleanly, up to the lease TTL when it
|
||||
is not. That gap is real and there is no way to have it both ways — traffic
|
||||
cannot go to a control plane that has nothing to route to.
|
||||
|
||||
Not the admin port, for a different reason: it serves pprof, and pprof can
|
||||
dump the process heap, where API keys sit in plaintext.
|
||||
|
||||
## Both control planes need the same store
|
||||
|
||||
Leadership lives in it, and so do keys, budgets and conversations. Two control
|
||||
planes on two SQLite files are two clusters that happen to share a name.
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
store: "@/etc/llmgw/secrets/store-dsn"
|
||||
```
|
||||
|
||||
See [docs/install.md](../../docs/install.md) for moving an existing
|
||||
installation onto PostgreSQL without reissuing every key.
|
||||
@@ -0,0 +1,65 @@
|
||||
# A floating address for two control planes.
|
||||
#
|
||||
# One IP that lives on whichever machine is up. Clients and the reverse proxy
|
||||
# in front of llmgw point at the address, not at either host, so a failover is
|
||||
# invisible to them.
|
||||
#
|
||||
# This is not a load balancer: all traffic goes to the holder. That is usually
|
||||
# the right trade for a control plane, which serves far less than the nodes
|
||||
# it schedules.
|
||||
#
|
||||
# Copy to /etc/keepalived/keepalived.conf on both machines, changing state,
|
||||
# priority and interface.
|
||||
|
||||
global_defs {
|
||||
enable_script_security
|
||||
script_user nobody
|
||||
}
|
||||
|
||||
# The address moves only while llmgw can actually serve, not merely while the
|
||||
# machine is powered on.
|
||||
#
|
||||
# /ready, not /health. /health is 200 on a standby, which is up, answering and
|
||||
# holding no endpoints at all — so the address would sit on a host that refuses
|
||||
# every request. /ready is 200 only when there is somewhere to send one.
|
||||
#
|
||||
# Not the admin port either: it serves pprof, which can dump the heap that
|
||||
# holds every API key.
|
||||
vrrp_script check_llmgw {
|
||||
script "/usr/bin/curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8080/ready"
|
||||
interval 2
|
||||
timeout 3
|
||||
# Two failures before giving up the address, so one slow moment does not
|
||||
# move it. Two successes before taking it back, so a flapping gateway does
|
||||
# not drag the address with it.
|
||||
fall 2
|
||||
rise 2
|
||||
weight -40
|
||||
}
|
||||
|
||||
vrrp_instance llmgw {
|
||||
# MASTER on the machine you would rather serve; BACKUP on the other.
|
||||
state MASTER
|
||||
interface eth0
|
||||
# Any number both machines agree on and nothing else on the segment uses.
|
||||
virtual_router_id 51
|
||||
# The higher one wins while both are healthy. The weight above is enough
|
||||
# to invert this when the preferred machine's gateway is not answering.
|
||||
priority 100
|
||||
advert_int 1
|
||||
|
||||
authentication {
|
||||
auth_type PASS
|
||||
# Change this. It stops another host on the segment claiming the
|
||||
# address, which is otherwise a single unicast packet away.
|
||||
auth_pass CHANGEME
|
||||
}
|
||||
|
||||
virtual_ipaddress {
|
||||
10.0.0.150/24
|
||||
}
|
||||
|
||||
track_script {
|
||||
check_llmgw
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user