Ilker ManapandClaude Opus 5
2bf7de1a43
CI / test (push) Successful in 1m36s
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
66 lines
2.1 KiB
Plaintext
66 lines
2.1 KiB
Plaintext
# A floating address for two control planes.
|
|
#
|
|
# One IP that lives on whichever machine is up. Clients and the reverse proxy
|
|
# in front of llmgw point at the address, not at either host, so a failover is
|
|
# invisible to them.
|
|
#
|
|
# This is not a load balancer: all traffic goes to the holder. That is usually
|
|
# the right trade for a control plane, which serves far less than the nodes
|
|
# it schedules.
|
|
#
|
|
# Copy to /etc/keepalived/keepalived.conf on both machines, changing state,
|
|
# priority and interface.
|
|
|
|
global_defs {
|
|
enable_script_security
|
|
script_user nobody
|
|
}
|
|
|
|
# The address moves only while llmgw can actually serve, not merely while the
|
|
# machine is powered on.
|
|
#
|
|
# /ready, not /health. /health is 200 on a standby, which is up, answering and
|
|
# holding no endpoints at all — so the address would sit on a host that refuses
|
|
# every request. /ready is 200 only when there is somewhere to send one.
|
|
#
|
|
# Not the admin port either: it serves pprof, which can dump the heap that
|
|
# holds every API key.
|
|
vrrp_script check_llmgw {
|
|
script "/usr/bin/curl -fsS --max-time 2 -o /dev/null http://127.0.0.1:8080/ready"
|
|
interval 2
|
|
timeout 3
|
|
# Two failures before giving up the address, so one slow moment does not
|
|
# move it. Two successes before taking it back, so a flapping gateway does
|
|
# not drag the address with it.
|
|
fall 2
|
|
rise 2
|
|
weight -40
|
|
}
|
|
|
|
vrrp_instance llmgw {
|
|
# MASTER on the machine you would rather serve; BACKUP on the other.
|
|
state MASTER
|
|
interface eth0
|
|
# Any number both machines agree on and nothing else on the segment uses.
|
|
virtual_router_id 51
|
|
# The higher one wins while both are healthy. The weight above is enough
|
|
# to invert this when the preferred machine's gateway is not answering.
|
|
priority 100
|
|
advert_int 1
|
|
|
|
authentication {
|
|
auth_type PASS
|
|
# Change this. It stops another host on the segment claiming the
|
|
# address, which is otherwise a single unicast packet away.
|
|
auth_pass CHANGEME
|
|
}
|
|
|
|
virtual_ipaddress {
|
|
10.0.0.150/24
|
|
}
|
|
|
|
track_script {
|
|
check_llmgw
|
|
}
|
|
}
|