Give two control planes a shared address, and a probe that means something

The last piece of HA, and it is a deployment pattern rather than code: llmgw
picks one leader and agents follow it, but client traffic arrives in front of
both and llmgw is not in front of anything.

Writing the pattern found a bug in the pattern. The first draft said to
health-check `/health` and claimed a standby still routes to replicas the
leader placed, so sending it traffic was fine. It does not. A standby's
reconciler places nothing and it holds no agent streams to adopt from, so it
publishes an empty endpoint list — a load balancer following that advice would
have sent half the traffic to something that answers every request with 503.

So there is now a `/ready` that means "this process has somewhere to send a
request": 503 on a standby, 503 on a leader whose replicas have all failed,
200 when there is something to serve. `/health` keeps meaning the process is
up, which is a different and less useful question. Both are open, because a
load balancer has no key.

There is a gap during failover where neither passes — three seconds when the
leader stops cleanly, up to the lease TTL when it does not — and the document
says so rather than implying otherwise. Traffic cannot go to a control plane
that has nothing to route to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 07:52:54 +02:00
1 parent a35e8796d2
commit 2bf7de1a43
6 files changed
+257 -1

No files matched your search

+23 -1
View File
@@ -82,10 +82,32 @@ func (r *Router) Handler() http.Handler {
mux.HandleFunc("POST /v1/completions", r.handleInference("/v1/completions"))
mux.HandleFunc("POST /v1/embeddings", r.handleInference("/v1/embeddings"))
mux.HandleFunc("GET /v1/models", r.handleModels)
// Liveness: this process is up. It says nothing about whether it can
// serve, which is why it is not the thing to put a load balancer on.
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK)
})
// Readiness: this process has somewhere to send a request.
//
// The distinction is not academic. A standby control plane is up, answers
// /health, and has no endpoints at all — its reconciler placed nothing and
// it holds no agent streams to adopt from — so a load balancer checking
// /health would send it half the traffic and half the traffic would be a
// 503. A leader whose replicas have all failed is in the same position for
// a different reason, and a load balancer should treat them the same.
//
// Open, like /health: a load balancer does not have a key, and this leaks
// one bit — whether anything is servable — which the next request would
// reveal anyway.
mux.HandleFunc("GET /ready", func(w http.ResponseWriter, _ *http.Request) {
if len(r.reg.IDs()) == 0 {
http.Error(w, "no endpoints are available to serve", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
// The landing page says how to connect and nothing about what is here, so
// it is safe to serve without a key — and it has to be, because the person
// reading it is the one who does not have one yet.
@@ -143,7 +165,7 @@ func (r *Router) Handler() http.Handler {
// route must not be able to add an unauthenticated one by omission. Health
// stays open because a load balancer has no key.
return auth.Middleware(r.auth, r.log,
"/health", "/", "/_static/", "/claim/",
"/health", "/ready", "/", "/_static/", "/claim/",
"/signup", "/login", "/logout", "/profile", "/profile/",
"/admin", "/admin/", "/chat", "/chat/")(mux)
}