Give two control planes a shared address, and a probe that means something
The last piece of HA, and it is a deployment pattern rather than code: llmgw picks one leader and agents follow it, but client traffic arrives in front of both and llmgw is not in front of anything. Writing the pattern found a bug in the pattern. The first draft said to health-check `/health` and claimed a standby still routes to replicas the leader placed, so sending it traffic was fine. It does not. A standby's reconciler places nothing and it holds no agent streams to adopt from, so it publishes an empty endpoint list — a load balancer following that advice would have sent half the traffic to something that answers every request with 503. So there is now a `/ready` that means "this process has somewhere to send a request": 503 on a standby, 503 on a leader whose replicas have all failed, 200 when there is something to serve. `/health` keeps meaning the process is up, which is a different and less useful question. Both are open, because a load balancer has no key. There is a gap during failover where neither passes — three seconds when the leader stops cleanly, up to the lease TTL when it does not — and the document says so rather than implying otherwise. Traffic cannot go to a control plane that has nothing to route to. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
a35e8796d2
commit
2bf7de1a43
6 files changed
+257
-1
No files matched your search
@@ -82,10 +82,32 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/completions", r.handleInference("/v1/completions"))
|
||||
mux.HandleFunc("POST /v1/embeddings", r.handleInference("/v1/embeddings"))
|
||||
mux.HandleFunc("GET /v1/models", r.handleModels)
|
||||
// Liveness: this process is up. It says nothing about whether it can
|
||||
// serve, which is why it is not the thing to put a load balancer on.
|
||||
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
// Readiness: this process has somewhere to send a request.
|
||||
//
|
||||
// The distinction is not academic. A standby control plane is up, answers
|
||||
// /health, and has no endpoints at all — its reconciler placed nothing and
|
||||
// it holds no agent streams to adopt from — so a load balancer checking
|
||||
// /health would send it half the traffic and half the traffic would be a
|
||||
// 503. A leader whose replicas have all failed is in the same position for
|
||||
// a different reason, and a load balancer should treat them the same.
|
||||
//
|
||||
// Open, like /health: a load balancer does not have a key, and this leaks
|
||||
// one bit — whether anything is servable — which the next request would
|
||||
// reveal anyway.
|
||||
mux.HandleFunc("GET /ready", func(w http.ResponseWriter, _ *http.Request) {
|
||||
if len(r.reg.IDs()) == 0 {
|
||||
http.Error(w, "no endpoints are available to serve", http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
})
|
||||
|
||||
// The landing page says how to connect and nothing about what is here, so
|
||||
// it is safe to serve without a key — and it has to be, because the person
|
||||
// reading it is the one who does not have one yet.
|
||||
@@ -143,7 +165,7 @@ func (r *Router) Handler() http.Handler {
|
||||
// route must not be able to add an unauthenticated one by omission. Health
|
||||
// stays open because a load balancer has no key.
|
||||
return auth.Middleware(r.auth, r.log,
|
||||
"/health", "/", "/_static/", "/claim/",
|
||||
"/health", "/ready", "/", "/_static/", "/claim/",
|
||||
"/signup", "/login", "/logout", "/profile", "/profile/",
|
||||
"/admin", "/admin/", "/chat", "/chat/")(mux)
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user