Track which replicas are actually being used, and summarise the fleet
CI / test (push) Successful in 1m28s

Two things a cluster with many nodes and many GPUs needs that were missing,
and they are the same thing seen from two distances.

Idle tracking. The control plane knew what was placed and whether it was
healthy, never whether anyone wanted it — so on a fleet with more models
than cards there was no way to answer the first question anyone asks, which
is what can I stop. The router is the only thing that knows, so it now
reports it: a timestamp written in Acquire, where a request already meets an
endpoint under a lock already held, and collected once a tick as a map. That
shape is the point. Per-request notification would be a firehose carrying
one bit — the newest timestamp wins — and this costs the same on four
replicas as on four hundred.

It travels through an interface in the direction opposite EndpointSink, for
the same reason that one exists: the control plane decides what should run,
the router knows whether anyone wants it, and neither should need to be in
the other's process to say so.

Unknown is kept distinct from idle throughout. A router that has just
restarted knows nothing, and a zero timestamp would be the oldest idle
replica in every comparison — so one restart would nominate an entire fleet
for eviction. Absent from the map, "-" in the table, negative in the JSON,
and IdleFor returns a second value rather than a duration anything could
act on by mistake.

ctl status. A per-node listing is right for four nodes and wrong for two
hundred: how full am I, what is broken, what is holding cards nobody is
using are aggregates, and reading them off a long table is a chore, which
means it gets skipped. Capacity per GPU class with usage bars, free whole
nodes per fabric — the number that decides whether a multi-node replica can
land, since free cards on the wrong fabric are not capacity for a model that
spans nodes — replica and model counts, models that cannot be placed, and
what has gone unasked-for. Derived from the same state the tables show, so
it cannot disagree with them.

Unhealthy GPUs count as neither used nor free, because counting them free is
how a cluster looks like it has room it does not.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 00:52:34 +02:00
1 parent 1431426db0
commit 9d8cd5e284
15 files changed
+928 -20

No files matched your search

+33
View File
@@ -120,6 +120,18 @@ type live struct {
inflight int
brk *breaker.Breaker
// lastUsed is when a request was last dispatched here.
//
// The fact nothing recorded. Whether a replica is earning the cards it
// holds is the first question on a fleet with more models than GPUs, and
// the control plane had no way to ask it: it knew what was placed and
// whether it was healthy, never whether anyone wanted it.
//
// Recorded here because Acquire is already the one place a request meets an
// endpoint, and already holds the lock. One timestamp per dispatch, and
// collection is one pass over the replicas rather than anything per
// request — which is what makes it affordable on a fleet of hundreds.
lastUsed time.Time
}
// Registry holds the live state of all endpoints. It is safe for concurrent use.
@@ -265,6 +277,7 @@ func (r *Registry) Acquire(id string) (release func(ok bool), granted bool) {
r.mu.Lock()
l.inflight++
l.lastUsed = r.now()
r.mu.Unlock()
var once sync.Once
@@ -282,6 +295,26 @@ func (r *Registry) Acquire(id string) (release func(ok bool), granted bool) {
}, true
}
// LastUsed reports when each endpoint last had a request dispatched to it.
//
// Endpoints that have never been used are absent rather than zero-valued: "not
// since this router started" and "not for an hour" are different facts, and a
// zero time would quietly become the oldest idle replica in any comparison —
// so a router restart would nominate everything for eviction at once.
func (r *Registry) LastUsed() map[string]time.Time {
r.mu.RLock()
defer r.mu.RUnlock()
out := make(map[string]time.Time, len(r.m))
for id, l := range r.m {
if l.lastUsed.IsZero() {
continue
}
out[id] = l.lastUsed
}
return out
}
// ScrapeFailed reports a failed metrics scrape.
//
// A scrape failure is not a request failure, so it does not touch the breaker:
+64
View File
@@ -243,3 +243,67 @@ func TestConcurrentAccess(t *testing.T) {
}
wg.Wait()
}
func TestLastUsedIsRecordedOnDispatch(t *testing.T) {
// The fact nothing recorded. Whether a replica is earning the cards it
// holds is the first question on a fleet with more models than GPUs.
r := NewRegistry(Config{})
clock := time.Date(2026, 9, 17, 12, 0, 0, 0, time.UTC)
r.SetClock(func() time.Time { return clock })
r.Replace([]Info{{ID: "a", URL: "http://x", Models: []string{"m"}}})
if len(r.LastUsed()) != 0 {
t.Fatal("an endpoint nothing has used reported a last-use time")
}
release, ok := r.Acquire("a")
if !ok {
t.Fatal("acquire refused")
}
release(true)
used := r.LastUsed()
if got, ok := used["a"]; !ok || !got.Equal(clock) {
t.Errorf("last used = %v (present=%v), want %v", got, ok, clock)
}
}
func TestAnUnusedEndpointIsAbsentRatherThanZero(t *testing.T) {
// A zero time would be the oldest idle replica in any comparison, so a
// router restart would nominate the whole fleet for eviction at once.
r := NewRegistry(Config{})
r.Replace([]Info{
{ID: "used", URL: "http://x", Models: []string{"m"}},
{ID: "never", URL: "http://y", Models: []string{"m"}},
})
release, _ := r.Acquire("used")
release(true)
used := r.LastUsed()
if _, ok := used["never"]; ok {
t.Error("an endpoint nothing has used appeared in the map")
}
if _, ok := used["used"]; !ok {
t.Error("the used endpoint is missing")
}
}
func TestLastUsedSurvivesAnEndpointListRefresh(t *testing.T) {
// The list is republished whenever anything changes, which on a large
// fleet is often. Losing usage on every republish would make idleness
// unmeasurable exactly where it matters most.
r := NewRegistry(Config{})
r.Replace([]Info{{ID: "a", URL: "http://x", Models: []string{"m"}}})
release, _ := r.Acquire("a")
release(true)
before := r.LastUsed()["a"]
// Same endpoint, republished alongside a new one.
r.Replace([]Info{
{ID: "a", URL: "http://x", Models: []string{"m"}},
{ID: "b", URL: "http://y", Models: []string{"m"}},
})
if got, ok := r.LastUsed()["a"]; !ok || !got.Equal(before) {
t.Errorf("last used = %v (present=%v) after a republish, want %v", got, ok, before)
}
}
+5
View File
@@ -335,6 +335,11 @@ func (r *Router) Registry() *endpoint.Registry { return r.reg }
// the update keep their live state — inflight counts, breaker state, drain rate
// — so a replica that was already serving does not start from cold just
// because another one moved.
// LastUsed reports when each endpoint was last sent a request, satisfying
// control.UsageSource so the control plane can tell a busy replica from one
// holding its cards out of habit.
func (r *Router) LastUsed() map[string]time.Time { return r.reg.LastUsed() }
func (r *Router) SetEndpoints(ctx context.Context, infos []endpoint.Info) {
if r.prefix != nil {
// A replica that has gone takes its cache with it. Leaving its entries