Stop an idle model to make room for one somebody asked for

On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.

Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.

Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.

Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 17:51:07 +02:00
1 parent 7fe59bd5fb
commit 9b32531ce3
6 files changed
+412 -8

No files matched your search

+9
View File
@@ -515,6 +515,15 @@ models used daily — keep those at `min: 1`. And stopping a replica throws away
the KV cache prefix routing spent its effort filling, so an idle timeout wants
to be minutes rather than seconds.
When the cluster is full, a model somebody asks for will stop an idle one to
make room. Only idle replicas of other on-demand models: a model at `min: 1` is
a promise that it stays up, and breaking that to serve a different request would
make every guarantee in the catalog conditional on load. Only single-node
replicas, because evicting a multi-node one frees several machines and costs
tens of minutes to rebuild — a decision worth making deliberately. One at a
time. And never a replica that has just started, which is what stops two models
on a cluster with room for one from taking turns evicting each other.
For a GPU model, name the class and the width instead:
```yaml