Stop an idle model to make room for one somebody asked for
On-demand loading worked right up to the moment the cluster was full and then stopped meaning anything: the model somebody had just asked for could not be placed, and the cards were held by a model nobody had asked for in an hour. That is the only situation in which any of it matters. Deliberately narrow. Only idle replicas of other on-demand models — a model at min 1 is a promise that it stays up, and breaking it to serve a different request would make every guarantee in the catalog conditional on load. Only single-node replicas, since evicting a multi-node one frees several machines and costs tens of minutes to rebuild. One at a time, because freeing everything that qualifies would empty a cluster for a request that needed one node. And never a replica with no reported usage: after a router restart that is all of them, and unknown is not idle. Writing the test found a gap the feature had rather than a bug in the test. Two on-demand models on a cluster with room for one would take turns evicting each other, at a model load per cycle — worse for both than one of them waiting. A replica is now safe from eviction for a few minutes after it starts serving, which bounds that. Which in turn found a latent bug: deriveState stamped ReadyAt with time.Now() inside a package that has an injectable clock, so the new lifetime rule could not be tested at all — the fake clock said three minutes and the wall clock said eight months. Invisible while nothing compared ReadyAt to anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7fe59bd5fb
commit
9b32531ce3
6 files changed
+412
-8
No files matched your search
@@ -515,6 +515,15 @@ models used daily — keep those at `min: 1`. And stopping a replica throws away
|
||||
the KV cache prefix routing spent its effort filling, so an idle timeout wants
|
||||
to be minutes rather than seconds.
|
||||
|
||||
When the cluster is full, a model somebody asks for will stop an idle one to
|
||||
make room. Only idle replicas of other on-demand models: a model at `min: 1` is
|
||||
a promise that it stays up, and breaking that to serve a different request would
|
||||
make every guarantee in the catalog conditional on load. Only single-node
|
||||
replicas, because evicting a multi-node one frees several machines and costs
|
||||
tens of minutes to rebuild — a decision worth making deliberately. One at a
|
||||
time. And never a replica that has just started, which is what stops two models
|
||||
on a cluster with room for one from taking turns evicting each other.
|
||||
|
||||
For a GPU model, name the class and the width instead:
|
||||
|
||||
```yaml
|
||||
|
||||
Reference in new issue
Block a user