Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s

The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.

No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.

The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.

Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 17:44:00 +02:00
1 parent b619546516
commit 7fe59bd5fb
11 files changed
+521 -5

No files matched your search

+27
View File
@@ -488,6 +488,33 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
No restart, and it survives one: models added this way are stored, while
anything in `server.yaml` is declarative and returns on every start.
### Models that load when someone asks
A cluster with more models than GPUs can hold every model in its catalog and
only the wanted ones in its cards:
```yaml
scaling: { min: 0, max: 1, idle_timeout: 15m }
```
`min: 0` with a `max` above it means load on demand. The first request for the
model starts it and waits briefly; if it is not serving by then the client gets
a 503 with `Retry-After` and an honest "the model is loading", rather than being
told nothing serves it — a client told to retry succeeds where one told the
model does not exist gives up. Once nobody has asked for `idle_timeout`, the
replicas are stopped and the cards go back to the pool.
`min: 0, max: 0` is different and equally useful: configured, and deliberately
not running. A model with no `scaling` stanza at all gets that, because making
every unscaled model loadable on demand would let any client request start a
70B nobody intended to run.
Two things to weigh. A cold start is visible to whoever triggers it, so this
suits the long tail people are experimenting with rather than the two or three
models used daily — keep those at `min: 1`. And stopping a replica throws away
the KV cache prefix routing spent its effort filling, so an idle timeout wants
to be minutes rather than seconds.
For a GPU model, name the class and the width instead:
```yaml