Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b619546516
commit
7fe59bd5fb
11 files changed
+521
-5
No files matched your search
@@ -488,6 +488,33 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
No restart, and it survives one: models added this way are stored, while
|
||||
anything in `server.yaml` is declarative and returns on every start.
|
||||
|
||||
### Models that load when someone asks
|
||||
|
||||
A cluster with more models than GPUs can hold every model in its catalog and
|
||||
only the wanted ones in its cards:
|
||||
|
||||
```yaml
|
||||
scaling: { min: 0, max: 1, idle_timeout: 15m }
|
||||
```
|
||||
|
||||
`min: 0` with a `max` above it means load on demand. The first request for the
|
||||
model starts it and waits briefly; if it is not serving by then the client gets
|
||||
a 503 with `Retry-After` and an honest "the model is loading", rather than being
|
||||
told nothing serves it — a client told to retry succeeds where one told the
|
||||
model does not exist gives up. Once nobody has asked for `idle_timeout`, the
|
||||
replicas are stopped and the cards go back to the pool.
|
||||
|
||||
`min: 0, max: 0` is different and equally useful: configured, and deliberately
|
||||
not running. A model with no `scaling` stanza at all gets that, because making
|
||||
every unscaled model loadable on demand would let any client request start a
|
||||
70B nobody intended to run.
|
||||
|
||||
Two things to weigh. A cold start is visible to whoever triggers it, so this
|
||||
suits the long tail people are experimenting with rather than the two or three
|
||||
models used daily — keep those at `min: 1`. And stopping a replica throws away
|
||||
the KV cache prefix routing spent its effort filling, so an idle timeout wants
|
||||
to be minutes rather than seconds.
|
||||
|
||||
For a GPU model, name the class and the width instead:
|
||||
|
||||
```yaml
|
||||
|
||||
Reference in new issue
Block a user