Load a model when someone asks for it, and stop it when nobody does

The feature the idle tracking was for. A cluster with more models than GPUs
can now hold every model in its catalog and only the wanted ones in its
cards: min 0 with a max above it means load on demand, and once nobody has
asked for idle_timeout the replicas stop and the cards go back.

No new field, because min 0 max 1 already reads correctly — keep none
running, allow up to one. Which made min 0 max 0 meaningful for the first
time: configured and deliberately not running. normalize() used to fill a
zero max in, on the reasonable ground that a model with no maximum cannot
run; that would now silently turn every switched-off model into an
on-demand one, including the ones already written that way in this
cluster's own configuration. A model with no scaling stanza stays off for
the same reason — defaulting it to on-demand would let any client request
start a 70B nobody intended to run.

The one real decision is what happens to the request that triggers a load.
Refusing makes the feature useless, since the first request for every model
fails; waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds holds a connection for ten minutes. So it waits
briefly and then says what is happening — 503 with Retry-After and "the
model is loading", which is a different answer from "no endpoint serves
this" and produces different client behaviour. An unknown model still gets
the unknown-model error, or on-demand would turn every typo into "loading"
and the useful message would disappear.

Use keeps a model alive rather than only the request that started it,
without which the timer measures time since the load and a busy model is
evicted on schedule however hard it is working.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 17:44:00 +02:00
1 parent b619546516
commit 7fe59bd5fb
11 files changed
+521 -5

No files matched your search

+24 -2
View File
@@ -16,8 +16,24 @@ import (
type Scaling struct {
Min int `json:"min,omitempty" koanf:"min"`
Max int `json:"max,omitempty" koanf:"max"`
// IdleTimeout is how long an on-demand model may go unasked-for before its
// replicas are stopped. Zero takes the cluster default.
//
// Minutes rather than seconds, always. Stopping a replica throws away the
// KV cache prefix routing spent its effort filling, and a model that flaps
// in and out loses that entirely — so the saving has to be worth more than
// the cache, which it only is over a real idle period.
IdleTimeout time.Duration `json:"idle_timeout,omitempty" koanf:"idle_timeout"`
}
// OnDemand reports whether this model is loaded when someone asks for it.
//
// min 0 with a max above it, which needs no new field and reads correctly: keep
// none running, allow up to max. min 0 and max 0 is a model that is configured
// and deliberately not running, which is a different and equally useful thing —
// the difference between "load it when wanted" and "do not load it".
func (s Scaling) OnDemand() bool { return s.Min == 0 && s.Max > 0 }
// SLO is the latency target a model is expected to meet. `ctl bench` measures
// capacity against it (plan §6.2.1).
type SLO struct {
@@ -193,8 +209,14 @@ func (m Model) normalize() Model {
if m.Scaling.Min < 0 {
m.Scaling.Min = 0
}
if m.Scaling.Max == 0 {
m.Scaling.Max = max(m.Scaling.Min, 1)
// Max 0 used to be an omission to fill in, because a model with no maximum
// could not run. It is now also a statement: with min 0 it means configured
// and deliberately not running, which is a different and useful thing from
// "load it when wanted". Filling it in would silently turn every switched-
// off model into an on-demand one — including the ones already written that
// way in a running cluster's configuration.
if m.Scaling.Max == 0 && m.Scaling.Min > 0 {
m.Scaling.Max = m.Scaling.Min
}
for i := range m.Profiles {
p := &m.Profiles[i]
+45 -2
View File
@@ -124,8 +124,51 @@ func TestDefaults(t *testing.T) {
if p.Weight <= 0 {
t.Errorf("weight=%v", p.Weight)
}
if m.Scaling.Max < 1 {
t.Errorf("scaling max=%d", m.Scaling.Max)
// Max is no longer filled in for a model that names no minimum. With min 0
// it now means "configured and deliberately not running", and defaulting it
// to 1 would make every model without a scaling stanza loadable on demand —
// so any client request could start a 70B nobody intended to run.
if m.Scaling.Min == 0 && m.Scaling.Max != 0 {
t.Errorf("scaling max=%d; an unscaled model should stay off, not become on-demand", m.Scaling.Max)
}
if m.Scaling.OnDemand() {
t.Error("a model with no scaling stanza was made on-demand")
}
}
func TestMaxDefaultsToTheMinimumWhenOneIsGiven(t *testing.T) {
// A model that says it wants two replicas and nothing about a maximum wants
// two, not one and not unbounded.
built, err := New([]Model{{
Name: "m", Source: "/m", Scaling: Scaling{Min: 2},
Profiles: []Profile{{Name: "p", Engine: "llamacpp"}},
}})
if err != nil {
t.Fatal(err)
}
m, _ := built.Resolve("m")
if m.Scaling.Max != 2 {
t.Errorf("max=%d, want 2", m.Scaling.Max)
}
if m.Scaling.OnDemand() {
t.Error("a model with min 2 was reported on-demand")
}
}
func TestOnDemandNeedsBothZeroMinimumAndARealMaximum(t *testing.T) {
for _, tc := range []struct {
name string
s Scaling
want bool
}{
{"off", Scaling{Min: 0, Max: 0}, false},
{"on demand", Scaling{Min: 0, Max: 1}, true},
{"always on", Scaling{Min: 1, Max: 1}, false},
{"always on, room to grow", Scaling{Min: 1, Max: 4}, false},
} {
if got := tc.s.OnDemand(); got != tc.want {
t.Errorf("%s: OnDemand() = %v, want %v", tc.name, got, tc.want)
}
}
}