Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b619546516
commit
7fe59bd5fb
11 files changed
+521
-5
No files matched your search
@@ -124,8 +124,51 @@ func TestDefaults(t *testing.T) {
|
||||
if p.Weight <= 0 {
|
||||
t.Errorf("weight=%v", p.Weight)
|
||||
}
|
||||
if m.Scaling.Max < 1 {
|
||||
t.Errorf("scaling max=%d", m.Scaling.Max)
|
||||
// Max is no longer filled in for a model that names no minimum. With min 0
|
||||
// it now means "configured and deliberately not running", and defaulting it
|
||||
// to 1 would make every model without a scaling stanza loadable on demand —
|
||||
// so any client request could start a 70B nobody intended to run.
|
||||
if m.Scaling.Min == 0 && m.Scaling.Max != 0 {
|
||||
t.Errorf("scaling max=%d; an unscaled model should stay off, not become on-demand", m.Scaling.Max)
|
||||
}
|
||||
if m.Scaling.OnDemand() {
|
||||
t.Error("a model with no scaling stanza was made on-demand")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMaxDefaultsToTheMinimumWhenOneIsGiven(t *testing.T) {
|
||||
// A model that says it wants two replicas and nothing about a maximum wants
|
||||
// two, not one and not unbounded.
|
||||
built, err := New([]Model{{
|
||||
Name: "m", Source: "/m", Scaling: Scaling{Min: 2},
|
||||
Profiles: []Profile{{Name: "p", Engine: "llamacpp"}},
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := built.Resolve("m")
|
||||
if m.Scaling.Max != 2 {
|
||||
t.Errorf("max=%d, want 2", m.Scaling.Max)
|
||||
}
|
||||
if m.Scaling.OnDemand() {
|
||||
t.Error("a model with min 2 was reported on-demand")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnDemandNeedsBothZeroMinimumAndARealMaximum(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
s Scaling
|
||||
want bool
|
||||
}{
|
||||
{"off", Scaling{Min: 0, Max: 0}, false},
|
||||
{"on demand", Scaling{Min: 0, Max: 1}, true},
|
||||
{"always on", Scaling{Min: 1, Max: 1}, false},
|
||||
{"always on, room to grow", Scaling{Min: 1, Max: 4}, false},
|
||||
} {
|
||||
if got := tc.s.OnDemand(); got != tc.want {
|
||||
t.Errorf("%s: OnDemand() = %v, want %v", tc.name, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user