Load a model when someone asks for it, and stop it when nobody does
CI / test (push) Successful in 1m29s
CI / test (push) Successful in 1m29s
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
b619546516
commit
7fe59bd5fb
11 files changed
+521
-5
No files matched your search
@@ -16,8 +16,24 @@ import (
|
||||
type Scaling struct {
|
||||
Min int `json:"min,omitempty" koanf:"min"`
|
||||
Max int `json:"max,omitempty" koanf:"max"`
|
||||
// IdleTimeout is how long an on-demand model may go unasked-for before its
|
||||
// replicas are stopped. Zero takes the cluster default.
|
||||
//
|
||||
// Minutes rather than seconds, always. Stopping a replica throws away the
|
||||
// KV cache prefix routing spent its effort filling, and a model that flaps
|
||||
// in and out loses that entirely — so the saving has to be worth more than
|
||||
// the cache, which it only is over a real idle period.
|
||||
IdleTimeout time.Duration `json:"idle_timeout,omitempty" koanf:"idle_timeout"`
|
||||
}
|
||||
|
||||
// OnDemand reports whether this model is loaded when someone asks for it.
|
||||
//
|
||||
// min 0 with a max above it, which needs no new field and reads correctly: keep
|
||||
// none running, allow up to max. min 0 and max 0 is a model that is configured
|
||||
// and deliberately not running, which is a different and equally useful thing —
|
||||
// the difference between "load it when wanted" and "do not load it".
|
||||
func (s Scaling) OnDemand() bool { return s.Min == 0 && s.Max > 0 }
|
||||
|
||||
// SLO is the latency target a model is expected to meet. `ctl bench` measures
|
||||
// capacity against it (plan §6.2.1).
|
||||
type SLO struct {
|
||||
@@ -193,8 +209,14 @@ func (m Model) normalize() Model {
|
||||
if m.Scaling.Min < 0 {
|
||||
m.Scaling.Min = 0
|
||||
}
|
||||
if m.Scaling.Max == 0 {
|
||||
m.Scaling.Max = max(m.Scaling.Min, 1)
|
||||
// Max 0 used to be an omission to fill in, because a model with no maximum
|
||||
// could not run. It is now also a statement: with min 0 it means configured
|
||||
// and deliberately not running, which is a different and useful thing from
|
||||
// "load it when wanted". Filling it in would silently turn every switched-
|
||||
// off model into an on-demand one — including the ones already written that
|
||||
// way in a running cluster's configuration.
|
||||
if m.Scaling.Max == 0 && m.Scaling.Min > 0 {
|
||||
m.Scaling.Max = m.Scaling.Min
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
|
||||
Reference in new issue
Block a user