Ilker ManapandClaude Opus 5
7fe59bd5fb
The feature the idle tracking was for. A cluster with more models than GPUs can now hold every model in its catalog and only the wanted ones in its cards: min 0 with a max above it means load on demand, and once nobody has asked for idle_timeout the replicas stop and the cards go back. No new field, because min 0 max 1 already reads correctly — keep none running, allow up to one. Which made min 0 max 0 meaningful for the first time: configured and deliberately not running. normalize() used to fill a zero max in, on the reasonable ground that a model with no maximum cannot run; that would now silently turn every switched-off model into an on-demand one, including the ones already written that way in this cluster's own configuration. A model with no scaling stanza stays off for the same reason — defaulting it to on-demand would let any client request start a 70B nobody intended to run. The one real decision is what happens to the request that triggers a load. Refusing makes the feature useless, since the first request for every model fails; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds holds a connection for ten minutes. So it waits briefly and then says what is happening — 503 with Retry-After and "the model is loading", which is a different answer from "no endpoint serves this" and produces different client behaviour. An unknown model still gets the unknown-model error, or on-demand would turn every typo into "loading" and the useful message would disappear. Use keeps a model alive rather than only the request that started it, without which the timer measures time since the load and a busy model is evicted on schedule however hard it is working. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
70 lines
2.4 KiB
Go
70 lines
2.4 KiB
Go
package router
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
)
|
|
|
|
// Activator loads a model that is configured but not running.
|
|
//
|
|
// The control plane implements it. Behind an interface for the same reason the
|
|
// endpoint list is: the router knows a request has arrived for something that
|
|
// is not up, the control plane decides what runs, and neither needs to be in
|
|
// the other's process to say so.
|
|
type Activator interface {
|
|
// Want reports whether the model is one that loads on demand. False means
|
|
// there is genuinely nothing serving it, which is a different answer.
|
|
Want(model string) bool
|
|
}
|
|
|
|
// SetActivator supplies the control plane that can load models on demand.
|
|
func (r *Router) SetActivator(a Activator) { r.activator = a }
|
|
|
|
// activate asks for a model and waits a bounded time for it to serve.
|
|
//
|
|
// The one real design decision in on-demand loading. A request that triggers a
|
|
// load can either wait or be refused, and both are wrong on their own: refusing
|
|
// makes the feature useless, because the first request for every model fails;
|
|
// waiting without a bound turns a request into a job, and a client that budgeted
|
|
// thirty seconds is holding a connection for ten minutes.
|
|
//
|
|
// So it waits, briefly, and then says what is happening. A forty-second wait is
|
|
// a slow request. A fifteen-minute wait is a job pretending to be one, and the
|
|
// client should be told to come back rather than left holding a socket.
|
|
func (r *Router) activate(ctx context.Context, model string) bool {
|
|
if r.activator == nil || !r.activator.Want(model) {
|
|
return false
|
|
}
|
|
|
|
wait := r.cfg.ActivationWait
|
|
if wait <= 0 {
|
|
wait = defaultActivationWait
|
|
}
|
|
ctx, cancel := context.WithTimeout(ctx, wait)
|
|
defer cancel()
|
|
|
|
// Polled rather than notified. A load takes tens of seconds at best, so the
|
|
// latency of a poll is noise against it, and a notification path would be
|
|
// another thing to get wrong for no gain a client could measure.
|
|
t := time.NewTicker(250 * time.Millisecond)
|
|
defer t.Stop()
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return false
|
|
case <-t.C:
|
|
if len(r.reg.Snapshots(model)) > 0 {
|
|
return true
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// defaultActivationWait bounds how long a request holds a connection waiting
|
|
// for a model to load.
|
|
//
|
|
// Long enough for a small model on a warm node, short enough that a client's
|
|
// own timeout is not what ends the request — being told to retry is a better
|
|
// outcome than a socket closing with nothing on it.
|
|
const defaultActivationWait = 45 * time.Second
|