Ilker ManapandClaude Opus 5.5
37a80a9b44
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not know — was started 155 times in eight hours, failing the same way each time, and phi-4, which fitted on no node, logged "no placement" 250 times in two days. Neither showed up anywhere an operator looked: the backoff never gave up, no metric said a model was failing or unplaceable, and the one alert for a model with nothing serving could not fire, because its series was published only while a replica was ready. After five failures in a row a model is parked: not placed again until its definition changes — a new source or profile — or `llmgw ctl retry` is run. Status, the model list and the admin API show it with its last error. The placement warning is logged when its reason changes or every ten minutes. A request for a parked or unplaceable model is answered at once with why, rather than "loading; retry shortly", which a client retries forever. Each request still clears the previous placement result and triggers a fresh attempt, so a model that fits once room is made is not refused on a stale answer; the router waits for that attempt, not for its full activation window. Metrics now cover model health (desired, ready at 0 for what should run, consecutive start failures, parked, unplaceable), engine installs per node, leadership, and certificate expiry read from disk at each scrape, with alerts for each. `ctl status` counts a model against what it should have now, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
93 lines
3.2 KiB
Go
93 lines
3.2 KiB
Go
package router
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
)
|
|
|
|
// Activator loads a model that is configured but not running.
|
|
//
|
|
// The control plane implements it. Behind an interface for the same reason the
|
|
// endpoint list is: the router knows a request has arrived for something that
|
|
// is not up, the control plane decides what runs, and neither needs to be in
|
|
// the other's process to say so.
|
|
type Activator interface {
|
|
// Want reports whether the model is one that loads on demand. False means
|
|
// there is genuinely nothing serving it, which is a different answer.
|
|
Want(model string) bool
|
|
}
|
|
|
|
// unservable is an activator that can also say a model will not come up at
|
|
// all — it failed every time it started, or fits on no node — so a request is
|
|
// told so at once rather than asked to retry something that never arrives.
|
|
type unservable interface {
|
|
Unservable(model string) (string, bool)
|
|
}
|
|
|
|
// whyUnservable is the activator's reason a model cannot be served, if any.
|
|
func (r *Router) whyUnservable(model string) (string, bool) {
|
|
u, ok := r.activator.(unservable)
|
|
if !ok {
|
|
return "", false
|
|
}
|
|
return u.Unservable(model)
|
|
}
|
|
|
|
// SetActivator supplies the control plane that can load models on demand.
|
|
func (r *Router) SetActivator(a Activator) { r.activator = a }
|
|
|
|
// activate asks for a model and waits a bounded time for it to serve.
|
|
//
|
|
// The one real design decision in on-demand loading. A request that triggers a
|
|
// load can either wait or be refused, and both are wrong on their own: refusing
|
|
// makes the feature useless, because the first request for every model fails;
|
|
// waiting without a bound turns a request into a job, and a client that budgeted
|
|
// thirty seconds is holding a connection for ten minutes.
|
|
//
|
|
// So it waits, briefly, and then says what is happening. A forty-second wait is
|
|
// a slow request. A fifteen-minute wait is a job pretending to be one, and the
|
|
// client should be told to come back rather than left holding a socket.
|
|
func (r *Router) activate(ctx context.Context, model string) bool {
|
|
// Asked first, always: asking is what makes the control plane try again,
|
|
// and a model that fitted nowhere an hour ago may fit now.
|
|
if r.activator == nil || !r.activator.Want(model) {
|
|
return false
|
|
}
|
|
|
|
wait := r.cfg.ActivationWait
|
|
if wait <= 0 {
|
|
wait = defaultActivationWait
|
|
}
|
|
ctx, cancel := context.WithTimeout(ctx, wait)
|
|
defer cancel()
|
|
|
|
// Polled rather than notified. A load takes tens of seconds at best, so the
|
|
// latency of a poll is noise against it, and a notification path would be
|
|
// another thing to get wrong for no gain a client could measure.
|
|
t := time.NewTicker(250 * time.Millisecond)
|
|
defer t.Stop()
|
|
for {
|
|
select {
|
|
case <-ctx.Done():
|
|
return false
|
|
case <-t.C:
|
|
if len(r.reg.Snapshots(model)) > 0 {
|
|
return true
|
|
}
|
|
// Asking set a placement going; if it found nowhere, there is
|
|
// nothing to wait for.
|
|
if _, no := r.whyUnservable(model); no {
|
|
return false
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// defaultActivationWait bounds how long a request holds a connection waiting
|
|
// for a model to load.
|
|
//
|
|
// Long enough for a small model on a warm node, short enough that a client's
|
|
// own timeout is not what ends the request — being told to retry is a better
|
|
// outcome than a socket closing with nothing on it.
|
|
const defaultActivationWait = 45 * time.Second
|