Add replicas under load, up to a model's max, and take them back when quiet
A model's max never did anything: desiredFor returned the minimum, or one for an on-demand model someone was asking for, whatever the queue. Six small CPU nodes run out of slots quickly, and a model with room to grow stayed at its minimum while requests waited. The router already scrapes each engine's running and waiting requests. The control plane now reads them per model on every tick, alongside when each replica was last used, and the reconciler keeps how many replicas load has added above a model's base. One or more requests waiting per ready replica, held for a minute, adds one; ten minutes with nothing waiting removes one of those again. It never goes above max or below the base, moves one step at a time, and does not add while the last addition is still placing or starting — on this fleet a replica is minutes of weights and gigabytes of memory, and a spike should not leave a pile of replicas behind it. Decisions are logged with the load behind them. Nothing live changes until a model's max is raised above its min. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
bd12b0909a
commit
adb1b3cf2a
5 files changed
+224
No files matched your search
@@ -9,6 +9,7 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"log/slog"
|
||||
"math"
|
||||
"net/http"
|
||||
@@ -510,6 +511,24 @@ func (r *Router) SetCatalog(c *catalog.Catalog) {
|
||||
// holding its cards out of habit.
|
||||
func (r *Router) LastUsed() map[string]time.Time { return r.reg.LastUsed() }
|
||||
|
||||
// ModelLoad sums what each model's replicas report: how many there are, and
|
||||
// requests running and waiting at their engines. Keyed by the catalog name,
|
||||
// which the control plane publishes first among each endpoint's names.
|
||||
func (r *Router) ModelLoad() map[string]reconciler.Load {
|
||||
out := map[string]reconciler.Load{}
|
||||
for _, s := range r.reg.Snapshots("") {
|
||||
if len(s.Models) == 0 {
|
||||
continue
|
||||
}
|
||||
l := out[s.Models[0]]
|
||||
l.Replicas++
|
||||
l.Running += s.Stats.Running
|
||||
l.Waiting += s.Stats.Waiting
|
||||
out[s.Models[0]] = l
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (r *Router) SetEndpoints(ctx context.Context, infos []endpoint.Info) {
|
||||
if r.prefix != nil {
|
||||
// A replica that has gone takes its cache with it. Leaving its entries
|
||||
|
||||
Reference in new issue
Block a user