Add replicas under load, up to a model's max, and take them back when quiet
A model's max never did anything: desiredFor returned the minimum, or one for an on-demand model someone was asking for, whatever the queue. Six small CPU nodes run out of slots quickly, and a model with room to grow stayed at its minimum while requests waited. The router already scrapes each engine's running and waiting requests. The control plane now reads them per model on every tick, alongside when each replica was last used, and the reconciler keeps how many replicas load has added above a model's base. One or more requests waiting per ready replica, held for a minute, adds one; ten minutes with nothing waiting removes one of those again. It never goes above max or below the base, moves one step at a time, and does not add while the last addition is still placing or starting — on this fleet a replica is minutes of weights and gigabytes of memory, and a spike should not leave a pile of replicas behind it. Decisions are logged with the load behind them. Nothing live changes until a model's max is raised above its min. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
bd12b0909a
commit
adb1b3cf2a
5 files changed
+224
No files matched your search
@@ -134,6 +134,16 @@ func (c *Controller) collectUsage() {
|
||||
return
|
||||
}
|
||||
c.rec.RecordUsage(c.usage.LastUsed())
|
||||
// And what the replicas are doing, when the source can say: what adds
|
||||
// and removes replicas between a model's min and max.
|
||||
if l, ok := c.usage.(LoadSource); ok {
|
||||
c.rec.RecordLoad(l.ModelLoad())
|
||||
}
|
||||
}
|
||||
|
||||
// LoadSource reports each model's replicas' load, keyed by catalog name.
|
||||
type LoadSource interface {
|
||||
ModelLoad() map[string]reconciler.Load
|
||||
}
|
||||
|
||||
// ReplaceCatalog puts a new catalog in force, without a restart.
|
||||
|
||||
Reference in new issue
Block a user