Ilker ManapandClaude Opus 5.5
adb1b3cf2a
A model's max never did anything: desiredFor returned the minimum, or one for an on-demand model someone was asking for, whatever the queue. Six small CPU nodes run out of slots quickly, and a model with room to grow stayed at its minimum while requests waited. The router already scrapes each engine's running and waiting requests. The control plane now reads them per model on every tick, alongside when each replica was last used, and the reconciler keeps how many replicas load has added above a model's base. One or more requests waiting per ready replica, held for a minute, adds one; ten minutes with nothing waiting removes one of those again. It never goes above max or below the base, moves one step at a time, and does not add while the last addition is still placing or starting — on this fleet a replica is minutes of weights and gigabytes of memory, and a spike should not leave a pile of replicas behind it. Decisions are logged with the load behind them. Nothing live changes until a model's max is raised above its min. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>