Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
74429f1c00
commit
bce54f0941
14 files changed
+382
-9
No files matched your search
@@ -924,6 +924,26 @@ Models added at runtime only. One declared in `server.yaml` is read again on
|
||||
every start, so a number set here would hold until the next restart and then
|
||||
revert — change it in the file instead.
|
||||
|
||||
Between `min` and `max` a model scales with its load: requests waiting at
|
||||
every replica add one, a quiet spell takes it back. For load that comes by the
|
||||
clock, a schedule says so ahead of time:
|
||||
|
||||
```yaml
|
||||
scaling:
|
||||
min: 1
|
||||
max: 4
|
||||
prewarm: 30m # start a window's replicas this long before it opens
|
||||
schedule:
|
||||
- {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
|
||||
- {from: "22:00", to: "02:00", min: 1} # past midnight
|
||||
```
|
||||
|
||||
Times are in `control.timezone` (`Europe/Istanbul`; empty is the control
|
||||
plane's own). The first window that holds wins; outside every window, `min`
|
||||
and `max` apply. An on-demand model (`min: 0`) with a window is kept loaded
|
||||
through it, and loaded again on the first request after. Days are `mon` …
|
||||
`sun`, `weekdays` or `weekends`.
|
||||
|
||||
How much traffic each replica is given is the profile's `weight`, and guessing
|
||||
it is how a cluster ends up with one saturated replica beside two idle ones.
|
||||
Measure it instead:
|
||||
|
||||
Reference in new issue
Block a user