Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:16:56 +02:00
1 parent 74429f1c00
commit bce54f0941
14 files changed
+382 -9

No files matched your search

+20
View File
@@ -924,6 +924,26 @@ Models added at runtime only. One declared in `server.yaml` is read again on
every start, so a number set here would hold until the next restart and then
revert — change it in the file instead.
Between `min` and `max` a model scales with its load: requests waiting at
every replica add one, a quiet spell takes it back. For load that comes by the
clock, a schedule says so ahead of time:
```yaml
scaling:
min: 1
max: 4
prewarm: 30m # start a window's replicas this long before it opens
schedule:
- {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
- {from: "22:00", to: "02:00", min: 1} # past midnight
```
Times are in `control.timezone` (`Europe/Istanbul`; empty is the control
plane's own). The first window that holds wins; outside every window, `min`
and `max` apply. An on-demand model (`min: 0`) with a window is kept loaded
through it, and loaded again on the first request after. Days are `mon` …
`sun`, `weekdays` or `weekends`.
How much traffic each replica is given is the profile's `weight`, and guessing
it is how a cluster ends up with one saturated replica beside two idle ones.
Measure it instead:
+63
View File
@@ -62,6 +62,7 @@ rather than repeated.
| `lost_after` | `time.Duration` | [Control](#control) |
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -215,9 +216,16 @@ rather than repeated.
| `python` | `string` | [SGLangConfig](#sglangconfig) |
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
| `from` | `string` | [ScaleWindow](#scalewindow) |
| `to` | `string` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [ScaleWindow](#scalewindow) |
| `max` | `int` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [Scaling](#scaling) |
| `max` | `int` | [Scaling](#scaling) |
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
| `target_waiting` | `float64` | [Scoring](#scoring) |
| `target_inflight` | `float64` | [Scoring](#scoring) |
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
@@ -641,6 +649,13 @@ PortRange is the per-node range for engine HTTP and dist-init ports.
MgmtInterface is the ethernet interface NCCL bootstraps over.
### `timezone`
`string`
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `enabled`
`bool`
@@ -1756,6 +1771,36 @@ TTFTp99 is the time to the first token 99 requests in 100 are within.
TPOTp99 is the time per output token, after the first, 99 streamed
requests in 100 are within.
## ScaleWindow
### `days`
`[]string`
Days are mon … sun, or weekdays, weekends; empty is every day.
### `from`
`string`
From and To are HH:MM; a window whose To is before its From runs
past midnight.
### `to`
`string`
### `min`
`int`
### `max`
`int`
Max, when set, is the window's ceiling for load scaling; zero keeps
the model's own.
## Scaling
### `min`
@@ -1778,6 +1823,24 @@ KV cache prefix routing spent its effort filling, and a model that flaps
in and out loses that entirely — so the saving has to be worth more than
the cache, which it only is over a real idle period.
### `schedule`
`[]ScaleWindow`
Schedule changes min and max by the time of day, in the control
plane's control.timezone: more replicas kept through working hours,
fewer at night, an on-demand model kept loaded while people are at
work. The first window that holds now wins; outside them all, Min and
Max apply.
### `prewarm`
`time.Duration`
Prewarm starts a window this long before it opens, so its replicas
have loaded by then: loading a model is minutes, and the people who
arrive at nine are the reason for the window.
## Scoring
### `target_waiting`