Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:16:56 +02:00
1 parent 74429f1c00
commit bce54f0941
14 files changed
+382 -9

No files matched your search

+63
View File
@@ -62,6 +62,7 @@ rather than repeated.
| `lost_after` | `time.Duration` | [Control](#control) |
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -215,9 +216,16 @@ rather than repeated.
| `python` | `string` | [SGLangConfig](#sglangconfig) |
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
| `from` | `string` | [ScaleWindow](#scalewindow) |
| `to` | `string` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [ScaleWindow](#scalewindow) |
| `max` | `int` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [Scaling](#scaling) |
| `max` | `int` | [Scaling](#scaling) |
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
| `target_waiting` | `float64` | [Scoring](#scoring) |
| `target_inflight` | `float64` | [Scoring](#scoring) |
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
@@ -641,6 +649,13 @@ PortRange is the per-node range for engine HTTP and dist-init ports.
MgmtInterface is the ethernet interface NCCL bootstraps over.
### `timezone`
`string`
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `enabled`
`bool`
@@ -1756,6 +1771,36 @@ TTFTp99 is the time to the first token 99 requests in 100 are within.
TPOTp99 is the time per output token, after the first, 99 streamed
requests in 100 are within.
## ScaleWindow
### `days`
`[]string`
Days are mon … sun, or weekdays, weekends; empty is every day.
### `from`
`string`
From and To are HH:MM; a window whose To is before its From runs
past midnight.
### `to`
`string`
### `min`
`int`
### `max`
`int`
Max, when set, is the window's ceiling for load scaling; zero keeps
the model's own.
## Scaling
### `min`
@@ -1778,6 +1823,24 @@ KV cache prefix routing spent its effort filling, and a model that flaps
in and out loses that entirely — so the saving has to be worth more than
the cache, which it only is over a real idle period.
### `schedule`
`[]ScaleWindow`
Schedule changes min and max by the time of day, in the control
plane's control.timezone: more replicas kept through working hours,
fewer at night, an on-demand model kept loaded while people are at
work. The first window that holds now wins; outside them all, Min and
Max apply.
### `prewarm`
`time.Duration`
Prewarm starts a window this long before it opens, so its replicas
have loaded by then: loading a model is minutes, and the people who
arrive at nine are the reason for the window.
## Scoring
### `target_waiting`