Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
74429f1c00
commit
bce54f0941
14 files changed
+382
-9
No files matched your search
@@ -62,6 +62,7 @@ rather than repeated.
|
||||
| `lost_after` | `time.Duration` | [Control](#control) |
|
||||
| `port_range` | `[]int` | [Control](#control) |
|
||||
| `mgmt_interface` | `string` | [Control](#control) |
|
||||
| `timezone` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -215,9 +216,16 @@ rather than repeated.
|
||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
|
||||
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
|
||||
| `from` | `string` | [ScaleWindow](#scalewindow) |
|
||||
| `to` | `string` | [ScaleWindow](#scalewindow) |
|
||||
| `min` | `int` | [ScaleWindow](#scalewindow) |
|
||||
| `max` | `int` | [ScaleWindow](#scalewindow) |
|
||||
| `min` | `int` | [Scaling](#scaling) |
|
||||
| `max` | `int` | [Scaling](#scaling) |
|
||||
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
|
||||
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
|
||||
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
|
||||
| `target_waiting` | `float64` | [Scoring](#scoring) |
|
||||
| `target_inflight` | `float64` | [Scoring](#scoring) |
|
||||
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
|
||||
@@ -641,6 +649,13 @@ PortRange is the per-node range for engine HTTP and dist-init ports.
|
||||
|
||||
MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
|
||||
### `timezone`
|
||||
|
||||
`string`
|
||||
|
||||
Timezone is where models' scaling schedules are read, an IANA name
|
||||
like Europe/Istanbul; empty is the control plane's own.
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
@@ -1756,6 +1771,36 @@ TTFTp99 is the time to the first token 99 requests in 100 are within.
|
||||
TPOTp99 is the time per output token, after the first, 99 streamed
|
||||
requests in 100 are within.
|
||||
|
||||
## ScaleWindow
|
||||
|
||||
### `days`
|
||||
|
||||
`[]string`
|
||||
|
||||
Days are mon … sun, or weekdays, weekends; empty is every day.
|
||||
|
||||
### `from`
|
||||
|
||||
`string`
|
||||
|
||||
From and To are HH:MM; a window whose To is before its From runs
|
||||
past midnight.
|
||||
|
||||
### `to`
|
||||
|
||||
`string`
|
||||
|
||||
### `min`
|
||||
|
||||
`int`
|
||||
|
||||
### `max`
|
||||
|
||||
`int`
|
||||
|
||||
Max, when set, is the window's ceiling for load scaling; zero keeps
|
||||
the model's own.
|
||||
|
||||
## Scaling
|
||||
|
||||
### `min`
|
||||
@@ -1778,6 +1823,24 @@ KV cache prefix routing spent its effort filling, and a model that flaps
|
||||
in and out loses that entirely — so the saving has to be worth more than
|
||||
the cache, which it only is over a real idle period.
|
||||
|
||||
### `schedule`
|
||||
|
||||
`[]ScaleWindow`
|
||||
|
||||
Schedule changes min and max by the time of day, in the control
|
||||
plane's control.timezone: more replicas kept through working hours,
|
||||
fewer at night, an on-demand model kept loaded while people are at
|
||||
work. The first window that holds now wins; outside them all, Min and
|
||||
Max apply.
|
||||
|
||||
### `prewarm`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Prewarm starts a window this long before it opens, so its replicas
|
||||
have loaded by then: loading a model is minutes, and the people who
|
||||
arrive at nine are the reason for the window.
|
||||
|
||||
## Scoring
|
||||
|
||||
### `target_waiting`
|
||||
|
||||
Reference in new issue
Block a user