Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 23:16:56 +02:00
1 parent 74429f1c00
commit bce54f0941
14 files changed
+382 -9

No files matched your search

+6 -1
View File
@@ -482,7 +482,12 @@ portal is not done — the page shows; the console and ctl change.
**SLO-based admission and token-level metrics (S–M).** TPOT and
inter-token histograms, output tokens/s per model and tenant in
`router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
14. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
min, max; past midnight), `scaling.prewarm`, `control.timezone`;
queue-depth scaling was there already. Left: `ctl scale` sets min and
max only (a schedule is set in the definition); scaling for an SLO
rather than for queue depth.
**Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
pre-warm, queue-depth scaling in `catalog.Scaling`,
`control/reconciler/demand.go`, `headroom.go`.
15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
+8
View File
@@ -71,6 +71,14 @@ APIs change between minors without a deprecation period.
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
and **E** on the console's Usage view.
### Scaling by the clock
- **`scaling.schedule`**: windows of the week with their own `min` and
`max` — more replicas through working hours, an on-demand model kept
loaded while people are at work — in `control.timezone`.
- **`scaling.prewarm`** opens a window early, so its replicas have loaded
when it starts.
### Token-level timing, and admission by SLO
- **Per-token metrics:** `llmgw_tpot_seconds`, `llmgw_inter_token_seconds`
+2
View File
@@ -5,6 +5,8 @@ package main
import (
"fmt"
"os"
// Time zones for scaling schedules, on a machine without the tz database.
_ "time/tzdata"
"gitea.mynodes.xyz/ilker/llmgw/internal/cli"
)
+20
View File
@@ -924,6 +924,26 @@ Models added at runtime only. One declared in `server.yaml` is read again on
every start, so a number set here would hold until the next restart and then
revert — change it in the file instead.
Between `min` and `max` a model scales with its load: requests waiting at
every replica add one, a quiet spell takes it back. For load that comes by the
clock, a schedule says so ahead of time:
```yaml
scaling:
min: 1
max: 4
prewarm: 30m # start a window's replicas this long before it opens
schedule:
- {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
- {from: "22:00", to: "02:00", min: 1} # past midnight
```
Times are in `control.timezone` (`Europe/Istanbul`; empty is the control
plane's own). The first window that holds wins; outside every window, `min`
and `max` apply. An on-demand model (`min: 0`) with a window is kept loaded
through it, and loaded again on the first request after. Days are `mon` …
`sun`, `weekdays` or `weekends`.
How much traffic each replica is given is the profile's `weight`, and guessing
it is how a cluster ends up with one saturated replica beside two idle ones.
Measure it instead:
+63
View File
@@ -62,6 +62,7 @@ rather than repeated.
| `lost_after` | `time.Duration` | [Control](#control) |
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -215,9 +216,16 @@ rather than repeated.
| `python` | `string` | [SGLangConfig](#sglangconfig) |
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
| `from` | `string` | [ScaleWindow](#scalewindow) |
| `to` | `string` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [ScaleWindow](#scalewindow) |
| `max` | `int` | [ScaleWindow](#scalewindow) |
| `min` | `int` | [Scaling](#scaling) |
| `max` | `int` | [Scaling](#scaling) |
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
| `target_waiting` | `float64` | [Scoring](#scoring) |
| `target_inflight` | `float64` | [Scoring](#scoring) |
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
@@ -641,6 +649,13 @@ PortRange is the per-node range for engine HTTP and dist-init ports.
MgmtInterface is the ethernet interface NCCL bootstraps over.
### `timezone`
`string`
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `enabled`
`bool`
@@ -1756,6 +1771,36 @@ TTFTp99 is the time to the first token 99 requests in 100 are within.
TPOTp99 is the time per output token, after the first, 99 streamed
requests in 100 are within.
## ScaleWindow
### `days`
`[]string`
Days are mon … sun, or weekdays, weekends; empty is every day.
### `from`
`string`
From and To are HH:MM; a window whose To is before its From runs
past midnight.
### `to`
`string`
### `min`
`int`
### `max`
`int`
Max, when set, is the window's ceiling for load scaling; zero keeps
the model's own.
## Scaling
### `min`
@@ -1778,6 +1823,24 @@ KV cache prefix routing spent its effort filling, and a model that flaps
in and out loses that entirely — so the saving has to be worth more than
the cache, which it only is over a real idle period.
### `schedule`
`[]ScaleWindow`
Schedule changes min and max by the time of day, in the control
plane's control.timezone: more replicas kept through working hours,
fewer at night, an on-demand model kept loaded while people are at
work. The first window that holds now wins; outside them all, Min and
Max apply.
### `prewarm`
`time.Duration`
Prewarm starts a window this long before it opens, so its replicas
have loaded by then: loading a model is minutes, and the people who
arrive at nine are the reason for the window.
## Scoring
### `target_waiting`
+7
View File
@@ -597,6 +597,13 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
LostAfter: cfg.Control.LostAfter,
MgmtInterface: cfg.Control.MgmtInterface,
}
if cfg.Control.Timezone != "" {
loc, err := time.LoadLocation(cfg.Control.Timezone)
if err != nil {
return nil, nil, fmt.Errorf("control.timezone: %w", err)
}
ccfg.Reconciler.Location = loc
}
if len(cfg.Control.PortRange) == 2 {
ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
}
+8
View File
@@ -384,6 +384,9 @@ type Control struct {
PortRange []int `koanf:"port_range"`
// MgmtInterface is the ethernet interface NCCL bootstraps over.
MgmtInterface string `koanf:"mgmt_interface"`
// Timezone is where models' scaling schedules are read, an IANA name
// like Europe/Istanbul; empty is the control plane's own.
Timezone string `koanf:"timezone"`
// Enabled runs the control plane even when configuration declares no
// models.
@@ -693,6 +696,11 @@ func (s Server) Validate() error {
if s.Listen == "" {
return fmt.Errorf("listen is empty")
}
if s.Control.Timezone != "" {
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
}
}
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
return err
}
+119
View File
@@ -28,6 +28,117 @@ type Scaling struct {
// in and out loses that entirely — so the saving has to be worth more than
// the cache, which it only is over a real idle period.
IdleTimeout time.Duration `json:"idle_timeout,omitempty" koanf:"idle_timeout"`
// Schedule changes min and max by the time of day, in the control
// plane's control.timezone: more replicas kept through working hours,
// fewer at night, an on-demand model kept loaded while people are at
// work. The first window that holds now wins; outside them all, Min and
// Max apply.
Schedule []ScaleWindow `json:"schedule,omitempty" koanf:"schedule"`
// Prewarm starts a window this long before it opens, so its replicas
// have loaded by then: loading a model is minutes, and the people who
// arrive at nine are the reason for the window.
Prewarm time.Duration `json:"prewarm,omitempty" koanf:"prewarm"`
}
// ScaleWindow is a part of the week with its own min and max.
type ScaleWindow struct {
// Days are mon … sun, or weekdays, weekends; empty is every day.
Days []string `json:"days,omitempty" koanf:"days"`
// From and To are HH:MM; a window whose To is before its From runs
// past midnight.
From string `json:"from" koanf:"from"`
To string `json:"to" koanf:"to"`
Min int `json:"min" koanf:"min"`
// Max, when set, is the window's ceiling for load scaling; zero keeps
// the model's own.
Max int `json:"max,omitempty" koanf:"max"`
}
// clock parses HH:MM into minutes past midnight.
func clock(s string) (int, bool) {
var h, m int
if n, err := fmt.Sscanf(s, "%d:%d", &h, &m); err != nil || n != 2 || h < 0 || h > 23 || m < 0 || m > 59 || len(s) != 5 {
return 0, false
}
return h*60 + m, true
}
var weekdays = map[string][]time.Weekday{
"mon": {time.Monday}, "tue": {time.Tuesday}, "wed": {time.Wednesday}, "thu": {time.Thursday},
"fri": {time.Friday}, "sat": {time.Saturday}, "sun": {time.Sunday},
"weekdays": {time.Monday, time.Tuesday, time.Wednesday, time.Thursday, time.Friday},
"weekends": {time.Saturday, time.Sunday},
}
func (w ScaleWindow) validate() error {
from, ok1 := clock(w.From)
to, ok2 := clock(w.To)
if !ok1 || !ok2 {
return fmt.Errorf("from and to are HH:MM, like 08:00")
}
if from == to {
return fmt.Errorf("from and to are the same time")
}
for _, d := range w.Days {
if _, ok := weekdays[strings.ToLower(d)]; !ok {
return fmt.Errorf("day %q is not mon … sun, weekdays or weekends", d)
}
}
if w.Min < 0 || w.Max < 0 || w.Max > 0 && w.Max < w.Min {
return fmt.Errorf("min %d and max %d: max is 0 or at least min", w.Min, w.Max)
}
return nil
}
// holds says whether t falls in the window, opened early by lead.
func (w ScaleWindow) holds(t time.Time, lead time.Duration) bool {
from, _ := clock(w.From)
to, _ := clock(w.To)
span := to - from
if span <= 0 {
span += 24 * 60 // past midnight
}
midnight := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
// The window as it starts yesterday, today and tomorrow — tomorrow's
// can open today, pulled forward by the lead.
for d := -1; d <= 1; d++ {
day := midnight.AddDate(0, 0, d)
if !w.on(day.Weekday()) {
continue
}
start := day.Add(time.Duration(from)*time.Minute - lead)
end := day.Add(time.Duration(from+span) * time.Minute)
if !t.Before(start) && t.Before(end) {
return true
}
}
return false
}
// on says whether the window starts on a day.
func (w ScaleWindow) on(wd time.Weekday) bool {
if len(w.Days) == 0 {
return true
}
for _, d := range w.Days {
for _, x := range weekdays[strings.ToLower(d)] {
if x == wd {
return true
}
}
}
return false
}
// Window is the window in force at t, in its own time zone, and false
// outside them all.
func (s Scaling) Window(t time.Time) (ScaleWindow, bool) {
for _, w := range s.Schedule {
if w.holds(t, s.Prewarm) {
return w, true
}
}
return ScaleWindow{}, false
}
// OnDemand reports whether this model is loaded when someone asks for it.
@@ -547,6 +658,14 @@ func (m Model) validate() error {
default:
return fmt.Errorf("model %q: role %q is not guard", m.Name, m.Role)
}
for i, w := range m.Scaling.Schedule {
if err := w.validate(); err != nil {
return fmt.Errorf("model %q: scaling.schedule[%d]: %w", m.Name, i, err)
}
}
if m.Scaling.Prewarm < 0 || m.Scaling.Prewarm > 12*time.Hour {
return fmt.Errorf("model %q: scaling.prewarm is between 0 and 12h", m.Name)
}
if m.Scaling.Max < m.Scaling.Min {
return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
}
+49
View File
@@ -0,0 +1,49 @@
package catalog
import (
"testing"
"time"
)
func TestAScheduleWindowHolds(t *testing.T) {
ist := time.FixedZone("TRT", 3*3600)
at := func(day, hm string) time.Time {
d, _ := time.ParseInLocation("2006-01-02 15:04", day+" "+hm, ist)
return d
}
// 2026-10-05 is a Monday.
work := Scaling{Schedule: []ScaleWindow{{Days: []string{"weekdays"}, From: "08:00", To: "18:00", Min: 2}},
Prewarm: 30 * time.Minute}
night := Scaling{Schedule: []ScaleWindow{{From: "22:00", To: "02:00", Min: 1}}}
for _, c := range []struct {
s Scaling
t time.Time
want bool
}{
{work, at("2026-10-05", "09:00"), true},
{work, at("2026-10-05", "07:40"), true}, // prewarm
{work, at("2026-10-05", "07:20"), false},
{work, at("2026-10-05", "18:00"), false},
{work, at("2026-10-04", "10:00"), false}, // Sunday
{work, at("2026-10-04", "23:50"), false},
{night, at("2026-10-05", "23:00"), true},
{night, at("2026-10-06", "01:30"), true}, // after midnight, from yesterday's start
{night, at("2026-10-06", "02:00"), false},
{night, at("2026-10-06", "12:00"), false},
} {
if _, got := c.s.Window(c.t); got != c.want {
t.Errorf("%v at %s: %v, want %v", c.s.Schedule[0], c.t.Format("Mon 15:04"), got, c.want)
}
}
// Opened the evening before by its lead.
early := Scaling{Schedule: []ScaleWindow{{Days: []string{"tue"}, From: "00:15", To: "06:00", Min: 1}}, Prewarm: time.Hour}
if _, ok := early.Window(at("2026-10-05", "23:30")); !ok {
t.Error("a window after midnight was not prewarmed the evening before")
}
for _, bad := range []ScaleWindow{{From: "8:00", To: "18:00"}, {From: "08:00", To: "08:00"},
{From: "08:00", To: "18:00", Days: []string{"monday"}}, {From: "08:00", To: "18:00", Min: 3, Max: 2}} {
if bad.validate() == nil {
t.Errorf("%+v accepted", bad)
}
}
}
+33 -6
View File
@@ -88,17 +88,43 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
r.mu.Lock()
extra := r.extra[m.Name]
r.mu.Unlock()
return min(base+extra, max(m.Scaling.Max, base))
return min(base+extra, max(r.maxFor(m), base))
}
// minFor is a model's minimum now: its schedule's window's, or its own
// outside every window.
func (r *Reconciler) minFor(m catalog.Model) int {
if w, ok := m.Scaling.Window(r.localNow()); ok {
return w.Min
}
return m.Scaling.Min
}
// maxFor is the ceiling load scaling reaches now.
func (r *Reconciler) maxFor(m catalog.Model) int {
if w, ok := m.Scaling.Window(r.localNow()); ok && w.Max > 0 {
return w.Max
}
return m.Scaling.Max
}
// localNow is now in the time zone schedules are written in.
func (r *Reconciler) localNow() time.Time {
if r.cfg.Location != nil {
return r.now().In(r.cfg.Location)
}
return r.now()
}
// baseFor is what a model runs without load: its minimum, or for an on-demand
// model one while somebody is asking and none once they stop.
// model one while somebody is asking and none once they stop. A schedule's
// window keeps its minimum running however quiet it is.
func (r *Reconciler) baseFor(m catalog.Model) int {
if m.IsExternal() {
return 0 // served elsewhere: nothing to place, whatever its scaling says
}
if !m.Scaling.OnDemand() {
return m.Scaling.Min
if lo := r.minFor(m); !m.Scaling.OnDemand() || lo > 0 {
return lo
}
r.mu.Lock()
last, asked := r.wanted[m.Name]
@@ -188,7 +214,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
for _, m := range r.cat.Load().Models() {
base := r.baseFor(m)
r.mu.Lock()
if base == 0 || m.Scaling.Max <= base {
ceiling := r.maxFor(m)
if base == 0 || ceiling <= base {
delete(r.extra, m.Name)
delete(r.busySince, m.Name)
delete(r.quietSince, m.Name)
@@ -217,7 +244,7 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
r.busySince[m.Name] = now
break
}
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < ceiling && !starting {
r.extra[m.Name] = extra + 1
r.busySince[m.Name] = now
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
+2 -2
View File
@@ -196,7 +196,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
cat := r.cat.Load()
now := r.now()
myRank := catalog.ClassRank(m.EffectiveClass())
belowMin := r.liveCount(m.Name) < m.Scaling.Min
belowMin := r.liveCount(m.Name) < r.minFor(m)
pol := r.policy()
var borrowed map[string]bool
@@ -229,7 +229,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
class := other.EffectiveClass()
rank := catalog.ClassRank(class)
sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
above := max(len(reps)-other.Scaling.Min, 0)
above := max(len(reps)-r.minFor(other), 0)
for i, rep := range reps {
aboveMin := i < above
tier := -1
@@ -50,6 +50,9 @@ type Dispatcher interface {
// Config tunes the reconciliation loop.
type Config struct {
// Location is the time zone models' scaling schedules are written in;
// nil is the control plane's own.
Location *time.Location
// Interval is how often desired and actual are compared when nothing else
// happens. Events wake the loop sooner.
Interval time.Duration
@@ -0,0 +1,52 @@
package reconciler
import (
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
func TestAScheduleRaisesTheMinimumThroughItsHours(t *testing.T) {
m := testModel(1)
m.Scaling = catalog.Scaling{Min: 1, Max: 4, Prewarm: 30 * time.Minute,
Schedule: []catalog.ScaleWindow{{From: "08:00", To: "18:00", Min: 2, Max: 3}}}
h := newHarness(t, m, gpuNode("1", 8))
at := func(hm string) {
target, _ := time.Parse("2006-01-02 15:04", "2026-01-02 "+hm)
h.clock.advance(target.Sub(h.clock.now()))
}
cur := func() catalog.Model { got, _ := h.live.Load().Resolve(m.Name); return got }
at("03:00")
if n := h.r.desiredFor(cur()); n != 1 {
t.Errorf("at night: %d", n)
}
at("07:45") // prewarmed
if n := h.r.desiredFor(cur()); n != 2 {
t.Errorf("before the window, prewarming: %d", n)
}
if mx := h.r.maxFor(cur()); mx != 3 {
t.Errorf("the window's ceiling: %d", mx)
}
at("18:30")
if n := h.r.desiredFor(cur()); n != 1 {
t.Errorf("after the window: %d", n)
}
}
func TestAnOnDemandModelIsKeptLoadedThroughItsWindow(t *testing.T) {
m := testModel(1)
m.Scaling = catalog.Scaling{Min: 0, Max: 1,
Schedule: []catalog.ScaleWindow{{From: "09:00", To: "17:00", Min: 1}}}
h := newHarness(t, m, gpuNode("1", 8))
target, _ := time.Parse("2006-01-02 15:04", "2026-01-02 10:00")
h.clock.advance(target.Sub(h.clock.now()))
got, _ := h.live.Load().Resolve(m.Name)
if n := h.r.desiredFor(got); n != 1 {
t.Errorf("unasked-for in its window: %d", n)
}
h.clock.advance(8 * time.Hour)
if n := h.r.desiredFor(got); n != 0 {
t.Errorf("unasked-for outside its window: %d", n)
}
}
+10
View File
@@ -64,6 +64,7 @@ var Reference = []Block{
{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
{Key: "timezone", Type: "string", Doc: "Timezone is where models' scaling schedules are read, an IANA name\nlike Europe/Istanbul; empty is the control plane's own.", Nested: ""},
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
}},
@@ -268,10 +269,19 @@ var Reference = []Block{
{Key: "ttft_p99", Type: "time.Duration", Doc: "TTFTp99 is the time to the first token 99 requests in 100 are within.", Nested: ""},
{Key: "tpot_p99", Type: "time.Duration", Doc: "TPOTp99 is the time per output token, after the first, 99 streamed\nrequests in 100 are within.", Nested: ""},
}},
{Package: "catalog", Name: "ScaleWindow", Title: "ScaleWindow", Doc: "", Fields: []Field{
{Key: "days", Type: "[]string", Doc: "Days are mon … sun, or weekdays, weekends; empty is every day.", Nested: ""},
{Key: "from", Type: "string", Doc: "From and To are HH:MM; a window whose To is before its From runs\npast midnight.", Nested: ""},
{Key: "to", Type: "string", Doc: "", Nested: ""},
{Key: "min", Type: "int", Doc: "", Nested: ""},
{Key: "max", Type: "int", Doc: "Max, when set, is the window's ceiling for load scaling; zero keeps\nthe model's own.", Nested: ""},
}},
{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
{Key: "min", Type: "int", Doc: "", Nested: ""},
{Key: "max", Type: "int", Doc: "", Nested: ""},
{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
{Key: "schedule", Type: "[]ScaleWindow", Doc: "Schedule changes min and max by the time of day, in the control\nplane's control.timezone: more replicas kept through working hours,\nfewer at night, an on-demand model kept loaded while people are at\nwork. The first window that holds now wins; outside them all, Min and\nMax apply.", Nested: ""},
{Key: "prewarm", Type: "time.Duration", Doc: "Prewarm starts a window this long before it opens, so its replicas\nhave loaded by then: loading a model is minutes, and the people who\narrive at nine are the reason for the window.", Nested: ""},
}},
{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},