Scaling by the clock: schedule windows with their own min and max, prewarm, control.timezone
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
74429f1c00
commit
bce54f0941
14 files changed
+382
-9
No files matched your search
+6
-1
@@ -482,7 +482,12 @@ portal is not done — the page shows; the console and ctl change.
|
||||
**SLO-based admission and token-level metrics (S–M).** TPOT and
|
||||
inter-token histograms, output tokens/s per model and tenant in
|
||||
`router/metrics.go`; use `catalog.SLO` in `admission/saturation.go`.
|
||||
14. **Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
|
||||
14. **Done (branch `features-c2`):** `scaling.schedule` (days, from, to,
|
||||
min, max; past midnight), `scaling.prewarm`, `control.timezone`;
|
||||
queue-depth scaling was there already. Left: `ctl scale` sets min and
|
||||
max only (a schedule is set in the definition); scaling for an SLO
|
||||
rather than for queue depth.
|
||||
**Scheduled and warm-pool autoscaling (S–M).** Time-of-day min/max,
|
||||
pre-warm, queue-depth scaling in `catalog.Scaling`,
|
||||
`control/reconciler/demand.go`, `headroom.go`.
|
||||
15. **Secrets management (S–M).** Vault/KMS for `api_key_file`, the DSN, the
|
||||
|
||||
@@ -71,6 +71,14 @@ APIs change between minors without a deprecation period.
|
||||
- **Other ways in:** `GET /api/v1/usage/export` (CSV, or `format=json`)
|
||||
and **E** on the console's Usage view.
|
||||
|
||||
### Scaling by the clock
|
||||
|
||||
- **`scaling.schedule`**: windows of the week with their own `min` and
|
||||
`max` — more replicas through working hours, an on-demand model kept
|
||||
loaded while people are at work — in `control.timezone`.
|
||||
- **`scaling.prewarm`** opens a window early, so its replicas have loaded
|
||||
when it starts.
|
||||
|
||||
### Token-level timing, and admission by SLO
|
||||
|
||||
- **Per-token metrics:** `llmgw_tpot_seconds`, `llmgw_inter_token_seconds`
|
||||
|
||||
@@ -5,6 +5,8 @@ package main
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
// Time zones for scaling schedules, on a machine without the tz database.
|
||||
_ "time/tzdata"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/cli"
|
||||
)
|
||||
|
||||
@@ -924,6 +924,26 @@ Models added at runtime only. One declared in `server.yaml` is read again on
|
||||
every start, so a number set here would hold until the next restart and then
|
||||
revert — change it in the file instead.
|
||||
|
||||
Between `min` and `max` a model scales with its load: requests waiting at
|
||||
every replica add one, a quiet spell takes it back. For load that comes by the
|
||||
clock, a schedule says so ahead of time:
|
||||
|
||||
```yaml
|
||||
scaling:
|
||||
min: 1
|
||||
max: 4
|
||||
prewarm: 30m # start a window's replicas this long before it opens
|
||||
schedule:
|
||||
- {days: [weekdays], from: "08:00", to: "18:00", min: 3, max: 6}
|
||||
- {from: "22:00", to: "02:00", min: 1} # past midnight
|
||||
```
|
||||
|
||||
Times are in `control.timezone` (`Europe/Istanbul`; empty is the control
|
||||
plane's own). The first window that holds wins; outside every window, `min`
|
||||
and `max` apply. An on-demand model (`min: 0`) with a window is kept loaded
|
||||
through it, and loaded again on the first request after. Days are `mon` …
|
||||
`sun`, `weekdays` or `weekends`.
|
||||
|
||||
How much traffic each replica is given is the profile's `weight`, and guessing
|
||||
it is how a cluster ends up with one saturated replica beside two idle ones.
|
||||
Measure it instead:
|
||||
|
||||
@@ -62,6 +62,7 @@ rather than repeated.
|
||||
| `lost_after` | `time.Duration` | [Control](#control) |
|
||||
| `port_range` | `[]int` | [Control](#control) |
|
||||
| `mgmt_interface` | `string` | [Control](#control) |
|
||||
| `timezone` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -215,9 +216,16 @@ rather than repeated.
|
||||
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
||||
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
||||
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
|
||||
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
|
||||
| `from` | `string` | [ScaleWindow](#scalewindow) |
|
||||
| `to` | `string` | [ScaleWindow](#scalewindow) |
|
||||
| `min` | `int` | [ScaleWindow](#scalewindow) |
|
||||
| `max` | `int` | [ScaleWindow](#scalewindow) |
|
||||
| `min` | `int` | [Scaling](#scaling) |
|
||||
| `max` | `int` | [Scaling](#scaling) |
|
||||
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
|
||||
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
|
||||
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
|
||||
| `target_waiting` | `float64` | [Scoring](#scoring) |
|
||||
| `target_inflight` | `float64` | [Scoring](#scoring) |
|
||||
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
|
||||
@@ -641,6 +649,13 @@ PortRange is the per-node range for engine HTTP and dist-init ports.
|
||||
|
||||
MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
|
||||
### `timezone`
|
||||
|
||||
`string`
|
||||
|
||||
Timezone is where models' scaling schedules are read, an IANA name
|
||||
like Europe/Istanbul; empty is the control plane's own.
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
@@ -1756,6 +1771,36 @@ TTFTp99 is the time to the first token 99 requests in 100 are within.
|
||||
TPOTp99 is the time per output token, after the first, 99 streamed
|
||||
requests in 100 are within.
|
||||
|
||||
## ScaleWindow
|
||||
|
||||
### `days`
|
||||
|
||||
`[]string`
|
||||
|
||||
Days are mon … sun, or weekdays, weekends; empty is every day.
|
||||
|
||||
### `from`
|
||||
|
||||
`string`
|
||||
|
||||
From and To are HH:MM; a window whose To is before its From runs
|
||||
past midnight.
|
||||
|
||||
### `to`
|
||||
|
||||
`string`
|
||||
|
||||
### `min`
|
||||
|
||||
`int`
|
||||
|
||||
### `max`
|
||||
|
||||
`int`
|
||||
|
||||
Max, when set, is the window's ceiling for load scaling; zero keeps
|
||||
the model's own.
|
||||
|
||||
## Scaling
|
||||
|
||||
### `min`
|
||||
@@ -1778,6 +1823,24 @@ KV cache prefix routing spent its effort filling, and a model that flaps
|
||||
in and out loses that entirely — so the saving has to be worth more than
|
||||
the cache, which it only is over a real idle period.
|
||||
|
||||
### `schedule`
|
||||
|
||||
`[]ScaleWindow`
|
||||
|
||||
Schedule changes min and max by the time of day, in the control
|
||||
plane's control.timezone: more replicas kept through working hours,
|
||||
fewer at night, an on-demand model kept loaded while people are at
|
||||
work. The first window that holds now wins; outside them all, Min and
|
||||
Max apply.
|
||||
|
||||
### `prewarm`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
Prewarm starts a window this long before it opens, so its replicas
|
||||
have loaded by then: loading a model is minutes, and the people who
|
||||
arrive at nine are the reason for the window.
|
||||
|
||||
## Scoring
|
||||
|
||||
### `target_waiting`
|
||||
|
||||
@@ -597,6 +597,13 @@ func startControlPlane(ctx context.Context, cfg config.Server, cat *catalog.Live
|
||||
LostAfter: cfg.Control.LostAfter,
|
||||
MgmtInterface: cfg.Control.MgmtInterface,
|
||||
}
|
||||
if cfg.Control.Timezone != "" {
|
||||
loc, err := time.LoadLocation(cfg.Control.Timezone)
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("control.timezone: %w", err)
|
||||
}
|
||||
ccfg.Reconciler.Location = loc
|
||||
}
|
||||
if len(cfg.Control.PortRange) == 2 {
|
||||
ccfg.Reconciler.PortRange = [2]int{cfg.Control.PortRange[0], cfg.Control.PortRange[1]}
|
||||
}
|
||||
|
||||
@@ -384,6 +384,9 @@ type Control struct {
|
||||
PortRange []int `koanf:"port_range"`
|
||||
// MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
MgmtInterface string `koanf:"mgmt_interface"`
|
||||
// Timezone is where models' scaling schedules are read, an IANA name
|
||||
// like Europe/Istanbul; empty is the control plane's own.
|
||||
Timezone string `koanf:"timezone"`
|
||||
|
||||
// Enabled runs the control plane even when configuration declares no
|
||||
// models.
|
||||
@@ -693,6 +696,11 @@ func (s Server) Validate() error {
|
||||
if s.Listen == "" {
|
||||
return fmt.Errorf("listen is empty")
|
||||
}
|
||||
if s.Control.Timezone != "" {
|
||||
if _, err := time.LoadLocation(s.Control.Timezone); err != nil {
|
||||
return fmt.Errorf("control.timezone %q: %w", s.Control.Timezone, err)
|
||||
}
|
||||
}
|
||||
if _, err := ParseBackupRecipients(s.BackupRecipients); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -28,6 +28,117 @@ type Scaling struct {
|
||||
// in and out loses that entirely — so the saving has to be worth more than
|
||||
// the cache, which it only is over a real idle period.
|
||||
IdleTimeout time.Duration `json:"idle_timeout,omitempty" koanf:"idle_timeout"`
|
||||
// Schedule changes min and max by the time of day, in the control
|
||||
// plane's control.timezone: more replicas kept through working hours,
|
||||
// fewer at night, an on-demand model kept loaded while people are at
|
||||
// work. The first window that holds now wins; outside them all, Min and
|
||||
// Max apply.
|
||||
Schedule []ScaleWindow `json:"schedule,omitempty" koanf:"schedule"`
|
||||
// Prewarm starts a window this long before it opens, so its replicas
|
||||
// have loaded by then: loading a model is minutes, and the people who
|
||||
// arrive at nine are the reason for the window.
|
||||
Prewarm time.Duration `json:"prewarm,omitempty" koanf:"prewarm"`
|
||||
}
|
||||
|
||||
// ScaleWindow is a part of the week with its own min and max.
|
||||
type ScaleWindow struct {
|
||||
// Days are mon … sun, or weekdays, weekends; empty is every day.
|
||||
Days []string `json:"days,omitempty" koanf:"days"`
|
||||
// From and To are HH:MM; a window whose To is before its From runs
|
||||
// past midnight.
|
||||
From string `json:"from" koanf:"from"`
|
||||
To string `json:"to" koanf:"to"`
|
||||
Min int `json:"min" koanf:"min"`
|
||||
// Max, when set, is the window's ceiling for load scaling; zero keeps
|
||||
// the model's own.
|
||||
Max int `json:"max,omitempty" koanf:"max"`
|
||||
}
|
||||
|
||||
// clock parses HH:MM into minutes past midnight.
|
||||
func clock(s string) (int, bool) {
|
||||
var h, m int
|
||||
if n, err := fmt.Sscanf(s, "%d:%d", &h, &m); err != nil || n != 2 || h < 0 || h > 23 || m < 0 || m > 59 || len(s) != 5 {
|
||||
return 0, false
|
||||
}
|
||||
return h*60 + m, true
|
||||
}
|
||||
|
||||
var weekdays = map[string][]time.Weekday{
|
||||
"mon": {time.Monday}, "tue": {time.Tuesday}, "wed": {time.Wednesday}, "thu": {time.Thursday},
|
||||
"fri": {time.Friday}, "sat": {time.Saturday}, "sun": {time.Sunday},
|
||||
"weekdays": {time.Monday, time.Tuesday, time.Wednesday, time.Thursday, time.Friday},
|
||||
"weekends": {time.Saturday, time.Sunday},
|
||||
}
|
||||
|
||||
func (w ScaleWindow) validate() error {
|
||||
from, ok1 := clock(w.From)
|
||||
to, ok2 := clock(w.To)
|
||||
if !ok1 || !ok2 {
|
||||
return fmt.Errorf("from and to are HH:MM, like 08:00")
|
||||
}
|
||||
if from == to {
|
||||
return fmt.Errorf("from and to are the same time")
|
||||
}
|
||||
for _, d := range w.Days {
|
||||
if _, ok := weekdays[strings.ToLower(d)]; !ok {
|
||||
return fmt.Errorf("day %q is not mon … sun, weekdays or weekends", d)
|
||||
}
|
||||
}
|
||||
if w.Min < 0 || w.Max < 0 || w.Max > 0 && w.Max < w.Min {
|
||||
return fmt.Errorf("min %d and max %d: max is 0 or at least min", w.Min, w.Max)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// holds says whether t falls in the window, opened early by lead.
|
||||
func (w ScaleWindow) holds(t time.Time, lead time.Duration) bool {
|
||||
from, _ := clock(w.From)
|
||||
to, _ := clock(w.To)
|
||||
span := to - from
|
||||
if span <= 0 {
|
||||
span += 24 * 60 // past midnight
|
||||
}
|
||||
midnight := time.Date(t.Year(), t.Month(), t.Day(), 0, 0, 0, 0, t.Location())
|
||||
// The window as it starts yesterday, today and tomorrow — tomorrow's
|
||||
// can open today, pulled forward by the lead.
|
||||
for d := -1; d <= 1; d++ {
|
||||
day := midnight.AddDate(0, 0, d)
|
||||
if !w.on(day.Weekday()) {
|
||||
continue
|
||||
}
|
||||
start := day.Add(time.Duration(from)*time.Minute - lead)
|
||||
end := day.Add(time.Duration(from+span) * time.Minute)
|
||||
if !t.Before(start) && t.Before(end) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// on says whether the window starts on a day.
|
||||
func (w ScaleWindow) on(wd time.Weekday) bool {
|
||||
if len(w.Days) == 0 {
|
||||
return true
|
||||
}
|
||||
for _, d := range w.Days {
|
||||
for _, x := range weekdays[strings.ToLower(d)] {
|
||||
if x == wd {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Window is the window in force at t, in its own time zone, and false
|
||||
// outside them all.
|
||||
func (s Scaling) Window(t time.Time) (ScaleWindow, bool) {
|
||||
for _, w := range s.Schedule {
|
||||
if w.holds(t, s.Prewarm) {
|
||||
return w, true
|
||||
}
|
||||
}
|
||||
return ScaleWindow{}, false
|
||||
}
|
||||
|
||||
// OnDemand reports whether this model is loaded when someone asks for it.
|
||||
@@ -547,6 +658,14 @@ func (m Model) validate() error {
|
||||
default:
|
||||
return fmt.Errorf("model %q: role %q is not guard", m.Name, m.Role)
|
||||
}
|
||||
for i, w := range m.Scaling.Schedule {
|
||||
if err := w.validate(); err != nil {
|
||||
return fmt.Errorf("model %q: scaling.schedule[%d]: %w", m.Name, i, err)
|
||||
}
|
||||
}
|
||||
if m.Scaling.Prewarm < 0 || m.Scaling.Prewarm > 12*time.Hour {
|
||||
return fmt.Errorf("model %q: scaling.prewarm is between 0 and 12h", m.Name)
|
||||
}
|
||||
if m.Scaling.Max < m.Scaling.Min {
|
||||
return fmt.Errorf("model %q: scaling max %d is below min %d", m.Name, m.Scaling.Max, m.Scaling.Min)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestAScheduleWindowHolds(t *testing.T) {
|
||||
ist := time.FixedZone("TRT", 3*3600)
|
||||
at := func(day, hm string) time.Time {
|
||||
d, _ := time.ParseInLocation("2006-01-02 15:04", day+" "+hm, ist)
|
||||
return d
|
||||
}
|
||||
// 2026-10-05 is a Monday.
|
||||
work := Scaling{Schedule: []ScaleWindow{{Days: []string{"weekdays"}, From: "08:00", To: "18:00", Min: 2}},
|
||||
Prewarm: 30 * time.Minute}
|
||||
night := Scaling{Schedule: []ScaleWindow{{From: "22:00", To: "02:00", Min: 1}}}
|
||||
for _, c := range []struct {
|
||||
s Scaling
|
||||
t time.Time
|
||||
want bool
|
||||
}{
|
||||
{work, at("2026-10-05", "09:00"), true},
|
||||
{work, at("2026-10-05", "07:40"), true}, // prewarm
|
||||
{work, at("2026-10-05", "07:20"), false},
|
||||
{work, at("2026-10-05", "18:00"), false},
|
||||
{work, at("2026-10-04", "10:00"), false}, // Sunday
|
||||
{work, at("2026-10-04", "23:50"), false},
|
||||
{night, at("2026-10-05", "23:00"), true},
|
||||
{night, at("2026-10-06", "01:30"), true}, // after midnight, from yesterday's start
|
||||
{night, at("2026-10-06", "02:00"), false},
|
||||
{night, at("2026-10-06", "12:00"), false},
|
||||
} {
|
||||
if _, got := c.s.Window(c.t); got != c.want {
|
||||
t.Errorf("%v at %s: %v, want %v", c.s.Schedule[0], c.t.Format("Mon 15:04"), got, c.want)
|
||||
}
|
||||
}
|
||||
// Opened the evening before by its lead.
|
||||
early := Scaling{Schedule: []ScaleWindow{{Days: []string{"tue"}, From: "00:15", To: "06:00", Min: 1}}, Prewarm: time.Hour}
|
||||
if _, ok := early.Window(at("2026-10-05", "23:30")); !ok {
|
||||
t.Error("a window after midnight was not prewarmed the evening before")
|
||||
}
|
||||
for _, bad := range []ScaleWindow{{From: "8:00", To: "18:00"}, {From: "08:00", To: "08:00"},
|
||||
{From: "08:00", To: "18:00", Days: []string{"monday"}}, {From: "08:00", To: "18:00", Min: 3, Max: 2}} {
|
||||
if bad.validate() == nil {
|
||||
t.Errorf("%+v accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -88,17 +88,43 @@ func (r *Reconciler) desiredFor(m catalog.Model) int {
|
||||
r.mu.Lock()
|
||||
extra := r.extra[m.Name]
|
||||
r.mu.Unlock()
|
||||
return min(base+extra, max(m.Scaling.Max, base))
|
||||
return min(base+extra, max(r.maxFor(m), base))
|
||||
}
|
||||
|
||||
// minFor is a model's minimum now: its schedule's window's, or its own
|
||||
// outside every window.
|
||||
func (r *Reconciler) minFor(m catalog.Model) int {
|
||||
if w, ok := m.Scaling.Window(r.localNow()); ok {
|
||||
return w.Min
|
||||
}
|
||||
return m.Scaling.Min
|
||||
}
|
||||
|
||||
// maxFor is the ceiling load scaling reaches now.
|
||||
func (r *Reconciler) maxFor(m catalog.Model) int {
|
||||
if w, ok := m.Scaling.Window(r.localNow()); ok && w.Max > 0 {
|
||||
return w.Max
|
||||
}
|
||||
return m.Scaling.Max
|
||||
}
|
||||
|
||||
// localNow is now in the time zone schedules are written in.
|
||||
func (r *Reconciler) localNow() time.Time {
|
||||
if r.cfg.Location != nil {
|
||||
return r.now().In(r.cfg.Location)
|
||||
}
|
||||
return r.now()
|
||||
}
|
||||
|
||||
// baseFor is what a model runs without load: its minimum, or for an on-demand
|
||||
// model one while somebody is asking and none once they stop.
|
||||
// model one while somebody is asking and none once they stop. A schedule's
|
||||
// window keeps its minimum running however quiet it is.
|
||||
func (r *Reconciler) baseFor(m catalog.Model) int {
|
||||
if m.IsExternal() {
|
||||
return 0 // served elsewhere: nothing to place, whatever its scaling says
|
||||
}
|
||||
if !m.Scaling.OnDemand() {
|
||||
return m.Scaling.Min
|
||||
if lo := r.minFor(m); !m.Scaling.OnDemand() || lo > 0 {
|
||||
return lo
|
||||
}
|
||||
r.mu.Lock()
|
||||
last, asked := r.wanted[m.Name]
|
||||
@@ -188,7 +214,8 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
for _, m := range r.cat.Load().Models() {
|
||||
base := r.baseFor(m)
|
||||
r.mu.Lock()
|
||||
if base == 0 || m.Scaling.Max <= base {
|
||||
ceiling := r.maxFor(m)
|
||||
if base == 0 || ceiling <= base {
|
||||
delete(r.extra, m.Name)
|
||||
delete(r.busySince, m.Name)
|
||||
delete(r.quietSince, m.Name)
|
||||
@@ -217,7 +244,7 @@ func (r *Reconciler) RecordLoad(loads map[string]Load) {
|
||||
r.busySince[m.Name] = now
|
||||
break
|
||||
}
|
||||
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < m.Scaling.Max && !starting {
|
||||
if now.Sub(since) >= r.cfg.ScaleUpAfter && base+extra < ceiling && !starting {
|
||||
r.extra[m.Name] = extra + 1
|
||||
r.busySince[m.Name] = now
|
||||
r.log.Info("scaling up for load", "model", m.Name, "replicas", base+extra+1,
|
||||
|
||||
@@ -196,7 +196,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
cat := r.cat.Load()
|
||||
now := r.now()
|
||||
myRank := catalog.ClassRank(m.EffectiveClass())
|
||||
belowMin := r.liveCount(m.Name) < m.Scaling.Min
|
||||
belowMin := r.liveCount(m.Name) < r.minFor(m)
|
||||
|
||||
pol := r.policy()
|
||||
var borrowed map[string]bool
|
||||
@@ -229,7 +229,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
|
||||
class := other.EffectiveClass()
|
||||
rank := catalog.ClassRank(class)
|
||||
sort.Slice(reps, func(i, j int) bool { return reps[i].CreatedAt.After(reps[j].CreatedAt) })
|
||||
above := max(len(reps)-other.Scaling.Min, 0)
|
||||
above := max(len(reps)-r.minFor(other), 0)
|
||||
for i, rep := range reps {
|
||||
aboveMin := i < above
|
||||
tier := -1
|
||||
|
||||
@@ -50,6 +50,9 @@ type Dispatcher interface {
|
||||
|
||||
// Config tunes the reconciliation loop.
|
||||
type Config struct {
|
||||
// Location is the time zone models' scaling schedules are written in;
|
||||
// nil is the control plane's own.
|
||||
Location *time.Location
|
||||
// Interval is how often desired and actual are compared when nothing else
|
||||
// happens. Events wake the loop sooner.
|
||||
Interval time.Duration
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
package reconciler
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
func TestAScheduleRaisesTheMinimumThroughItsHours(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling = catalog.Scaling{Min: 1, Max: 4, Prewarm: 30 * time.Minute,
|
||||
Schedule: []catalog.ScaleWindow{{From: "08:00", To: "18:00", Min: 2, Max: 3}}}
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
at := func(hm string) {
|
||||
target, _ := time.Parse("2006-01-02 15:04", "2026-01-02 "+hm)
|
||||
h.clock.advance(target.Sub(h.clock.now()))
|
||||
}
|
||||
cur := func() catalog.Model { got, _ := h.live.Load().Resolve(m.Name); return got }
|
||||
at("03:00")
|
||||
if n := h.r.desiredFor(cur()); n != 1 {
|
||||
t.Errorf("at night: %d", n)
|
||||
}
|
||||
at("07:45") // prewarmed
|
||||
if n := h.r.desiredFor(cur()); n != 2 {
|
||||
t.Errorf("before the window, prewarming: %d", n)
|
||||
}
|
||||
if mx := h.r.maxFor(cur()); mx != 3 {
|
||||
t.Errorf("the window's ceiling: %d", mx)
|
||||
}
|
||||
at("18:30")
|
||||
if n := h.r.desiredFor(cur()); n != 1 {
|
||||
t.Errorf("after the window: %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOnDemandModelIsKeptLoadedThroughItsWindow(t *testing.T) {
|
||||
m := testModel(1)
|
||||
m.Scaling = catalog.Scaling{Min: 0, Max: 1,
|
||||
Schedule: []catalog.ScaleWindow{{From: "09:00", To: "17:00", Min: 1}}}
|
||||
h := newHarness(t, m, gpuNode("1", 8))
|
||||
target, _ := time.Parse("2006-01-02 15:04", "2026-01-02 10:00")
|
||||
h.clock.advance(target.Sub(h.clock.now()))
|
||||
got, _ := h.live.Load().Resolve(m.Name)
|
||||
if n := h.r.desiredFor(got); n != 1 {
|
||||
t.Errorf("unasked-for in its window: %d", n)
|
||||
}
|
||||
h.clock.advance(8 * time.Hour)
|
||||
if n := h.r.desiredFor(got); n != 0 {
|
||||
t.Errorf("unasked-for outside its window: %d", n)
|
||||
}
|
||||
}
|
||||
@@ -64,6 +64,7 @@ var Reference = []Block{
|
||||
{Key: "lost_after", Type: "time.Duration", Doc: "", Nested: ""},
|
||||
{Key: "port_range", Type: "[]int", Doc: "PortRange is the per-node range for engine HTTP and dist-init ports.", Nested: ""},
|
||||
{Key: "mgmt_interface", Type: "string", Doc: "MgmtInterface is the ethernet interface NCCL bootstraps over.", Nested: ""},
|
||||
{Key: "timezone", Type: "string", Doc: "Timezone is where models' scaling schedules are read, an IANA name\nlike Europe/Istanbul; empty is the control plane's own.", Nested: ""},
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
|
||||
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
|
||||
}},
|
||||
@@ -268,10 +269,19 @@ var Reference = []Block{
|
||||
{Key: "ttft_p99", Type: "time.Duration", Doc: "TTFTp99 is the time to the first token 99 requests in 100 are within.", Nested: ""},
|
||||
{Key: "tpot_p99", Type: "time.Duration", Doc: "TPOTp99 is the time per output token, after the first, 99 streamed\nrequests in 100 are within.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ScaleWindow", Title: "ScaleWindow", Doc: "", Fields: []Field{
|
||||
{Key: "days", Type: "[]string", Doc: "Days are mon … sun, or weekdays, weekends; empty is every day.", Nested: ""},
|
||||
{Key: "from", Type: "string", Doc: "From and To are HH:MM; a window whose To is before its From runs\npast midnight.", Nested: ""},
|
||||
{Key: "to", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "min", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "max", Type: "int", Doc: "Max, when set, is the window's ceiling for load scaling; zero keeps\nthe model's own.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "Scaling", Title: "Scaling", Doc: "", Fields: []Field{
|
||||
{Key: "min", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "max", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "idle_timeout", Type: "time.Duration", Doc: "IdleTimeout is how long an on-demand model may go unasked-for before its\nreplicas are stopped. Zero takes the cluster default.\n\nMinutes rather than seconds, always. Stopping a replica throws away the\nKV cache prefix routing spent its effort filling, and a model that flaps\nin and out loses that entirely — so the saving has to be worth more than\nthe cache, which it only is over a real idle period.", Nested: ""},
|
||||
{Key: "schedule", Type: "[]ScaleWindow", Doc: "Schedule changes min and max by the time of day, in the control\nplane's control.timezone: more replicas kept through working hours,\nfewer at night, an on-demand model kept loaded while people are at\nwork. The first window that holds now wins; outside them all, Min and\nMax apply.", Nested: ""},
|
||||
{Key: "prewarm", Type: "time.Duration", Doc: "Prewarm starts a window this long before it opens, so its replicas\nhave loaded by then: loading a model is minutes, and the people who\narrive at nine are the reason for the window.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Scoring", Title: "Scoring", Doc: "", Fields: []Field{
|
||||
{Key: "target_waiting", Type: "float64", Doc: "", Nested: ""},
|
||||
|
||||
Reference in new issue
Block a user