Files
llmgw/internal/router/admission/saturation.go
T
ilkerandClaude Opus 5 dc38dd6096
CI / test (push) Successful in 48s
Phase 1.5: admission control
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00

99 lines
3.7 KiB
Go

// Package admission answers "may this request proceed now?" (plan §7.2).
//
// It is deliberately separate from scheduling, which answers "where should it
// go?". Kept apart, the router can hold requests in its own queue and apply
// priority and tenant fairness *before* they enter an engine queue. An engine
// queue is FIFO and knows nothing about tenants: once a batch job has put 500
// requests into it, no amount of clever endpoint selection will rescue the
// interactive request that arrives next.
package admission
import (
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
// Thresholds define what "saturated" means for a pool.
type Thresholds struct {
// TargetWaitingPerReplica is the healthy queue depth: enough for continuous
// batching to stay efficient, not enough for queueing delay to show up.
// Measure it with ctl bench (plan §10); this single number governs how the
// whole system behaves under load.
TargetWaitingPerReplica float64
// KVCeiling is the KV occupancy treated as full.
KVCeiling float64
}
func (t *Thresholds) normalize() {
if t.TargetWaitingPerReplica <= 0 {
t.TargetWaitingPerReplica = 4
}
if t.KVCeiling <= 0 || t.KVCeiling > 1 {
t.KVCeiling = 0.9
}
}
// Saturation reduces a pool's state to a single number, where 1 means "at the
// healthy limit" and above 1 means overloaded.
//
// Two signals are combined because either alone misleads: a pool can have an
// empty queue while its KV cache is nearly full (long contexts), or plenty of
// KV headroom while requests pile up (slow decode). The worse of the two wins.
//
// unreported is load that has been admitted but that no endpoint has accounted
// for yet. Without it a simultaneous burst all passes the check before any of
// it becomes visible — which is precisely when admission needs to bite.
func Saturation(eps []endpoint.Snapshot, unreported int, t Thresholds) float64 {
t.normalize()
var sum float64
var n int
for _, e := range eps {
// A replica the breaker has taken out is not available capacity, so
// counting it would make the pool look healthier than it is.
if e.Breaker.String() == "open" {
continue
}
// The engine's own queue is the truth, but it is one scrape interval
// old. Fill that gap with what the router knows without any lag: how
// many requests it has dispatched here, minus what this engine can run
// at once. Anything beyond that capacity is queued by definition.
//
// Subtracting the capacity is the whole point. Counting raw inflight
// would make a fully busy but perfectly healthy engine look like it had
// a queue as long as its batch, every single scrape interval.
waiting := float64(e.Stats.Waiting)
if e.RunningCapacity > 0 {
if queued := e.Inflight - e.RunningCapacity; float64(queued) > waiting {
waiting = float64(queued)
}
} else if e.Inflight > 0 {
// Nothing observed yet, so capacity is unknown. Treat dispatched
// work as queued: conservative for the first scrape, and it relaxes
// as soon as the engine reports anything.
waiting = max(waiting, float64(e.Inflight))
}
q := waiting / t.TargetWaitingPerReplica
kv := e.Stats.KVCacheUsage / t.KVCeiling
sum += max(q, kv)
n++
}
_ = n
if n == 0 {
// Every replica is out. Report full saturation so low bands stop
// queueing work that has nowhere to go; the dispatch layer will return
// a clearer error than a queue timeout.
return 1
}
// Spread unaccounted admissions evenly; they will land somewhere.
if unreported > 0 {
sum += float64(unreported) / t.TargetWaitingPerReplica
}
// The mean, not the minimum: one idle replica does not make a busy pool
// idle, and admission is about the pool's capacity as a whole.
return sum / float64(n)
}