Phase 1.5: admission control

Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
ilkerandClaude Opus 5 committed 2026-09-16 01:53:38 +02:00
1 parent a3f71aa5ee
commit dc38dd6096
16 files changed
+1744 -24

No files matched your search

+26
View File
@@ -43,6 +43,32 @@ router:
idle: 60s # between streamed tokens; catches a wedged engine
total: 30m
# Flow control: decides *when* a request may proceed, before scheduling
# decides where it goes. Without it a batch workload starves interactive
# traffic, because an engine queue is FIFO and knows nothing about tenants.
admission:
enabled: true
# Until authentication lands, the band comes from a header and is trusted.
# Do not expose this port to callers who should not be able to promote
# themselves.
band_header: x-llmgw-band
default_band: gold
# A band waits once pool saturation passes its ceiling, so the cluster
# degrades from the bottom up. A ceiling of 1.0 means "flow up to the
# healthy limit"; use a value above 1 for a band that must never wait.
bands:
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
# Drop a request that has waited this long: the client has usually given up,
# and sending the GPU work nobody is waiting for helps no one.
queue_ttl: 30s
# Reject immediately once a band's queue is this deep, rather than letting
# it accumulate silently.
queue_capacity_per_band: 2000
interval: 10ms
# What the scorers normalize against. Measure these with `ctl bench` rather
# than guessing: target_waiting in particular governs how the whole system
# behaves under load. Too low starves the GPU, too high brings back queueing