95 Releases 95 Tags

RSS Feed

• v0.3.0 dc38dd6096

  Compare

  v0.3.0 — Phase 1.5: admission control Pre-Release

  ilker released this 2026-09-16 01:53:38 +02:00 | 843 commits to main since this release

  Flow control. A batch workload no longer starves interactive traffic.

  Added

  • Admission control (internal/router/admission), inserted between quota
    checks and scheduling. It answers when a request may proceed; the scheduler
    still answers where it goes. Keeping them apart is the point: an engine
    queue is FIFO and knows nothing about tenants, so once a batch job is in it,
    no amount of endpoint selection rescues the request behind it.
  • Saturation detector. One number per pool, the worse of queue depth and KV
    occupancy. Either signal alone misleads — a pool can have an empty queue and a
    nearly full cache, or the reverse.
  • Priority bands. Each band has a saturation ceiling; above it, that band
    waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
    "flow up to the healthy limit"; above 1 means never wait.
  • Tenant fairness. Within a band, dispatch turns rotate between tenants, so
    one caller sending a thousand requests occupies one slot in the rotation
    rather than a thousand.
  • Bounded queues. Capacity per band (immediate 503 when full) and a TTL
    (504), because sending the GPU work the client has already abandoned helps
    nobody. Cancelled requests leave the queue instead of holding capacity.
  • Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total,
    llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry
    x-llmgw-queue-ms when a request waited.

  Phase 1.5 exit criterion, measured in test/e2e

  While 500 concurrent bronze requests run continuously, interactive p99 in the
  platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget.
  With admission switched off the same load pushes it to 55–60x. Both
  directions are asserted, so the test fails if admission regresses and if the
  load generator ever stops being a real load.

  Two lags, two different fixes

  Getting this right took longer than the queueing itself:

  • Engine metrics are scraped, so they lag by an interval. The router's inflight
    count does not — but it includes running requests, and treating those as
    queue depth makes a healthy engine look overloaded every interval. What is
    genuinely queued is inflight − batch capacity, and batch capacity is not in
    the metric contract, so it is learned as the high-water mark of observed
    running.
  • Between admission granting a request and the router dispatching it, the
    request is invisible to every signal. A simultaneous burst therefore passed
    the check en masse — exactly the batch arrival pattern this layer exists to
    survive. The controller now counts its own outstanding leases against
    dispatched inflight.

  Known limitations

  • The band comes from a trusted header until authentication arrives in v1.0.0.
    Anyone who can reach the port can promote themselves to platinum.
  • The tenant key is the session id, for the same reason.
  • Displacement (dropping queued low-band work to make room) is not implemented;
    the plan defers it, and bounded queues plus TTL cover most of the need.

  ---

  Full notes: CHANGELOG.md

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)