Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 dc38dd6096 Phase 1.5: admission control
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.

- saturation detector: one number per pool, the worse of queue depth and KV
  occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
  the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
  one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
  queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.

Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.

Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.

One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:53:38 +02:00

8.8 KiB
Raw Blame History

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router: streaming proxy, filter/score/pick released
v0.3.0 1.5 Admission control: priority bands, saturation detector released
v0.4.0 2 Agent: GPU/IB discovery, single-node engines in progress
v0.5.0 3 Control plane: catalog, placement, reconciler planned
v0.6.0 4 Multi-node replicas and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

v0.3.0 — 2026-09-16

Flow control. A batch workload no longer starves interactive traffic.

Added

  • Admission control (internal/router/admission), inserted between quota checks and scheduling. It answers when a request may proceed; the scheduler still answers where it goes. Keeping them apart is the point: an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no amount of endpoint selection rescues the request behind it.
  • Saturation detector. One number per pool, the worse of queue depth and KV occupancy. Either signal alone misleads — a pool can have an empty queue and a nearly full cache, or the reverse.
  • Priority bands. Each band has a saturation ceiling; above it, that band waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means "flow up to the healthy limit"; above 1 means never wait.
  • Tenant fairness. Within a band, dispatch turns rotate between tenants, so one caller sending a thousand requests occupies one slot in the rotation rather than a thousand.
  • Bounded queues. Capacity per band (immediate 503 when full) and a TTL (504), because sending the GPU work the client has already abandoned helps nobody. Cancelled requests leave the queue instead of holding capacity.
  • Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total, llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry x-llmgw-queue-ms when a request waited.

Phase 1.5 exit criterion, measured in test/e2e

While 500 concurrent bronze requests run continuously, interactive p99 in the platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget. With admission switched off the same load pushes it to 55–60x. Both directions are asserted, so the test fails if admission regresses and if the load generator ever stops being a real load.

Two lags, two different fixes

Getting this right took longer than the queueing itself:

  • Engine metrics are scraped, so they lag by an interval. The router's inflight count does not — but it includes running requests, and treating those as queue depth makes a healthy engine look overloaded every interval. What is genuinely queued is inflight − batch capacity, and batch capacity is not in the metric contract, so it is learned as the high-water mark of observed running.
  • Between admission granting a request and the router dispatching it, the request is invisible to every signal. A simultaneous burst therefore passed the check en masse — exactly the batch arrival pattern this layer exists to survive. The controller now counts its own outstanding leases against dispatched inflight.

Known limitations

  • The band comes from a trusted header until authentication arrives in v1.0.0. Anyone who can reach the port can promote themselves to platinum.
  • The tenant key is the session id, for the same reason.
  • Displacement (dropping queued low-band work to make room) is not implemented; the plan defers it, and bounded queues plus TTL cover most of the need.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

  • Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
  • Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
  • OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
  • Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
  • Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
  • Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

  • 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
  • Every client disconnecting frees the KV blocks on every engine.
  • Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

  • No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
  • The endpoint list is static; the control plane arrives in v0.5.0.
  • No authentication, rate limiting or budgets yet (v1.0.0).
  • Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

  • continuous batching with a bounded concurrent sequence count
  • KV cache block accounting, queueing when blocks run out, including head-of-line blocking
  • automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
  • per-token latency that grows with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics under vLLM's real metric names
  • failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

  • router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
  • router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
  • router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
  • router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

  • llmgw server, llmgw agent and llmgw ctl are stubs.
  • The scheduler has no HTTP layer yet; nothing routes real requests.
  • No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
  • Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

  • The project language is English throughout: code, comments, tests, commits and the design document.
  • Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.