Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00

5.8 KiB

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router: streaming proxy, filter/score/pick released
v0.3.0 1.5 Admission control: priority bands, saturation detector in progress
v0.4.0 2 Agent: GPU/IB discovery, single-node engines planned
v0.5.0 3 Control plane: catalog, placement, reconciler planned
v0.6.0 4 Multi-node replicas and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

  • Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
  • Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
  • OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
  • Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
  • Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
  • Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

  • 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
  • Every client disconnecting frees the KV blocks on every engine.
  • Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

  • No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
  • The endpoint list is static; the control plane arrives in v0.5.0.
  • No authentication, rate limiting or budgets yet (v1.0.0).
  • Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

  • continuous batching with a bounded concurrent sequence count
  • KV cache block accounting, queueing when blocks run out, including head-of-line blocking
  • automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
  • per-token latency that grows with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics under vLLM's real metric names
  • failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

  • router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
  • router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
  • router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
  • router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

  • llmgw server, llmgw agent and llmgw ctl are stubs.
  • The scheduler has no HTTP layer yet; nothing routes real requests.
  • No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
  • Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

  • The project language is English throughout: code, comments, tests, commits and the design document.
  • Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.