Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 40c884a50a
CI / test (push) Successful in 18s
Add a changelog and map releases onto roadmap phases
Versions track the phases in the plan: a phase is released once its exit
criterion is met, not when its code lands. That keeps "released" meaning
something — v0.1.0 ships sim-engine and the routing libraries, and says plainly
that the gateway does not serve traffic yet.

scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent
and never destructive, so it can be re-run after the roadmap shifts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:29:41 +02:00

3.1 KiB

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router: streaming proxy, filter/score/pick in progress
v0.3.0 1.5 Admission control: priority bands, saturation detector planned
v0.4.0 2 Agent: GPU/IB discovery, single-node engines planned
v0.5.0 3 Control plane: catalog, placement, reconciler planned
v0.6.0 4 Multi-node replicas and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

  • continuous batching with a bounded concurrent sequence count
  • KV cache block accounting, queueing when blocks run out, including head-of-line blocking
  • automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
  • per-token latency that grows with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics under vLLM's real metric names
  • failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

  • router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
  • router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
  • router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
  • router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

  • llmgw server, llmgw agent and llmgw ctl are stubs.
  • The scheduler has no HTTP layer yet; nothing routes real requests.
  • No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
  • Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

  • The project language is English throughout: code, comments, tests, commits and the design document.
  • Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.