Versions track the phases in the plan: a phase is released once its exit criterion is met, not when its code lands. That keeps "released" meaning something — v0.1.0 ships sim-engine and the routing libraries, and says plainly that the gateway does not serve traffic yet. scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent and never destructive, so it can be re-run after the roadmap shifts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.1 KiB
3.1 KiB
Changelog
Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | in progress |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | planned |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.
v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. The gateway does not serve traffic yet — llmgw server
returns "not implemented".
Added
sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsunder vLLM's real metric names- failure injection (
--fail-rate) and deterministic runs (--seed)
Routing primitives — libraries, not yet wired into a server:
router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.
Known limitations
llmgw server,llmgw agentandllmgw ctlare stubs.- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
Notes
- The project language is English throughout: code, comments, tests, commits and the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
vllm:kv_cache_usage_perc, notvllm:gpu_cache_usage_perc.