CI / test (push) Successful in 18s
Versions track the phases in the plan: a phase is released once its exit criterion is met, not when its code lands. That keeps "released" meaning something — v0.1.0 ships sim-engine and the routing libraries, and says plainly that the gateway does not serve traffic yet. scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent and never destructive, so it can be re-run after the roadmap shifts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
68 lines
3.1 KiB
Markdown
68 lines
3.1 KiB
Markdown
# Changelog
|
|
|
|
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
|
is only released once its **exit criterion** is met, not when its code lands.
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | in progress |
|
|
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | planned |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
|
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
|
APIs change between minors without a deprecation period.
|
|
|
|
## v0.1.0 — 2026-09-16
|
|
|
|
First tagged version. It is useful for exactly one thing: developing and testing
|
|
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
|
returns "not implemented".
|
|
|
|
### Added
|
|
|
|
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
|
that models the behaviours a router has to cope with:
|
|
|
|
- continuous batching with a bounded concurrent sequence count
|
|
- KV cache block accounting, queueing when blocks run out, including
|
|
head-of-line blocking
|
|
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
|
effect on time to first token
|
|
- per-token latency that grows with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` under vLLM's real metric names
|
|
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
|
|
|
**Routing primitives** — libraries, not yet wired into a server:
|
|
|
|
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
|
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
|
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
|
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
|
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
|
detection and drain rate derived from engine counters.
|
|
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
|
endpoints missing any of the mandatory three. The metric name mapping is
|
|
overridable, because names drift between engine versions.
|
|
- `router/breaker`: cooldown and circuit breaking in one component, with
|
|
exponential backoff and half-open probes.
|
|
|
|
### Known limitations
|
|
|
|
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
|
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
|
- No admission control, so a batch workload can still starve interactive traffic
|
|
once the router exists. That is what v0.3.0 is for.
|
|
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
|
|
|
### Notes
|
|
|
|
- The project language is English throughout: code, comments, tests, commits and
|
|
the design document.
|
|
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
|
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|