Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 40c884a50a
CI / test (push) Successful in 18s
Add a changelog and map releases onto roadmap phases
Versions track the phases in the plan: a phase is released once its exit
criterion is met, not when its code lands. That keeps "released" meaning
something — v0.1.0 ships sim-engine and the routing libraries, and says plainly
that the gateway does not serve traffic yet.

scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent
and never destructive, so it can be re-run after the roadmap shifts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:29:41 +02:00

68 lines
3.1 KiB
Markdown

# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | in progress |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | planned |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.