CI / test (push) Successful in 1m1s
llmgw server is real now. It fronts a static set of engine replicas with an OpenAI-compatible API, scoring endpoints through the pipeline added in the previous commit. - proxy: SSE relayed event by event with a flush after each, cancellation propagated upstream, and separate deadlines for dial, headers, first token, inter-token idle and total. The idle watchdog is what distinguishes a wedged engine from a slow one. - router: dispatch with retry and model fallback. Retry only happens while nothing has reached the client; after that, splicing two responses together would be worse than the short stream the client already sees. - config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides. Validation rejects duplicate endpoint ids and aliases that shadow a model, because both misroute silently at request time rather than failing loudly. - metrics: requests, duration, TTFT, tokens, plus per-scorer values and per-filter drops on a separate admin listener with health and pprof. Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across 3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV blocks on every engine; router overhead p99 is about 0.9 ms against the plan's 2 ms budget. The handlers live in internal/router rather than an internal/router/openai package as the plan's tree sketched: they need the dispatch loop, and splitting them apart would have meant an interface with exactly one implementation. Plan §4 updated to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
117 lines
5.8 KiB
Markdown
117 lines
5.8 KiB
Markdown
# Changelog
|
|
|
|
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
|
is only released once its **exit criterion** is met, not when its code lands.
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
|
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | in progress |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
|
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
|
APIs change between minors without a deprecation period.
|
|
|
|
## v0.2.0 — 2026-09-16
|
|
|
|
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
|
engine replicas with an OpenAI-compatible API.
|
|
|
|
### Added
|
|
|
|
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
|
propagates cancellation: when a client disconnects, the upstream request is
|
|
aborted so the GPU stops working. Separate deadlines for dial, response
|
|
headers, first token, inter-token idle and total duration — a wedged engine is
|
|
indistinguishable from a slow one without the idle watchdog.
|
|
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
|
reached the client; once bytes are out, splicing two responses together would
|
|
be worse than the short stream. Models can name fallback models, and the
|
|
response says which one served it via `x-llmgw-served-model`.
|
|
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
|
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
|
Only routing-relevant fields are parsed; the rest of the body passes through
|
|
untouched.
|
|
- **Usage accounting that does not depend on the client.** The router can ask
|
|
engines for usage the client never requested, then drop that chunk on the way
|
|
back, so token counts are recorded either way.
|
|
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
|
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
|
duplicate endpoint ids and aliases that would shadow another model — both
|
|
would otherwise misroute silently at request time.
|
|
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
|
values and per-filter drop counts. Routing profile weights cannot be tuned
|
|
without seeing what the scorers actually produced. Served on a separate admin
|
|
listener alongside health and pprof.
|
|
|
|
### Phase 1 exit criterion, measured in `test/e2e`
|
|
|
|
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
|
- Every client disconnecting frees the KV blocks on every engine.
|
|
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
|
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
|
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
|
|
|
### Known limitations
|
|
|
|
- No admission control, so a batch workload can still starve interactive
|
|
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
|
- The endpoint list is static; the control plane arrives in v0.5.0.
|
|
- No authentication, rate limiting or budgets yet (v1.0.0).
|
|
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
|
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
|
|
|
## v0.1.0 — 2026-09-16
|
|
|
|
First tagged version. It is useful for exactly one thing: developing and testing
|
|
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
|
returns "not implemented".
|
|
|
|
### Added
|
|
|
|
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
|
that models the behaviours a router has to cope with:
|
|
|
|
- continuous batching with a bounded concurrent sequence count
|
|
- KV cache block accounting, queueing when blocks run out, including
|
|
head-of-line blocking
|
|
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
|
effect on time to first token
|
|
- per-token latency that grows with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` under vLLM's real metric names
|
|
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
|
|
|
**Routing primitives** — libraries, not yet wired into a server:
|
|
|
|
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
|
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
|
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
|
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
|
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
|
detection and drain rate derived from engine counters.
|
|
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
|
endpoints missing any of the mandatory three. The metric name mapping is
|
|
overridable, because names drift between engine versions.
|
|
- `router/breaker`: cooldown and circuit breaking in one component, with
|
|
exponential backoff and half-open probes.
|
|
|
|
### Known limitations
|
|
|
|
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
|
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
|
- No admission control, so a batch workload can still starve interactive traffic
|
|
once the router exists. That is what v0.3.0 is for.
|
|
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
|
|
|
### Notes
|
|
|
|
- The project language is English throughout: code, comments, tests, commits and
|
|
the design document.
|
|
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
|
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|