Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 a3f71aa5ee
CI / test (push) Successful in 1m1s
Phase 1: the router serves traffic
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00

117 lines
5.8 KiB
Markdown

# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | in progress |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## v0.2.0 — 2026-09-16
The router serves traffic. `llmgw server` is real: it fronts a static set of
engine replicas with an OpenAI-compatible API.
### Added
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog.
- **Retry and fallback.** Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it via `x-llmgw-served-model`.
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched.
- **Usage accounting that does not depend on the client.** The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way.
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time.
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
### Phase 1 exit criterion, measured in `test/e2e`
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
### Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing.
- The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.