-
v0.6.0 Stable
ilker released this
2026-09-18 00:33:39 +02:00 | 750 commits to main since this releaseDownloads
-
v0.5.0 — Phases 2 and 3: agent and control plane
Pre-ReleaseCI / test (push) Failing after 50s2026-09-16 06:27:21 +02:00 | 833 commits to main since this releasePhases 2 and 3. The cluster runs itself: agents report their hardware, the
control plane places replicas, andctlmakes all of it visible.This release covers both phases because their exit criteria depended on the same
piece — the agent stream — and neither was demonstrable without it.ctl
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drainnodessurfaces unhealthy GPUs and why they are excluded, which is usually
the answer when a node looks idle but nothing schedules onto it.explain-placementprints every candidate considered and the reason it was
rejected. "Unschedulable" on its own says nothing about which of six
constraints bit.drainis a rolling move, not a stop: the replica is drained, a replacement
is placed elsewhere, and the original is stopped only once the replacement is
serving. Bounded by a grace period, because draining the only node that can
host a model would otherwise never finish and leave it undrainable.cordonis the weaker operation: no new placement, nothing disturbed.
The management API is on the admin listener, not the client-facing one. The two
have different audiences: one serves inference to whoever holds an API key, the
other lets a caller cordon nodes.ctl logsis not here. It needs a request/response pair the agent protocol does
not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
Full notes: CHANGELOG.md
Downloads
-
v0.3.0 — Phase 1.5: admission control
Pre-ReleaseCI / test (push) Successful in 48s2026-09-16 01:53:38 +02:00 | 843 commits to main since this releaseFlow control. A batch workload no longer starves interactive traffic.
Added
- Admission control (
internal/router/admission), inserted between quota
checks and scheduling. It answers when a request may proceed; the scheduler
still answers where it goes. Keeping them apart is the point: an engine
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
no amount of endpoint selection rescues the request behind it. - Saturation detector. One number per pool, the worse of queue depth and KV
occupancy. Either signal alone misleads — a pool can have an empty queue and a
nearly full cache, or the reverse. - Priority bands. Each band has a saturation ceiling; above it, that band
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
"flow up to the healthy limit"; above 1 means never wait. - Tenant fairness. Within a band, dispatch turns rotate between tenants, so
one caller sending a thousand requests occupies one slot in the rotation
rather than a thousand. - Bounded queues. Capacity per band (immediate 503 when full) and a TTL
(504), because sending the GPU work the client has already abandoned helps
nobody. Cancelled requests leave the queue instead of holding capacity. - Metrics:
llmgw_admission_wait_seconds,llmgw_admission_rejected_total,
llmgw_admission_queue_depth,llmgw_pool_saturation. Responses carry
x-llmgw-queue-mswhen a request waited.
Phase 1.5 exit criterion, measured in
test/e2eWhile 500 concurrent bronze requests run continuously, interactive p99 in the
platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget.
With admission switched off the same load pushes it to 55–60x. Both
directions are asserted, so the test fails if admission regresses and if the
load generator ever stops being a real load.Two lags, two different fixes
Getting this right took longer than the queueing itself:
- Engine metrics are scraped, so they lag by an interval. The router's inflight
count does not — but it includes running requests, and treating those as
queue depth makes a healthy engine look overloaded every interval. What is
genuinely queued isinflight − batch capacity, and batch capacity is not in
the metric contract, so it is learned as the high-water mark of observed
running. - Between admission granting a request and the router dispatching it, the
request is invisible to every signal. A simultaneous burst therefore passed
the check en masse — exactly the batch arrival pattern this layer exists to
survive. The controller now counts its own outstanding leases against
dispatched inflight.
Known limitations
- The band comes from a trusted header until authentication arrives in v1.0.0.
Anyone who can reach the port can promote themselves to platinum. - The tenant key is the session id, for the same reason.
- Displacement (dropping queued low-band work to make room) is not implemented;
the plan defers it, and bounded queues plus TTL cover most of the need.
Full notes: CHANGELOG.md
Downloads
- Admission control (
-
v0.2.0 — Phase 1: static router
Pre-ReleaseCI / test (push) Successful in 1m1s2026-09-16 01:41:13 +02:00 | 844 commits to main since this releaseThe router serves traffic.
llmgw serveris real: it fronts a static set of
engine replicas with an OpenAI-compatible API.Added
- Streaming proxy. Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog. - Retry and fallback. Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it viax-llmgw-served-model. - OpenAI surface.
/v1/chat/completions,/v1/completions,
/v1/embeddingsand/v1/models, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched. - Usage accounting that does not depend on the client. The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way. - Configuration (
internal/config): YAML through koanf, overlaid on
defaults, withLLMGW_-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time. - Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
Phase 1 exit criterion, measured in
test/e2e- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
LLMGW_E2E_STRICT=1on quiet hardware to enforce the real target.
Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing. - The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
Full notes: CHANGELOG.md
Downloads
- Streaming proxy. Relays SSE event by event with a flush after each, and
-
v0.1.0 — Phase 0: skeleton and sim-engine
Pre-ReleaseCI / test (push) Successful in 18s2026-09-16 01:29:41 +02:00 | 846 commits to main since this releaseFirst tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. The gateway does not serve traffic yet —llmgw server
returns "not implemented".Added
sim-engine (
llmgw sim-engine) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking - automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token - per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsunder vLLM's real metric names- failure injection (
--fail-rate) and deterministic runs (--seed)
Routing primitives — libraries, not yet wired into a server:
router/scheduler: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.router/endpoint: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.router/scrape: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.router/breaker: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
Known limitations
llmgw server,llmgw agentandllmgw ctlare stubs.- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for. - Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
Notes
- The project language is English throughout: code, comments, tests, commits and
the design document. - Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
vllm:kv_cache_usage_perc, notvllm:gpu_cache_usage_perc.
Full notes and the version/phase map: CHANGELOG.md
Downloads