95 Releases 95 Tags

RSS Feed

• v0.2.0 a3f71aa5ee

  Compare

  v0.2.0 — Phase 1: static router Pre-Release

  ilker released this 2026-09-16 01:41:13 +02:00 | 844 commits to main since this release

  The router serves traffic. llmgw server is real: it fronts a static set of
  engine replicas with an OpenAI-compatible API.

  Added

  • Streaming proxy. Relays SSE event by event with a flush after each, and
    propagates cancellation: when a client disconnects, the upstream request is
    aborted so the GPU stops working. Separate deadlines for dial, response
    headers, first token, inter-token idle and total duration — a wedged engine is
    indistinguishable from a slow one without the idle watchdog.
  • Retry and fallback. Another endpoint is tried only while nothing has
    reached the client; once bytes are out, splicing two responses together would
    be worse than the short stream. Models can name fallback models, and the
    response says which one served it via x-llmgw-served-model.
  • OpenAI surface. /v1/chat/completions, /v1/completions,
    /v1/embeddings and /v1/models, with aliases resolved from the catalog.
    Only routing-relevant fields are parsed; the rest of the body passes through
    untouched.
  • Usage accounting that does not depend on the client. The router can ask
    engines for usage the client never requested, then drop that chunk on the way
    back, so token counts are recorded either way.
  • Configuration (internal/config): YAML through koanf, overlaid on
    defaults, with LLMGW_-prefixed environment overrides. Validation rejects
    duplicate endpoint ids and aliases that would shadow another model — both
    would otherwise misroute silently at request time.
  • Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
    values and per-filter drop counts. Routing profile weights cannot be tuned
    without seeing what the scorers actually produced. Served on a separate admin
    listener alongside health and pprof.

  Phase 1 exit criterion, measured in test/e2e

  • 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
  • Every client disconnecting frees the KV blocks on every engine.
  • Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
    looser bound by default since a shared machine cannot hold 2 ms reliably; set
    LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

  Known limitations

  • No admission control, so a batch workload can still starve interactive
    traffic. That is v0.3.0, and it is the most valuable thing still missing.
  • The endpoint list is static; the control plane arrives in v0.5.0.
  • No authentication, rate limiting or budgets yet (v1.0.0).
  • Prompt length is estimated as bytes/4. It becomes exact only with a real
    tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

  ---

  Full notes: CHANGELOG.md

  Downloads

  • Source Code (ZIP)
  • Source Code (TAR.GZ)