Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 f0134a2f95
CI / test (push) Successful in 24s
Phase 3 (2/3): catalog and reconciler
The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.

- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
  fallbacks resolve at load rather than at request time, because a typo in a
  fallback should stop a deployment instead of surfacing during the outage when
  the fallback is finally needed. The node/GPU/TP arithmetic is checked here
  too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
  already allocated and several from one snapshot would be handed the same GPUs.
  Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
  needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
  that will never complete while holding GPUs, so every shard is stopped and the
  replica rebuilt, with a jittered exponential backoff carried on the model
  rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
  network blip must not tear down replicas that are still serving, and
  rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.

The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.

Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:12:06 +02:00

16 KiB
Raw Blame History

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router: streaming proxy, filter/score/pick released
v0.3.0 1.5 Admission control: priority bands, saturation detector released
v0.4.0 2 Agent: GPU/IB discovery, single-node engines in progress
v0.5.0 3 Control plane: catalog, placement, reconciler planned
v0.6.0 4 Multi-node replicas and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

Unreleased — phases 2 and 3 in progress

Agent groundwork. Everything here works without GPUs and is covered by tests; what is missing is the control plane connection, so there is no release yet.

Added

  • IB discovery (internal/agent/discovery): reads /sys/class/infiniband, parses the formats sysfs actually uses ("4: ACTIVE", "200 Gb/sec (4X HDR)", hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node without IB is not an error — it is a single-node island.

  • GPU discovery via nvidia-smi, the cgo-free path. NVML needs cgo, which would force the whole binary, server included, to be built with a C toolchain. Health flags anything that should keep a card out of placement: uncorrected ECC errors, and memory already held by another process.

  • GPU class derivation. Sizes come from the driver in GiB, which does not always match the marketing name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g. Consistency matters more than matching the box, because the number decides whether a model fits.

  • Engine adapters for vLLM and SGLang: command and environment construction, kept free of process handling so it is testable without GPUs. vLLM supports both Ray and multiprocessing launch modes, because which one works depends on the version.

  • NCCL environment generation, the part plan §5.3 marks critical. NCCL_IB_HCA lists only the chosen fabric's adapters — leaving it unset lets NCCL pick an adapter on another fabric where the peer is unreachable, and the job hangs rather than failing. Bootstrap is pinned to the management interface for the same reason.

  • Validation before launch: a tensor-parallel group may not mix GPU types, the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below compute capability 8.9 rather than silently falling back to a slower path.

  • Supervisor (internal/agent/supervisor): processes run in their own process group and are stopped as a group, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader. Bounded log tail, a state machine that separates Starting from Loading (a 70B model takes minutes, and an operator needs to know that is normal), and orphan adoption through pid files fingerprinted with the command line — pids get reused, and adopting an unrelated process is worse than leaking the original.

  • Readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a model that is still loading.

  • llmgw agent inventory reports what the scheduler would see on a node, as a table or JSON. Run it before adding a node: it is the fastest way to find a card another process is using, or a port with no subnet manager.

  • NVML backend behind the nvml build tag, which adds NVLink topology. Nothing else exposes which GPUs share an NVLink domain, and that is what keeps a tensor-parallel group off the slow path inside a node. Discover falls back to nvidia-smi if NVML fails, and says so loudly rather than silently losing the topology. CI compiles the tagged build so it cannot rot unnoticed.

Not verified

The NVML path compiles but has never run against a driver — there is no GPU machine to try it on. Treat the NVLink domains it reports as unverified until someone runs llmgw agent inventory on a real node.

Phase 3 exit criterion: half met

TestLostNodeMovesReplicas covers the reconciliation half — a node going silent becomes unreachable, then lost, and its replica is rebuilt on a survivor. The other half, the router continuing to serve throughout, needs the EndpointSource wiring and the agent stream.

Still missing for phase 2

  • The agent-to-control-plane stream. It pairs with phase 3's control plane rather than standing alone, so llmgw agent itself is still a stub.

Phase 3: placement

  • Cluster inventory (internal/control/inventory): nodes, their hardware, their fabric attachments, and what is allocated where. A re-registering agent cannot clear an operator's cordon, and a heartbeat revives an unreachable node without doing so either. A GPU with an ECC fault or another process on it is not counted as free. Replicas are released whole, because they fail whole.
  • Placement (internal/control/placement): picks the highest-priority profile that fits and explains every candidate it rejected. Two constraints are enforced structurally rather than checked afterwards — there is no code path that can combine nodes from two fabrics, and only GPUs of the profile's class are ever considered.
  • Bin packing fills partly used nodes first, so whole nodes stay free for large models that cannot be split. Multi-node replicas get the opposite treatment: they are the least movable thing in the cluster, so they take the emptiest machines.
  • NVLink-aware GPU selection keeps a tensor-parallel group inside one NVLink domain where the topology is known. Without the NVML build it falls back to the lowest free indices, which is usually right but only by accident of how vendors number slots.
  • Property-based invariant tests (plan §13): 400 randomly generated clusters, checking on every placement that a multi-node replica lives on one fabric, that each shard receives only that fabric's adapters, and that no placement mixes GPU classes or uses an unhealthy or already-allocated GPU. The table-driven tests cover the cases someone thought of; this covers the ones nobody did.
  • Catalog (internal/control/catalog): models, profiles, aliases, fallbacks and scaling bounds. Aliases and fallbacks are resolved at load rather than at request time — a typo in a fallback should stop a deployment, not surface during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load instead of minutes into model loading.
  • Reconciler (internal/control/reconciler): the desired-versus-actual loop. It places one replica per pass, because placement depends on what is already allocated and starting several from one snapshot would hand them the same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves.
  • Replica failure is whole-replica failure. One dead shard makes the others useless — they wait on collectives that will never complete while holding GPUs — so the reconciler stops every shard and rebuilds, with a jittered exponential backoff carried on the model rather than the replica it just destroyed.
  • Node expiry in two stages. Unreachable first, Lost later. A brief network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading.
  • Scale-down drains before stopping, so in-flight requests finish instead of being cut off mid-stream.

v0.3.0 — 2026-09-16

Flow control. A batch workload no longer starves interactive traffic.

Added

  • Admission control (internal/router/admission), inserted between quota checks and scheduling. It answers when a request may proceed; the scheduler still answers where it goes. Keeping them apart is the point: an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no amount of endpoint selection rescues the request behind it.
  • Saturation detector. One number per pool, the worse of queue depth and KV occupancy. Either signal alone misleads — a pool can have an empty queue and a nearly full cache, or the reverse.
  • Priority bands. Each band has a saturation ceiling; above it, that band waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means "flow up to the healthy limit"; above 1 means never wait.
  • Tenant fairness. Within a band, dispatch turns rotate between tenants, so one caller sending a thousand requests occupies one slot in the rotation rather than a thousand.
  • Bounded queues. Capacity per band (immediate 503 when full) and a TTL (504), because sending the GPU work the client has already abandoned helps nobody. Cancelled requests leave the queue instead of holding capacity.
  • Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total, llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry x-llmgw-queue-ms when a request waited.

Phase 1.5 exit criterion, measured in test/e2e

While 500 concurrent bronze requests run continuously, interactive p99 in the platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget. With admission switched off the same load pushes it to 55–60x. Both directions are asserted, so the test fails if admission regresses and if the load generator ever stops being a real load.

Two lags, two different fixes

Getting this right took longer than the queueing itself:

  • Engine metrics are scraped, so they lag by an interval. The router's inflight count does not — but it includes running requests, and treating those as queue depth makes a healthy engine look overloaded every interval. What is genuinely queued is inflight − batch capacity, and batch capacity is not in the metric contract, so it is learned as the high-water mark of observed running.
  • Between admission granting a request and the router dispatching it, the request is invisible to every signal. A simultaneous burst therefore passed the check en masse — exactly the batch arrival pattern this layer exists to survive. The controller now counts its own outstanding leases against dispatched inflight.

Known limitations

  • The band comes from a trusted header until authentication arrives in v1.0.0. Anyone who can reach the port can promote themselves to platinum.
  • The tenant key is the session id, for the same reason.
  • Displacement (dropping queued low-band work to make room) is not implemented; the plan defers it, and bounded queues plus TTL cover most of the need.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

  • Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
  • Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
  • OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
  • Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
  • Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
  • Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

  • 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
  • Every client disconnecting frees the KV blocks on every engine.
  • Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

  • No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
  • The endpoint list is static; the control plane arrives in v0.5.0.
  • No authentication, rate limiting or budgets yet (v1.0.0).
  • Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

  • continuous batching with a bounded concurrent sequence count
  • KV cache block accounting, queueing when blocks run out, including head-of-line blocking
  • automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
  • per-token latency that grows with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics under vLLM's real metric names
  • failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

  • router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
  • router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
  • router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
  • router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

  • llmgw server, llmgw agent and llmgw ctl are stubs.
  • The scheduler has no HTTP layer yet; nothing routes real requests.
  • No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
  • Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

  • The project language is English throughout: code, comments, tests, commits and the design document.
  • Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.