llmgw/CHANGELOG.md

Changelog

Versions map onto the roadmap phases in llmgw-plan.md §11. A phase is only released once its exit criterion is met, not when its code lands.

Version	Phase	Scope	State
v0.1.0	0	Skeleton and sim-engine	released
v0.2.0	1	Static router: streaming proxy, filter/score/pick	released
v0.3.0	1.5	Admission control: priority bands, saturation detector	released
v0.4.0	2	Agent: GPU/IB discovery, single-node engines	released as part of v0.5.0
v0.5.0	3	Control plane: catalog, placement, reconciler	planned
v0.6.0	4	Multi-node replicas and fabric awareness	planned
v1.0.0	5	Tenancy, prefix affinity, benchmarking	planned

Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period.

Unreleased

llama.cpp, and CPU profiles

A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI API without a GPU, so placement, supervision, scraping, scheduling and admission can all be exercised on ordinary machines.

A profile may now omit gpu_class, which is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it — for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. CPU replicas are spread across nodes rather than bin packed: unlike GPUs there is no allocation to stop two of them sharing a machine's cores.

Metric mappings are per engine

Running, Waiting and KVCacheUsage now have a vocabulary per engine rather than one for the cluster. A mixed fleet has two names for the same three numbers, and a single global mapping left half of it unmeasurable.

A mapping that leaves a metric unnamed now means the engine does not publish it, and the field stays zero. Naming one the engine does not serve is still a contract violation. Running and Waiting can never be omitted.

Mutual TLS on the agent stream

A private CA (llmgw ctl cert init-ca | issue-server | issue-client | inspect), certificates on both ends, and the control plane refusing any agent whose claimed node id does not match its certificate's common name. Public authorities cannot supply these: they stopped issuing the clientAuth usage.

Plaintext HTTP/2 remains the default. agent_tls: on the server and tls: on the agent turn it on.

Fixes

• An agent told to stop did not stop. Cancelling the context is not enough — HTTP/2 watches it while the request body is being written, and a bidirectional stream spends its life blocked on the read side. Measured on a deployed node: systemctl stop took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
• An agent could not explain why it was refused. The session cancelled its own context and then asked that context whether it was shutting down, so every stream error read as a clean close: no message, and the backoff reset.
• A broken metric contract was invisible. It was logged per scrape at debug level, so a cluster could look healthy while every routing decision was made blind. It is now a warning, once, with a matching line when it clears.

v0.5.0 — 2026-09-16

Phases 2 and 3. The cluster runs itself: agents report their hardware, the control plane places replicas, and ctl makes all of it visible.

This release covers both phases because their exit criteria depended on the same piece — the agent stream — and neither was demonstrable without it.

ctl

llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain

• nodes surfaces unhealthy GPUs and why they are excluded, which is usually the answer when a node looks idle but nothing schedules onto it.
• explain-placement prints every candidate considered and the reason it was rejected. "Unschedulable" on its own says nothing about which of six constraints bit.
• drain is a rolling move, not a stop: the replica is drained, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. Bounded by a grace period, because draining the only node that can host a model would otherwise never finish and leave it undrainable.
• cordon is the weaker operation: no new placement, nothing disturbed.

The management API is on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes.

ctl logs is not here. It needs a request/response pair the agent protocol does not have. The failure path already ships an engine log tail with every status report, so the case that matters most is covered; reading a healthy shard's output on demand is not.

Unreleased — phase 4 in progress

Agent groundwork. Everything here works without GPUs and is covered by tests; what is missing is the control plane connection, so there is no release yet.

Added

• IB discovery (internal/agent/discovery): reads /sys/class/infiniband, parses the formats sysfs actually uses ("4: ACTIVE", "200 Gb/sec (4X HDR)", hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node without IB is not an error — it is a single-node island.

• GPU discovery via nvidia-smi, the cgo-free path. NVML needs cgo, which would force the whole binary, server included, to be built with a C toolchain. Health flags anything that should keep a card out of placement: uncorrected ECC errors, and memory already held by another process.

• GPU class derivation. Sizes come from the driver in GiB, which does not always match the marketing name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g. Consistency matters more than matching the box, because the number decides whether a model fits.

• Engine adapters for vLLM and SGLang: command and environment construction, kept free of process handling so it is testable without GPUs. vLLM supports both Ray and multiprocessing launch modes, because which one works depends on the version.

• NCCL environment generation, the part plan §5.3 marks critical. NCCL_IB_HCA lists only the chosen fabric's adapters — leaving it unset lets NCCL pick an adapter on another fabric where the peer is unreachable, and the job hangs rather than failing. Bootstrap is pinned to the management interface for the same reason.

• Validation before launch: a tensor-parallel group may not mix GPU types, the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below compute capability 8.9 rather than silently falling back to a slower path.

• Supervisor (internal/agent/supervisor): processes run in their own process group and are stopped as a group, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader. Bounded log tail, a state machine that separates Starting from Loading (a 70B model takes minutes, and an operator needs to know that is normal), and orphan adoption through pid files fingerprinted with the command line — pids get reused, and adopting an unrelated process is worse than leaking the original.

• Readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a model that is still loading.

• llmgw agent inventory reports what the scheduler would see on a node, as a table or JSON. Run it before adding a node: it is the fastest way to find a card another process is using, or a port with no subnet manager.

• NVML backend behind the nvml build tag, which adds NVLink topology. Nothing else exposes which GPUs share an NVLink domain, and that is what keeps a tensor-parallel group off the slow path inside a node. Discover falls back to nvidia-smi if NVML fails, and says so loudly rather than silently losing the topology. CI compiles the tagged build so it cannot rot unnoticed.

Not verified

The NVML path compiles but has never run against a driver — there is no GPU machine to try it on. Treat the NVLink domains it reports as unverified until someone runs llmgw agent inventory on a real node.

• Control plane wiring (internal/control): Ready replicas become router endpoints, with no static configuration involved. The router keeps live state for endpoints that survive an update — inflight counts, breaker state, drain rate — so a replica that was already serving does not start from cold because another one moved. The sink is an interface, so when the router and control plane split into separate processes this becomes a watch stream and nothing above it changes.

Phase 3 exit criterion, measured in test/e2e

A node hosting a live replica is killed while four clients stream continuously. The control plane marks it unreachable, then lost, rebuilds the replica on a survivor, and republishes. Across three runs: ~370 requests completed, zero failures, and no replica remained on the dead node.

This proves the control logic, not the transport: the agent fleet is simulated, and the real path still needs the stream.

A real bug the race detector caught

startReplica read the replica's fields after publishing it to the map, and scaleUp walked replica pointers outside the lock. Both were racing with agent status reports, which arrive from another goroutine and mutate exactly those fields. Under the simulated fleet the reports came back fast enough to hit the window every time. Everything is snapshotted under the lock now and the functions work from values.

The agent stream

• Protocol (api/proto/llmgw/v1): one bidirectional connect-go stream carrying registration, heartbeats and shard status upward, and start, stop and drain commands downward. The agent dials out, because GPU nodes sit behind firewalls far more often than a control plane does and an outbound connection needs no inbound rule on every node in the fleet.
• Registration carries the whole inventory, so there is no separate discovery call and a reconnecting agent refreshes what the scheduler knows. The fabric labels travel with it — they cannot be inferred from the wire, so losing them in transit would silently disable multi-node placement.
• Agent service (internal/control/agentsvc) is both the stream handler and the reconciler's dispatcher: it holds the open streams, so it is the only thing that can deliver a command to a node.
• Agent (internal/agent) validates a profile before launching anything — an engine that refuses a profile does so after minutes of loading — then supervises the process and reports every transition, including the engine's output tail on failure.
• Reconnects forever with jittered backoff. A control plane restart must not require touching every GPU node, and a fleet must not reconnect in lockstep and knock the control plane over again. Engines keep serving throughout: the router talks to them directly.
• llmgw agent is real, and llmgw server runs the control plane whenever any model declares a profile. The static endpoint list remains for router-only runs against sim-engines.

Verified with real binaries

Server and agent as separate processes over a real stream: the agent registers, a replica is placed, an engine is launched through the vLLM adapter's actual argument construction, streaming traffic is served through the router, and killing the agent moves the node through unreachable to lost with placement explaining exactly why it can no longer run anything.

Surviving a control plane restart

Not persisting replica state left one hole that had to be closed before it mattered: restarting the control plane orphaned live engines. They kept holding GPUs the inventory believed were free, so the next placement failed to allocate with an error naming the new process rather than the old one.

The fix is on the agent, not in a database, and it cannot disagree with reality the way a stale row can:

• The supervisor now writes the full shard spec, not just a pid and command line. A pid identifies a process; the spec is what lets it be described back to the control plane and matched to a replica.
• Adopt takes over a still-running engine from a previous agent. It cannot be waited on — it is not this process's child — so it is watched by polling, and stopped by signalling its group directly. Its earlier output is gone, and the log says so rather than presenting emptiness as silence.
• Every Register, not only the first, reports what the agent is running. A reconnect may be a control plane restart and has to recover the same way.
• The reconciler rebuilds replicas from those reports and reclaims their GPUs. Anything whose model or profile left the catalog is stopped: nothing will route to it and it is holding cards.
• An engine the agent cannot identify — a corrupt state file — is killed. A process holding GPUs that nothing can describe is worse than no process, because placement will hand those GPUs out again.

One mechanism covers agent restarts and control plane restarts.

A test that was passing for no reason

TestAgentReconnects used httptest.CloseClientConnections, which does nothing to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops tracking hijacked connections. The agent never reconnected and the assertion — on the connection map, which still held a stale entry — passed anyway. Tests now sever connections through their own listener and count opened streams.

Multi-node launch

Placement and environment generation had landed early, which made multi-node look further along than it was: a multi-node replica could not come up at all. Three things were missing, and all three came down to two assumptions that do not hold once a replica spans nodes.

Not every rank serves HTTP. The agent health-checked every shard at its own port, but only the leader answers. Workers timed out, reported Failed, and took the replica with them. Engine.ServesHTTP(spec) now says which ranks serve; a worker is judged by staying alive instead. That is weaker evidence on purpose — the replica's real readiness is the leader answering, which only happens once every worker has joined.

A shard is not always one process. Engine.Commands(spec) returns a sequence: every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's exit. vLLM under Ray is why — the leader runs a Ray head and then the server against it, and a worker runs only the join, with no vLLM process at all. RayCommand had been written and tested but never called, so launch_mode: ray would have run vllm serve against a cluster that did not exist.

A command can also require the rendezvous point to be listening before the next one starts, bounded by a timeout so a head that never appears fails the shard rather than hanging it. Sidecars are killed with the shard: a Ray head left behind holds the port the next replica needs.

Start order was backwards. Workers were started first, on the theory that the head should wait for them. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to and spends its startup budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own.

None of this is verified against real engines. It is tested with stub processes; the Ray sequencing and which launch mode a given vLLM version wants can only be confirmed on hardware.

Still missing for phase 2

• mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted network.

Phase 3: placement

• Cluster inventory (internal/control/inventory): nodes, their hardware, their fabric attachments, and what is allocated where. A re-registering agent cannot clear an operator's cordon, and a heartbeat revives an unreachable node without doing so either. A GPU with an ECC fault or another process on it is not counted as free. Replicas are released whole, because they fail whole.
• Placement (internal/control/placement): picks the highest-priority profile that fits and explains every candidate it rejected. Two constraints are enforced structurally rather than checked afterwards — there is no code path that can combine nodes from two fabrics, and only GPUs of the profile's class are ever considered.
• Bin packing fills partly used nodes first, so whole nodes stay free for large models that cannot be split. Multi-node replicas get the opposite treatment: they are the least movable thing in the cluster, so they take the emptiest machines.
• NVLink-aware GPU selection keeps a tensor-parallel group inside one NVLink domain where the topology is known. Without the NVML build it falls back to the lowest free indices, which is usually right but only by accident of how vendors number slots.
• Property-based invariant tests (plan §13): 400 randomly generated clusters, checking on every placement that a multi-node replica lives on one fabric, that each shard receives only that fabric's adapters, and that no placement mixes GPU classes or uses an unhealthy or already-allocated GPU. The table-driven tests cover the cases someone thought of; this covers the ones nobody did.
• Catalog (internal/control/catalog): models, profiles, aliases, fallbacks and scaling bounds. Aliases and fallbacks are resolved at load rather than at request time — a typo in a fallback should stop a deployment, not surface during the outage when the fallback is finally needed. The node/GPU/TP arithmetic is checked here too, so a bad profile fails at config load instead of minutes into model loading.
• Reconciler (internal/control/reconciler): the desired-versus-actual loop. It places one replica per pass, because placement depends on what is already allocated and starting several from one snapshot would hand them the same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and vLLM under Ray needs its workers attached before the head serves.
• Replica failure is whole-replica failure. One dead shard makes the others useless — they wait on collectives that will never complete while holding GPUs — so the reconciler stops every shard and rebuilds, with a jittered exponential backoff carried on the model rather than the replica it just destroyed.
• Node expiry in two stages. Unreachable first, Lost later. A brief network blip must not tear down replicas that are still serving, and rebuilding a multi-node replica costs tens of minutes of model loading.
• Scale-down drains before stopping, so in-flight requests finish instead of being cut off mid-stream.

v0.3.0 — 2026-09-16

Flow control. A batch workload no longer starves interactive traffic.

Added

• Admission control (internal/router/admission), inserted between quota checks and scheduling. It answers when a request may proceed; the scheduler still answers where it goes. Keeping them apart is the point: an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no amount of endpoint selection rescues the request behind it.
• Saturation detector. One number per pool, the worse of queue depth and KV occupancy. Either signal alone misleads — a pool can have an empty queue and a nearly full cache, or the reverse.
• Priority bands. Each band has a saturation ceiling; above it, that band waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means "flow up to the healthy limit"; above 1 means never wait.
• Tenant fairness. Within a band, dispatch turns rotate between tenants, so one caller sending a thousand requests occupies one slot in the rotation rather than a thousand.
• Bounded queues. Capacity per band (immediate 503 when full) and a TTL (504), because sending the GPU work the client has already abandoned helps nobody. Cancelled requests leave the queue instead of holding capacity.
• Metrics: llmgw_admission_wait_seconds, llmgw_admission_rejected_total, llmgw_admission_queue_depth, llmgw_pool_saturation. Responses carry x-llmgw-queue-ms when a request waited.

Phase 1.5 exit criterion, measured in test/e2e

While 500 concurrent bronze requests run continuously, interactive p99 in the platinum band stays at 1.04–1.24x its idle value, against a 1.5x budget. With admission switched off the same load pushes it to 55–60x. Both directions are asserted, so the test fails if admission regresses and if the load generator ever stops being a real load.

Two lags, two different fixes

Getting this right took longer than the queueing itself:

• Engine metrics are scraped, so they lag by an interval. The router's inflight count does not — but it includes running requests, and treating those as queue depth makes a healthy engine look overloaded every interval. What is genuinely queued is inflight − batch capacity, and batch capacity is not in the metric contract, so it is learned as the high-water mark of observed running.
• Between admission granting a request and the router dispatching it, the request is invisible to every signal. A simultaneous burst therefore passed the check en masse — exactly the batch arrival pattern this layer exists to survive. The controller now counts its own outstanding leases against dispatched inflight.

Known limitations

• The band comes from a trusted header until authentication arrives in v1.0.0. Anyone who can reach the port can promote themselves to platinum.
• The tenant key is the session id, for the same reason.
• Displacement (dropping queued low-band work to make room) is not implemented; the plan defers it, and bounded queues plus TTL cover most of the need.

v0.2.0 — 2026-09-16

The router serves traffic. llmgw server is real: it fronts a static set of engine replicas with an OpenAI-compatible API.

Added

• Streaming proxy. Relays SSE event by event with a flush after each, and propagates cancellation: when a client disconnects, the upstream request is aborted so the GPU stops working. Separate deadlines for dial, response headers, first token, inter-token idle and total duration — a wedged engine is indistinguishable from a slow one without the idle watchdog.
• Retry and fallback. Another endpoint is tried only while nothing has reached the client; once bytes are out, splicing two responses together would be worse than the short stream. Models can name fallback models, and the response says which one served it via x-llmgw-served-model.
• OpenAI surface. /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models, with aliases resolved from the catalog. Only routing-relevant fields are parsed; the rest of the body passes through untouched.
• Usage accounting that does not depend on the client. The router can ask engines for usage the client never requested, then drop that chunk on the way back, so token counts are recorded either way.
• Configuration (internal/config): YAML through koanf, overlaid on defaults, with LLMGW_-prefixed environment overrides. Validation rejects duplicate endpoint ids and aliases that would shadow another model — both would otherwise misroute silently at request time.
• Metrics (plan §8): request counts, duration, TTFT, tokens, plus per-scorer values and per-filter drop counts. Routing profile weights cannot be tuned without seeing what the scorers actually produced. Served on a separate admin listener alongside health and pprof.

Phase 1 exit criterion, measured in test/e2e

• 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
• Every client disconnecting frees the KV blocks on every engine.
• Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a looser bound by default since a shared machine cannot hold 2 ms reliably; set LLMGW_E2E_STRICT=1 on quiet hardware to enforce the real target.

Known limitations

• No admission control, so a batch workload can still starve interactive traffic. That is v0.3.0, and it is the most valuable thing still missing.
• The endpoint list is static; the control plane arrives in v0.5.0.
• No authentication, rate limiting or budgets yet (v1.0.0).
• Prompt length is estimated as bytes/4. It becomes exact only with a real tokenizer, which the plan defers to v2 along with tier 2 prefix routing.

v0.1.0 — 2026-09-16

First tagged version. It is useful for exactly one thing: developing and testing a router without GPUs. The gateway does not serve traffic yet — llmgw server returns "not implemented".

Added

sim-engine (llmgw sim-engine) — an OpenAI-compatible fake inference engine that models the behaviours a router has to cope with:

• continuous batching with a bounded concurrent sequence count
• KV cache block accounting, queueing when blocks run out, including head-of-line blocking
• automatic prefix caching (chained block hashes over a block-level LRU) and its effect on time to first token
• per-token latency that grows with batch size
• dropping a request and freeing its blocks when the client disconnects
• /metrics under vLLM's real metric names
• failure injection (--fail-rate) and deterministic runs (--seed)

Routing primitives — libraries, not yet wired into a server:

• router/scheduler: the filter → score → pick pipeline. Scorers for queue depth, KV occupancy, router-side inflight, drain rate, capacity weight and session affinity (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
• router/endpoint: live endpoint registry with inflight counting, staleness detection and drain rate derived from engine counters.
• router/scrape: parses the engine metric contract (plan §7.6) and refuses endpoints missing any of the mandatory three. The metric name mapping is overridable, because names drift between engine versions.
• router/breaker: cooldown and circuit breaking in one component, with exponential backoff and half-open probes.

Known limitations

• llmgw server, llmgw agent and llmgw ctl are stubs.
• The scheduler has no HTTP layer yet; nothing routes real requests.
• No admission control, so a batch workload can still starve interactive traffic once the router exists. That is what v0.3.0 is for.
• Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.

Notes

• The project language is English throughout: code, comments, tests, commits and the design document.
• Plan §7.6 corrects an error from plan v0.1: the KV usage metric is vllm:kv_cache_usage_perc, not vllm:gpu_cache_usage_perc.