Files
llmgw/CHANGELOG.md
T
ilkerandClaude Opus 5 f0134a2f95
CI / test (push) Successful in 24s
Phase 3 (2/3): catalog and reconciler
The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.

- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
  fallbacks resolve at load rather than at request time, because a typo in a
  fallback should stop a deployment instead of surfacing during the outage when
  the fallback is finally needed. The node/GPU/TP arithmetic is checked here
  too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
  already allocated and several from one snapshot would be handed the same GPUs.
  Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
  needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
  that will never complete while holding GPUs, so every shard is stopped and the
  replica rebuilt, with a jittered exponential backoff carried on the model
  rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
  network blip must not tear down replicas that are still serving, and
  rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.

The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.

Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:12:06 +02:00

293 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Changelog
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
is only released once its **exit criterion** is met, not when its code lands.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | in progress |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased — phases 2 and 3 in progress
Agent groundwork. Everything here works without GPUs and is covered by tests;
what is missing is the control plane connection, so there is no release yet.
### Added
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
without IB is not an error — it is a single-node island.
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
would force the whole binary, server included, to be built with a C
toolchain. Health flags anything that should keep a card out of placement:
uncorrected ECC errors, and memory already held by another process.
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
always match the marketing name: an L40S is sold as 48GB and reports 44.99
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
box, because the number decides whether a model fits.
- **Engine adapters** for vLLM and SGLang: command and environment
construction, kept free of process handling so it is testable without GPUs.
vLLM supports both Ray and multiprocessing launch modes, because which one
works depends on the version.
- **NCCL environment generation**, the part plan §5.3 marks critical.
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
NCCL pick an adapter on another fabric where the peer is unreachable, and the
job hangs rather than failing. Bootstrap is pinned to the management
interface for the same reason.
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
compute capability 8.9 rather than silently falling back to a slower path.
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
process group and are stopped as a group, because engines fork workers that
hold GPU memory and outlive a signal sent only to the leader. Bounded log
tail, a state machine that separates Starting from Loading (a 70B model takes
minutes, and an operator needs to know that is normal), and orphan adoption
through pid files fingerprinted with the command line — pids get reused, and
adopting an unrelated process is worse than leaking the original.
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
says the weights loaded. Routing on `/health` alone sends traffic into a model
that is still loading.
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
as a table or JSON. Run it before adding a node: it is the fastest way to find
a card another process is using, or a port with no subnet manager.
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
a tensor-parallel group off the slow path inside a node. `Discover` falls back
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
topology. CI compiles the tagged build so it cannot rot unnoticed.
### Not verified
The NVML path compiles but has never run against a driver — there is no GPU
machine to try it on. Treat the NVLink domains it reports as unverified until
someone runs `llmgw agent inventory` on a real node.
### Phase 3 exit criterion: half met
`TestLostNodeMovesReplicas` covers the reconciliation half — a node going
silent becomes unreachable, then lost, and its replica is rebuilt on a survivor.
The other half, the router continuing to serve throughout, needs the
`EndpointSource` wiring and the agent stream.
### Still missing for phase 2
- The agent-to-control-plane stream. It pairs with phase 3's control plane
rather than standing alone, so `llmgw agent` itself is still a stub.
### Phase 3: placement
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
their fabric attachments, and what is allocated where. A re-registering agent
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
without doing so either. A GPU with an ECC fault or another process on it is
not counted as free. Replicas are released whole, because they fail whole.
- **Placement** (`internal/control/placement`): picks the highest-priority
profile that fits and explains every candidate it rejected. Two constraints
are enforced structurally rather than checked afterwards — there is no code
path that can combine nodes from two fabrics, and only GPUs of the profile's
class are ever considered.
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
large models that cannot be split. Multi-node replicas get the opposite
treatment: they are the least movable thing in the cluster, so they take the
emptiest machines.
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
domain where the topology is known. Without the NVML build it falls back to
the lowest free indices, which is usually right but only by accident of how
vendors number slots.
- **Property-based invariant tests** (plan §13): 400 randomly generated
clusters, checking on every placement that a multi-node replica lives on one
fabric, that each shard receives only that fabric's adapters, and that no
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
The table-driven tests cover the cases someone thought of; this covers the
ones nobody did.
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
rather than at request time — a typo in a fallback should stop a deployment,
not surface during the outage when the fallback is finally needed. The
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
load instead of minutes into model loading.
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
loop. It places one replica per pass, because placement depends on what is
already allocated and starting several from one snapshot would hand them the
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
vLLM under Ray needs its workers attached before the head serves.
- **Replica failure is whole-replica failure.** One dead shard makes the others
useless — they wait on collectives that will never complete while holding
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
exponential backoff carried on the model rather than the replica it just
destroyed.
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
blip must not tear down replicas that are still serving, and rebuilding a
multi-node replica costs tens of minutes of model loading.
- **Scale-down drains before stopping**, so in-flight requests finish instead of
being cut off mid-stream.
## v0.3.0 — 2026-09-16
Flow control. A batch workload no longer starves interactive traffic.
### Added
- **Admission control** (`internal/router/admission`), inserted between quota
checks and scheduling. It answers *when* a request may proceed; the scheduler
still answers *where* it goes. Keeping them apart is the point: an engine
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
no amount of endpoint selection rescues the request behind it.
- **Saturation detector.** One number per pool, the worse of queue depth and KV
occupancy. Either signal alone misleads — a pool can have an empty queue and a
nearly full cache, or the reverse.
- **Priority bands.** Each band has a saturation ceiling; above it, that band
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
"flow up to the healthy limit"; above 1 means never wait.
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
one caller sending a thousand requests occupies one slot in the rotation
rather than a thousand.
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
(504), because sending the GPU work the client has already abandoned helps
nobody. Cancelled requests leave the queue instead of holding capacity.
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
`x-llmgw-queue-ms` when a request waited.
### Phase 1.5 exit criterion, measured in `test/e2e`
While 500 concurrent bronze requests run continuously, interactive p99 in the
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
With admission switched off the same load pushes it to **55–60x**. Both
directions are asserted, so the test fails if admission regresses *and* if the
load generator ever stops being a real load.
### Two lags, two different fixes
Getting this right took longer than the queueing itself:
- Engine metrics are scraped, so they lag by an interval. The router's inflight
count does not — but it includes *running* requests, and treating those as
queue depth makes a healthy engine look overloaded every interval. What is
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
the metric contract, so it is learned as the high-water mark of observed
`running`.
- Between admission granting a request and the router dispatching it, the
request is invisible to every signal. A simultaneous burst therefore passed
the check en masse — exactly the batch arrival pattern this layer exists to
survive. The controller now counts its own outstanding leases against
dispatched inflight.
### Known limitations
- The band comes from a trusted header until authentication arrives in v1.0.0.
Anyone who can reach the port can promote themselves to platinum.
- The tenant key is the session id, for the same reason.
- Displacement (dropping queued low-band work to make room) is not implemented;
the plan defers it, and bounded queues plus TTL cover most of the need.
## v0.2.0 — 2026-09-16
The router serves traffic. `llmgw server` is real: it fronts a static set of
engine replicas with an OpenAI-compatible API.
### Added
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
propagates cancellation: when a client disconnects, the upstream request is
aborted so the GPU stops working. Separate deadlines for dial, response
headers, first token, inter-token idle and total duration — a wedged engine is
indistinguishable from a slow one without the idle watchdog.
- **Retry and fallback.** Another endpoint is tried only while nothing has
reached the client; once bytes are out, splicing two responses together would
be worse than the short stream. Models can name fallback models, and the
response says which one served it via `x-llmgw-served-model`.
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
Only routing-relevant fields are parsed; the rest of the body passes through
untouched.
- **Usage accounting that does not depend on the client.** The router can ask
engines for usage the client never requested, then drop that chunk on the way
back, so token counts are recorded either way.
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
duplicate endpoint ids and aliases that would shadow another model — both
would otherwise misroute silently at request time.
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
values and per-filter drop counts. Routing profile weights cannot be tuned
without seeing what the scorers actually produced. Served on a separate admin
listener alongside health and pprof.
### Phase 1 exit criterion, measured in `test/e2e`
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
- Every client disconnecting frees the KV blocks on every engine.
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
looser bound by default since a shared machine cannot hold 2 ms reliably; set
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
### Known limitations
- No admission control, so a batch workload can still starve interactive
traffic. That is v0.3.0, and it is the most valuable thing still missing.
- The endpoint list is static; the control plane arrives in v0.5.0.
- No authentication, rate limiting or budgets yet (v1.0.0).
- Prompt length is estimated as bytes/4. It becomes exact only with a real
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
## v0.1.0 — 2026-09-16
First tagged version. It is useful for exactly one thing: developing and testing
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
returns "not implemented".
### Added
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
that models the behaviours a router has to cope with:
- continuous batching with a bounded concurrent sequence count
- KV cache block accounting, queueing when blocks run out, including
head-of-line blocking
- automatic prefix caching (chained block hashes over a block-level LRU) and its
effect on time to first token
- per-token latency that grows with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` under vLLM's real metric names
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
**Routing primitives** — libraries, not yet wired into a server:
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
- `router/endpoint`: live endpoint registry with inflight counting, staleness
detection and drain rate derived from engine counters.
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
endpoints missing any of the mandatory three. The metric name mapping is
overridable, because names drift between engine versions.
- `router/breaker`: cooldown and circuit breaking in one component, with
exponential backoff and half-open probes.
### Known limitations
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
- The scheduler has no HTTP layer yet; nothing routes real requests.
- No admission control, so a batch workload can still starve interactive traffic
once the router exists. That is what v0.3.0 is for.
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
### Notes
- The project language is English throughout: code, comments, tests, commits and
the design document.
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.