CI / test (push) Successful in 58s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
502 lines
27 KiB
Markdown
502 lines
27 KiB
Markdown
# Changelog
|
||
|
||
Versions map onto the roadmap phases in [llmgw-plan.md](llmgw-plan.md) §11. A phase
|
||
is only released once its **exit criterion** is met, not when its code lands.
|
||
|
||
| Version | Phase | Scope | State |
|
||
|---|---|---|---|
|
||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||
|
||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||
APIs change between minors without a deprecation period.
|
||
|
||
## Unreleased
|
||
|
||
### llama.cpp, and CPU profiles
|
||
|
||
A third engine adapter, and the reason for it: llama.cpp serves the same OpenAI
|
||
API without a GPU, so placement, supervision, scraping, scheduling and admission
|
||
can all be exercised on ordinary machines.
|
||
|
||
A profile may now omit `gpu_class`, which is how it says it runs on the CPU. Only
|
||
engines that can actually serve without a card may omit it — for vLLM an omitted
|
||
class stays an error, because otherwise a typo becomes a replica placed on a
|
||
machine with no hardware and diagnosed minutes later during model loading. CPU
|
||
replicas are spread across nodes rather than bin packed: unlike GPUs there is no
|
||
allocation to stop two of them sharing a machine's cores.
|
||
|
||
### Metric mappings are per engine
|
||
|
||
`Running`, `Waiting` and `KVCacheUsage` now have a vocabulary per engine rather
|
||
than one for the cluster. A mixed fleet has two names for the same three numbers,
|
||
and a single global mapping left half of it unmeasurable.
|
||
|
||
A mapping that leaves a metric unnamed now means the engine does not publish it,
|
||
and the field stays zero. Naming one the engine does not serve is still a contract
|
||
violation. `Running` and `Waiting` can never be omitted.
|
||
|
||
### Mutual TLS on the agent stream
|
||
|
||
A private CA (`llmgw ctl cert init-ca | issue-server | issue-client | inspect`),
|
||
certificates on both ends, and the control plane refusing any agent whose claimed
|
||
node id does not match its certificate's common name. Public authorities cannot
|
||
supply these: they stopped issuing the `clientAuth` usage.
|
||
|
||
Plaintext HTTP/2 remains the default. `agent_tls:` on the server and `tls:` on the
|
||
agent turn it on.
|
||
|
||
### Fixes
|
||
|
||
- **An agent told to stop did not stop.** Cancelling the context is not enough —
|
||
HTTP/2 watches it while the request body is being written, and a bidirectional
|
||
stream spends its life blocked on the read side. Measured on a deployed node:
|
||
`systemctl stop` took 90 seconds and ended in SIGKILL. Now 0.01 seconds.
|
||
- **An agent could not explain why it was refused.** The session cancelled its own
|
||
context and then asked that context whether it was shutting down, so every
|
||
stream error read as a clean close: no message, and the backoff reset.
|
||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||
level, so a cluster could look healthy while every routing decision was made
|
||
blind. It is now a warning, once, with a matching line when it clears.
|
||
|
||
## v0.5.0 — 2026-09-16
|
||
|
||
Phases 2 and 3. The cluster runs itself: agents report their hardware, the
|
||
control plane places replicas, and `ctl` makes all of it visible.
|
||
|
||
This release covers both phases because their exit criteria depended on the same
|
||
piece — the agent stream — and neither was demonstrable without it.
|
||
|
||
### ctl
|
||
|
||
```
|
||
llmgw ctl nodes | models | replicas | explain-placement | cordon | uncordon | drain
|
||
```
|
||
|
||
- `nodes` surfaces unhealthy GPUs and why they are excluded, which is usually
|
||
the answer when a node looks idle but nothing schedules onto it.
|
||
- `explain-placement` prints every candidate considered and the reason it was
|
||
rejected. "Unschedulable" on its own says nothing about which of six
|
||
constraints bit.
|
||
- `drain` is a rolling move, not a stop: the replica is drained, a replacement
|
||
is placed elsewhere, and the original is stopped only once the replacement is
|
||
serving. Bounded by a grace period, because draining the only node that can
|
||
host a model would otherwise never finish and leave it undrainable.
|
||
- `cordon` is the weaker operation: no new placement, nothing disturbed.
|
||
|
||
The management API is on the admin listener, not the client-facing one. The two
|
||
have different audiences: one serves inference to whoever holds an API key, the
|
||
other lets a caller cordon nodes.
|
||
|
||
`ctl logs` is not here. It needs a request/response pair the agent protocol does
|
||
not have. The failure path already ships an engine log tail with every status
|
||
report, so the case that matters most is covered; reading a healthy shard's
|
||
output on demand is not.
|
||
|
||
## Unreleased — phase 4 in progress
|
||
|
||
Agent groundwork. Everything here works without GPUs and is covered by tests;
|
||
what is missing is the control plane connection, so there is no release yet.
|
||
|
||
### Added
|
||
|
||
- **IB discovery** (`internal/agent/discovery`): reads `/sys/class/infiniband`,
|
||
parses the formats sysfs actually uses (`"4: ACTIVE"`, `"200 Gb/sec (4X HDR)"`,
|
||
hex or decimal LIDs), and distinguishes native InfiniBand from RoCE. A node
|
||
without IB is not an error — it is a single-node island.
|
||
- **GPU discovery** via `nvidia-smi`, the cgo-free path. NVML needs cgo, which
|
||
would force the whole binary, server included, to be built with a C
|
||
toolchain. Health flags anything that should keep a card out of placement:
|
||
uncorrected ECC errors, and memory already held by another process.
|
||
- **GPU class derivation.** Sizes come from the driver in GiB, which does not
|
||
always match the marketing name: an L40S is sold as 48GB and reports 44.99
|
||
GiB, so its class is `l40s-45g`. Consistency matters more than matching the
|
||
box, because the number decides whether a model fits.
|
||
- **Engine adapters** for vLLM and SGLang: command and environment
|
||
construction, kept free of process handling so it is testable without GPUs.
|
||
vLLM supports both Ray and multiprocessing launch modes, because which one
|
||
works depends on the version.
|
||
- **NCCL environment generation**, the part plan §5.3 marks critical.
|
||
`NCCL_IB_HCA` lists only the chosen fabric's adapters — leaving it unset lets
|
||
NCCL pick an adapter on another fabric where the peer is unreachable, and the
|
||
job hangs rather than failing. Bootstrap is pinned to the management
|
||
interface for the same reason.
|
||
- **Validation before launch**: a tensor-parallel group may not mix GPU types,
|
||
the node/GPU/TP arithmetic has to add up, and an FP8 profile is refused below
|
||
compute capability 8.9 rather than silently falling back to a slower path.
|
||
- **Supervisor** (`internal/agent/supervisor`): processes run in their own
|
||
process group and are stopped as a group, because engines fork workers that
|
||
hold GPU memory and outlive a signal sent only to the leader. Bounded log
|
||
tail, a state machine that separates Starting from Loading (a 70B model takes
|
||
minutes, and an operator needs to know that is normal), and orphan adoption
|
||
through pid files fingerprinted with the command line — pids get reused, and
|
||
adopting an unrelated process is worse than leaking the original.
|
||
- **Readiness** is two questions: `/health` says the process is up, `/v1/models`
|
||
says the weights loaded. Routing on `/health` alone sends traffic into a model
|
||
that is still loading.
|
||
- **`llmgw agent inventory`** reports what the scheduler would see on a node,
|
||
as a table or JSON. Run it before adding a node: it is the fastest way to find
|
||
a card another process is using, or a port with no subnet manager.
|
||
|
||
- **NVML backend** behind the `nvml` build tag, which adds NVLink topology.
|
||
Nothing else exposes which GPUs share an NVLink domain, and that is what keeps
|
||
a tensor-parallel group off the slow path inside a node. `Discover` falls back
|
||
to nvidia-smi if NVML fails, and says so loudly rather than silently losing the
|
||
topology. CI compiles the tagged build so it cannot rot unnoticed.
|
||
|
||
### Not verified
|
||
|
||
The NVML path compiles but has never run against a driver — there is no GPU
|
||
machine to try it on. Treat the NVLink domains it reports as unverified until
|
||
someone runs `llmgw agent inventory` on a real node.
|
||
|
||
- **Control plane wiring** (`internal/control`): Ready replicas become router
|
||
endpoints, with no static configuration involved. The router keeps live state
|
||
for endpoints that survive an update — inflight counts, breaker state, drain
|
||
rate — so a replica that was already serving does not start from cold because
|
||
another one moved. The sink is an interface, so when the router and control
|
||
plane split into separate processes this becomes a watch stream and nothing
|
||
above it changes.
|
||
|
||
### Phase 3 exit criterion, measured in `test/e2e`
|
||
|
||
A node hosting a live replica is killed while four clients stream continuously.
|
||
The control plane marks it unreachable, then lost, rebuilds the replica on a
|
||
survivor, and republishes. Across three runs: **~370 requests completed, zero
|
||
failures**, and no replica remained on the dead node.
|
||
|
||
This proves the control logic, not the transport: the agent fleet is simulated,
|
||
and the real path still needs the stream.
|
||
|
||
### A real bug the race detector caught
|
||
|
||
`startReplica` read the replica's fields after publishing it to the map, and
|
||
`scaleUp` walked replica pointers outside the lock. Both were racing with agent
|
||
status reports, which arrive from another goroutine and mutate exactly those
|
||
fields. Under the simulated fleet the reports came back fast enough to hit the
|
||
window every time. Everything is snapshotted under the lock now and the
|
||
functions work from values.
|
||
|
||
### The agent stream
|
||
|
||
- **Protocol** (`api/proto/llmgw/v1`): one bidirectional connect-go stream
|
||
carrying registration, heartbeats and shard status upward, and start, stop and
|
||
drain commands downward. The agent dials out, because GPU nodes sit behind
|
||
firewalls far more often than a control plane does and an outbound connection
|
||
needs no inbound rule on every node in the fleet.
|
||
- **Registration carries the whole inventory**, so there is no separate
|
||
discovery call and a reconnecting agent refreshes what the scheduler knows.
|
||
The fabric labels travel with it — they cannot be inferred from the wire, so
|
||
losing them in transit would silently disable multi-node placement.
|
||
- **Agent service** (`internal/control/agentsvc`) is both the stream handler and
|
||
the reconciler's dispatcher: it holds the open streams, so it is the only
|
||
thing that can deliver a command to a node.
|
||
- **Agent** (`internal/agent`) validates a profile before launching anything —
|
||
an engine that refuses a profile does so after minutes of loading — then
|
||
supervises the process and reports every transition, including the engine's
|
||
output tail on failure.
|
||
- **Reconnects forever with jittered backoff.** A control plane restart must not
|
||
require touching every GPU node, and a fleet must not reconnect in lockstep
|
||
and knock the control plane over again. Engines keep serving throughout: the
|
||
router talks to them directly.
|
||
- **`llmgw agent` is real**, and `llmgw server` runs the control plane whenever
|
||
any model declares a profile. The static endpoint list remains for
|
||
router-only runs against sim-engines.
|
||
|
||
### Verified with real binaries
|
||
|
||
Server and agent as separate processes over a real stream: the agent registers,
|
||
a replica is placed, an engine is launched through the vLLM adapter's actual
|
||
argument construction, streaming traffic is served through the router, and
|
||
killing the agent moves the node through unreachable to lost with placement
|
||
explaining exactly why it can no longer run anything.
|
||
|
||
### Surviving a control plane restart
|
||
|
||
Not persisting replica state left one hole that had to be closed before it
|
||
mattered: restarting the control plane orphaned live engines. They kept holding
|
||
GPUs the inventory believed were free, so the next placement failed to allocate
|
||
with an error naming the new process rather than the old one.
|
||
|
||
The fix is on the agent, not in a database, and it cannot disagree with reality
|
||
the way a stale row can:
|
||
|
||
- The supervisor now writes the full shard spec, not just a pid and command
|
||
line. A pid identifies a process; the spec is what lets it be described back
|
||
to the control plane and matched to a replica.
|
||
- `Adopt` takes over a still-running engine from a previous agent. It cannot be
|
||
waited on — it is not this process's child — so it is watched by polling, and
|
||
stopped by signalling its group directly. Its earlier output is gone, and the
|
||
log says so rather than presenting emptiness as silence.
|
||
- Every `Register`, not only the first, reports what the agent is running. A
|
||
reconnect may be a control plane restart and has to recover the same way.
|
||
- The reconciler rebuilds replicas from those reports and reclaims their GPUs.
|
||
Anything whose model or profile left the catalog is stopped: nothing will
|
||
route to it and it is holding cards.
|
||
- An engine the agent cannot identify — a corrupt state file — is killed. A
|
||
process holding GPUs that nothing can describe is worse than no process,
|
||
because placement will hand those GPUs out again.
|
||
|
||
One mechanism covers agent restarts and control plane restarts.
|
||
|
||
### A test that was passing for no reason
|
||
|
||
`TestAgentReconnects` used `httptest.CloseClientConnections`, which does nothing
|
||
to an h2c stream: prior-knowledge h2c hijacks the connection, and httptest stops
|
||
tracking hijacked connections. The agent never reconnected and the assertion —
|
||
on the connection map, which still held a stale entry — passed anyway. Tests now
|
||
sever connections through their own listener and count opened streams.
|
||
|
||
### Multi-node launch
|
||
|
||
Placement and environment generation had landed early, which made multi-node
|
||
look further along than it was: a multi-node replica could not come up at all.
|
||
Three things were missing, and all three came down to two assumptions that do
|
||
not hold once a replica spans nodes.
|
||
|
||
**Not every rank serves HTTP.** The agent health-checked every shard at its own
|
||
port, but only the leader answers. Workers timed out, reported Failed, and took
|
||
the replica with them. `Engine.ServesHTTP(spec)` now says which ranks serve; a
|
||
worker is judged by staying alive instead. That is weaker evidence on purpose —
|
||
the replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined.
|
||
|
||
**A shard is not always one process.** `Engine.Commands(spec)` returns a
|
||
sequence: every command but the last is a sidecar kept alive for the shard's
|
||
lifetime and torn down with it, and the last one's exit is the shard's exit.
|
||
vLLM under Ray is why — the leader runs a Ray head and then the server against
|
||
it, and a worker runs only the join, with no vLLM process at all. `RayCommand`
|
||
had been written and tested but never called, so `launch_mode: ray` would have
|
||
run `vllm serve` against a cluster that did not exist.
|
||
|
||
A command can also require the rendezvous point to be listening before the next
|
||
one starts, bounded by a timeout so a head that never appears fails the shard
|
||
rather than hanging it. Sidecars are killed with the shard: a Ray head left
|
||
behind holds the port the next replica needs.
|
||
|
||
**Start order was backwards.** Workers were started first, on the theory that
|
||
the head should wait for them. Both engines put the rendezvous point on rank 0,
|
||
so a worker started first has nothing to connect to and spends its startup
|
||
budget retrying. vLLM's server blocks until the Ray cluster has the GPUs it
|
||
needs, so the leader can start immediately and waits on its own.
|
||
|
||
None of this is verified against real engines. It is tested with stub processes;
|
||
the Ray sequencing and which launch mode a given vLLM version wants can only be
|
||
confirmed on hardware.
|
||
|
||
### Still missing for phase 2
|
||
|
||
- mTLS. The stream is plaintext HTTP/2 today, so it must stay on a trusted
|
||
network.
|
||
|
||
### Phase 3: placement
|
||
|
||
- **Cluster inventory** (`internal/control/inventory`): nodes, their hardware,
|
||
their fabric attachments, and what is allocated where. A re-registering agent
|
||
cannot clear an operator's cordon, and a heartbeat revives an unreachable node
|
||
without doing so either. A GPU with an ECC fault or another process on it is
|
||
not counted as free. Replicas are released whole, because they fail whole.
|
||
- **Placement** (`internal/control/placement`): picks the highest-priority
|
||
profile that fits and explains every candidate it rejected. Two constraints
|
||
are enforced structurally rather than checked afterwards — there is no code
|
||
path that can combine nodes from two fabrics, and only GPUs of the profile's
|
||
class are ever considered.
|
||
- **Bin packing** fills partly used nodes first, so whole nodes stay free for
|
||
large models that cannot be split. Multi-node replicas get the opposite
|
||
treatment: they are the least movable thing in the cluster, so they take the
|
||
emptiest machines.
|
||
- **NVLink-aware GPU selection** keeps a tensor-parallel group inside one NVLink
|
||
domain where the topology is known. Without the NVML build it falls back to
|
||
the lowest free indices, which is usually right but only by accident of how
|
||
vendors number slots.
|
||
- **Property-based invariant tests** (plan §13): 400 randomly generated
|
||
clusters, checking on every placement that a multi-node replica lives on one
|
||
fabric, that each shard receives only that fabric's adapters, and that no
|
||
placement mixes GPU classes or uses an unhealthy or already-allocated GPU.
|
||
The table-driven tests cover the cases someone thought of; this covers the
|
||
ones nobody did.
|
||
- **Catalog** (`internal/control/catalog`): models, profiles, aliases,
|
||
fallbacks and scaling bounds. Aliases and fallbacks are resolved at load
|
||
rather than at request time — a typo in a fallback should stop a deployment,
|
||
not surface during the outage when the fallback is finally needed. The
|
||
node/GPU/TP arithmetic is checked here too, so a bad profile fails at config
|
||
load instead of minutes into model loading.
|
||
- **Reconciler** (`internal/control/reconciler`): the desired-versus-actual
|
||
loop. It places one replica per pass, because placement depends on what is
|
||
already allocated and starting several from one snapshot would hand them the
|
||
same GPUs. Workers start before rank 0, since SGLang's ranks rendezvous and
|
||
vLLM under Ray needs its workers attached before the head serves.
|
||
- **Replica failure is whole-replica failure.** One dead shard makes the others
|
||
useless — they wait on collectives that will never complete while holding
|
||
GPUs — so the reconciler stops every shard and rebuilds, with a jittered
|
||
exponential backoff carried on the model rather than the replica it just
|
||
destroyed.
|
||
- **Node expiry in two stages.** Unreachable first, Lost later. A brief network
|
||
blip must not tear down replicas that are still serving, and rebuilding a
|
||
multi-node replica costs tens of minutes of model loading.
|
||
- **Scale-down drains before stopping**, so in-flight requests finish instead of
|
||
being cut off mid-stream.
|
||
|
||
## v0.3.0 — 2026-09-16
|
||
|
||
Flow control. A batch workload no longer starves interactive traffic.
|
||
|
||
### Added
|
||
|
||
- **Admission control** (`internal/router/admission`), inserted between quota
|
||
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||
still answers *where* it goes. Keeping them apart is the point: an engine
|
||
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||
no amount of endpoint selection rescues the request behind it.
|
||
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||
nearly full cache, or the reverse.
|
||
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||
"flow up to the healthy limit"; above 1 means never wait.
|
||
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||
one caller sending a thousand requests occupies one slot in the rotation
|
||
rather than a thousand.
|
||
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||
(504), because sending the GPU work the client has already abandoned helps
|
||
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||
`x-llmgw-queue-ms` when a request waited.
|
||
|
||
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||
|
||
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||
With admission switched off the same load pushes it to **55–60x**. Both
|
||
directions are asserted, so the test fails if admission regresses *and* if the
|
||
load generator ever stops being a real load.
|
||
|
||
### Two lags, two different fixes
|
||
|
||
Getting this right took longer than the queueing itself:
|
||
|
||
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||
count does not — but it includes *running* requests, and treating those as
|
||
queue depth makes a healthy engine look overloaded every interval. What is
|
||
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||
the metric contract, so it is learned as the high-water mark of observed
|
||
`running`.
|
||
- Between admission granting a request and the router dispatching it, the
|
||
request is invisible to every signal. A simultaneous burst therefore passed
|
||
the check en masse — exactly the batch arrival pattern this layer exists to
|
||
survive. The controller now counts its own outstanding leases against
|
||
dispatched inflight.
|
||
|
||
### Known limitations
|
||
|
||
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||
Anyone who can reach the port can promote themselves to platinum.
|
||
- The tenant key is the session id, for the same reason.
|
||
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||
|
||
## v0.2.0 — 2026-09-16
|
||
|
||
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||
engine replicas with an OpenAI-compatible API.
|
||
|
||
### Added
|
||
|
||
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||
propagates cancellation: when a client disconnects, the upstream request is
|
||
aborted so the GPU stops working. Separate deadlines for dial, response
|
||
headers, first token, inter-token idle and total duration — a wedged engine is
|
||
indistinguishable from a slow one without the idle watchdog.
|
||
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||
reached the client; once bytes are out, splicing two responses together would
|
||
be worse than the short stream. Models can name fallback models, and the
|
||
response says which one served it via `x-llmgw-served-model`.
|
||
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||
Only routing-relevant fields are parsed; the rest of the body passes through
|
||
untouched.
|
||
- **Usage accounting that does not depend on the client.** The router can ask
|
||
engines for usage the client never requested, then drop that chunk on the way
|
||
back, so token counts are recorded either way.
|
||
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||
duplicate endpoint ids and aliases that would shadow another model — both
|
||
would otherwise misroute silently at request time.
|
||
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||
without seeing what the scorers actually produced. Served on a separate admin
|
||
listener alongside health and pprof.
|
||
|
||
### Phase 1 exit criterion, measured in `test/e2e`
|
||
|
||
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||
- Every client disconnecting frees the KV blocks on every engine.
|
||
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||
|
||
### Known limitations
|
||
|
||
- No admission control, so a batch workload can still starve interactive
|
||
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||
|
||
## v0.1.0 — 2026-09-16
|
||
|
||
First tagged version. It is useful for exactly one thing: developing and testing
|
||
a router without GPUs. **The gateway does not serve traffic yet** — `llmgw server`
|
||
returns "not implemented".
|
||
|
||
### Added
|
||
|
||
**sim-engine** (`llmgw sim-engine`) — an OpenAI-compatible fake inference engine
|
||
that models the behaviours a router has to cope with:
|
||
|
||
- continuous batching with a bounded concurrent sequence count
|
||
- KV cache block accounting, queueing when blocks run out, including
|
||
head-of-line blocking
|
||
- automatic prefix caching (chained block hashes over a block-level LRU) and its
|
||
effect on time to first token
|
||
- per-token latency that grows with batch size
|
||
- dropping a request and freeing its blocks when the client disconnects
|
||
- `/metrics` under vLLM's real metric names
|
||
- failure injection (`--fail-rate`) and deterministic runs (`--seed`)
|
||
|
||
**Routing primitives** — libraries, not yet wired into a server:
|
||
|
||
- `router/scheduler`: the filter → score → pick pipeline. Scorers for queue
|
||
depth, KV occupancy, router-side inflight, drain rate, capacity weight and
|
||
session affinity (bounded-load consistent hashing). Pickers: max-score, p2c,
|
||
softmax, round-robin. Built-in profiles: chat, batch, long-context, debug.
|
||
- `router/endpoint`: live endpoint registry with inflight counting, staleness
|
||
detection and drain rate derived from engine counters.
|
||
- `router/scrape`: parses the engine metric contract (plan §7.6) and refuses
|
||
endpoints missing any of the mandatory three. The metric name mapping is
|
||
overridable, because names drift between engine versions.
|
||
- `router/breaker`: cooldown and circuit breaking in one component, with
|
||
exponential backoff and half-open probes.
|
||
|
||
### Known limitations
|
||
|
||
- `llmgw server`, `llmgw agent` and `llmgw ctl` are stubs.
|
||
- The scheduler has no HTTP layer yet; nothing routes real requests.
|
||
- No admission control, so a batch workload can still starve interactive traffic
|
||
once the router exists. That is what v0.3.0 is for.
|
||
- Prefix awareness is tier 0 only (session affinity). Tiers 1 and 2 are later.
|
||
|
||
### Notes
|
||
|
||
- The project language is English throughout: code, comments, tests, commits and
|
||
the design document.
|
||
- Plan §7.6 corrects an error from plan v0.1: the KV usage metric is
|
||
`vllm:kv_cache_usage_perc`, not `vllm:gpu_cache_usage_perc`.
|