One bidirectional connect-go stream per node, carrying registration, heartbeats
and shard status up, and start, stop and drain down. The agent dials out: GPU
nodes sit behind firewalls far more often than a control plane does, and an
outbound connection needs no inbound rule on every node in the fleet.
- registration carries the whole hardware inventory, so there is no separate
discovery call and a reconnecting agent refreshes what the scheduler knows.
The fabric labels travel with it; they cannot be inferred from the wire, so
losing them in transit would silently disable multi-node placement.
- agentsvc is both the stream handler and the reconciler's dispatcher, because
it holds the open streams and is therefore the only thing that can deliver a
command to a node.
- the agent validates a profile before launching anything, since an engine that
refuses a profile does so after minutes of loading, then supervises the
process and reports every transition with the engine's output tail on failure.
- reconnects forever with jittered backoff. A control plane restart must not
require touching every GPU node, and a fleet must not reconnect in lockstep
and knock it over again. Engines keep serving throughout, because the router
talks to them directly.
llmgw agent is real now, and llmgw server runs a control plane whenever any
model declares a profile. The static endpoint list stays for router-only runs.
Verified with real binaries as separate processes: the agent registers, a
replica is placed, an engine launches through the vLLM adapter's actual argument
construction, streaming traffic is served, and killing the agent moves the node
through unreachable to lost with placement explaining why nothing can run.
config.Model and catalog.Model were the same thing described twice, so the
router now reads catalog entries directly and the validation rules live in one
place.
Still missing: mTLS. The stream is plaintext HTTP/2 and must stay on a trusted
network until that lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.
Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.
The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Inserted between quota checks and scheduling: admission answers when a request
may proceed, the scheduler still answers where it goes. Keeping them apart is
the point — an engine queue is FIFO and knows nothing about tenants, so once a
batch job is in it, no endpoint selection rescues the request behind it.
- saturation detector: one number per pool, the worse of queue depth and KV
occupancy, since either alone misleads.
- priority bands with per-band saturation ceilings, so the cluster degrades from
the bottom up.
- tenant round-robin within a band: one caller sending a thousand requests takes
one slot in the rotation, not a thousand.
- bounded queues with capacity (503) and TTL (504); cancelled requests leave the
queue instead of holding capacity.
- metrics for queue depth, wait time, rejections and pool saturation.
Exit criterion, measured in test/e2e: with 500 concurrent bronze requests
running continuously, interactive p99 in the platinum band stays at 1.04-1.24x
idle against a 1.5x budget. With admission off the same load pushes it to
55-60x. Both directions are asserted, so the test fails if admission regresses
and also if the load generator stops being a real load.
Two lags had to be closed, and they needed different answers. Engine metrics are
scraped and lag an interval; router inflight does not, but it counts running
requests, and treating those as queue makes a healthy engine look overloaded
every interval. What is actually queued is inflight minus batch capacity, and
capacity is not in the metric contract, so it is learned as the high-water mark
of observed running. Separately, between admission granting a request and the
router dispatching it the request is invisible everywhere, so a simultaneous
burst passed the check en masse; the controller now counts its own outstanding
leases against dispatched inflight.
One plan correction: a saturation ceiling of 1.0 was documented as "never held
back". It is not — it means "flow up to the healthy limit". A band that must
never wait needs a ceiling above 1. Plan §7.2.2 updated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.
- proxy: SSE relayed event by event with a flush after each, cancellation
propagated upstream, and separate deadlines for dial, headers, first token,
inter-token idle and total. The idle watchdog is what distinguishes a wedged
engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
nothing has reached the client; after that, splicing two responses together
would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
Validation rejects duplicate endpoint ids and aliases that shadow a model,
because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
per-filter drops on a separate admin listener with health and pprof.
Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.
The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The code behind plan §7.3-7.6:
- breaker: folds cooldown and circuit breaking into one component
(FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff
and half-open probes.
- endpoint: live state registry. Inflight counting, staleness detection, drain
rate derived from counters. A scrape failure does not trip the breaker — the
engine may be serving fine while only its metrics endpoint is down.
- scrape: parses the §7.6 metric contract. Errors when any of the mandatory
three is missing; the metric name mapping is overridable, since names drift
between versions (plan §14).
- scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a
set of scorer weights; chat, batch and long-context are the same code with
different weights. Scorers: queue, kv (hard penalty above the ceiling),
inflight, load, weight, session (bounded-load consistent hashing). Pickers:
max-score, p2c, softmax, round-robin.
Decision carries the rationale — which scorer produced what, which filter
dropped how many. Tuning the weights without that is guesswork.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>