Commit Graph
52 Commits
Author SHA1 Message Date
ilkerandClaude Opus 5 c1b9ea1bc2 Phase 3 (3/3): wire the control plane to the router
CI / test (push) Successful in 50s
Ready replicas become router endpoints with no static configuration involved.
Endpoints that survive an update keep their live state — inflight counts,
breaker state, drain rate — so a replica that was already serving does not start
from cold because another one moved. The sink is an interface, so when the
router and control plane split into separate processes it becomes a watch stream
and nothing above it changes.

Phase 3's exit criterion now holds end to end. A node hosting a live replica is
killed while four clients stream continuously; the control plane marks it
unreachable, then lost, rebuilds the replica on a survivor and republishes.
Across three runs ~370 requests completed with zero failures and no replica was
left on the dead node. This proves the control logic, not the transport: the
agent fleet is simulated and the real path still needs the stream.

The race detector caught a genuine bug while this went in. startReplica read the
replica's fields after publishing it to the map, and scaleUp walked replica
pointers outside the lock; both raced with agent status reports, which arrive
from another goroutine and mutate exactly those fields. Against the simulated
fleet the reports came back fast enough to hit the window on every run. State is
snapshotted under the lock now and those functions work from values.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:16:35 +02:00
ilkerandClaude Opus 5 f0134a2f95 Phase 3 (2/3): catalog and reconciler
CI / test (push) Successful in 24s
The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.

- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
  fallbacks resolve at load rather than at request time, because a typo in a
  fallback should stop a deployment instead of surfacing during the outage when
  the fallback is finally needed. The node/GPU/TP arithmetic is checked here
  too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
  already allocated and several from one snapshot would be handed the same GPUs.
  Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
  needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
  that will never complete while holding GPUs, so every shard is stopped and the
  replica rebuilt, with a jittered exponential backoff carried on the model
  rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
  network blip must not tear down replicas that are still serving, and
  rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.

The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.

Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 02:12:06 +02:00