The desired-versus-actual loop, with a Dispatcher interface where the agent
stream will go. The loop's logic is where the interesting failures live, not the
transport, so it is tested against a fake dispatcher and a synthetic clock.
- catalog: models, profiles, aliases, fallbacks, scaling bounds. Aliases and
fallbacks resolve at load rather than at request time, because a typo in a
fallback should stop a deployment instead of surfacing during the outage when
the fallback is finally needed. The node/GPU/TP arithmetic is checked here
too, so a bad profile fails at config load rather than minutes into loading.
- reconciler: one replica placed per pass, since placement depends on what is
already allocated and several from one snapshot would be handed the same GPUs.
Workers start before rank 0: SGLang's ranks rendezvous and vLLM under Ray
needs its workers attached before the head serves.
- a failed shard fails the whole replica. The others are waiting on collectives
that will never complete while holding GPUs, so every shard is stopped and the
replica rebuilt, with a jittered exponential backoff carried on the model
rather than on the replica that was just destroyed.
- node expiry has two stages, Unreachable then Lost, deliberately far apart: a
network blip must not tear down replicas that are still serving, and
rebuilding a multi-node replica costs tens of minutes of model loading.
- scale-down drains before stopping, so in-flight requests finish.
The phase 3 exit criterion is half met. TestLostNodeMovesReplicas covers the
reconciliation half end to end. The router continuing to serve throughout needs
the EndpointSource wiring and the agent stream.
Writing the tests caught a real property of the design worth recording: advancing
the clock makes every node look silent, so placement correctly stops. The test
harness now heartbeats nodes as it advances time, except where a node going
silent is the point.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The two hard constraints from plan §1.1 are enforced structurally rather than
checked after the fact: there is no code path that can combine nodes from two
fabrics, and only GPUs of the profile's class are ever considered. Getting
either wrong produces a replica that starts, passes health checks, and then
hangs or crawls under load — which looks like success until traffic arrives.
- inventory: nodes, hardware, fabric attachments, allocations. A re-registering
agent cannot clear an operator's cordon, since a cordon is usually in place
because something is wrong with the node. A GPU with an ECC fault or another
process on it is not free in any useful sense. Replicas are released whole,
because they fail whole.
- placement: highest-priority profile that fits, with every rejected candidate
explained. "Unschedulable" on its own tells an operator nothing about which
of six constraints bit.
- bin packing fills partly used nodes first so whole nodes stay free for large
models that cannot be split; multi-node replicas get the emptiest machines
instead, being the least movable thing in the cluster.
- NVLink-aware GPU selection keeps a tensor-parallel group inside one domain
where the topology is known, falling back to lowest free indices otherwise.
Property-based tests generate 400 random clusters and assert the invariants on
every placement produced: one fabric per multi-node replica, only that fabric's
HCAs per shard, one GPU class, no unhealthy or already-allocated GPU. The
table-driven tests cover the cases someone thought of; these cover the rest.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5