Commit Graph
2 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 e725adc2f0 Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:03:17 +02:00
Ilker ManapandClaude Opus 5 9b32531ce3 Stop an idle model to make room for one somebody asked for
On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.

Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.

Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.

Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:51:07 +02:00