Commit Graph
5 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 1e7086e13e How long an on-demand model holds a card is a live setting
On one card shared by flux and sf3d, a mesh asked for after a picture
waited out the picture model's five minutes of protection and two of idle.
on_demand.evict_after_idle and on_demand.min_lifetime set those for the
cluster, in force on the leader as they change; unset keeps 2m and 5m.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 08:58:38 +02:00
Ilker ManapandClaude Opus 5.5 fa6e68c8e8 A model put first is not evicted, idle, for a lower class
The VLM is to come first on the one card. Its class makes it take the card
from flux, triposr or sf3d when asked for, but idle eviction ignored class:
a 3D job two minutes into a lull would have taken the card from it. Idle
eviction offers only replicas of no higher class than the model wanting
room, and the wait such a model is told runs to the holder's idle timeout.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-13 10:15:14 +02:00
Ilker ManapandClaude Opus 5.5 cf2edfd995 A model evicted for being idle is no longer wanted
flux evicted to make room for triposr was placed again two seconds later:
still on the wanted list, and first in the catalog, it took back the card
it had just given up, and the 3D job went on waiting. Its last replica
evicted, it leaves the list until it is asked for again.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 21:11:40 +02:00
Ilker ManapandClaude Opus 5.5 e725adc2f0 Tracing beyond the router: reconcile passes, placement and a replica's start traced down to the node
A reconcile pass that does something is a trace, with placement (where it
went, or every candidate turned down), scale-up/down, drain, stop, eviction,
preemption, reaping and restarts beneath it. A replica's start is a span open
from placement until it serves; the start command carries its context in a
new ControlMessage.trace field, and the agent continues the trace through
the weights fetch, engine launch, rendezvous and readiness wait. Stops,
drains, adoption and engine installs are traced on the node, model jobs are
a trace with a span per step linked to the replica's start, and agent.yaml
takes the same tracing block as server.yaml. Off, every span is the no-op one
and commands carry nothing.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:03:17 +02:00
Ilker ManapandClaude Opus 5 9b32531ce3 Stop an idle model to make room for one somebody asked for
On-demand loading worked right up to the moment the cluster was full and
then stopped meaning anything: the model somebody had just asked for could
not be placed, and the cards were held by a model nobody had asked for in an
hour. That is the only situation in which any of it matters.

Deliberately narrow. Only idle replicas of other on-demand models — a model
at min 1 is a promise that it stays up, and breaking it to serve a different
request would make every guarantee in the catalog conditional on load. Only
single-node replicas, since evicting a multi-node one frees several machines
and costs tens of minutes to rebuild. One at a time, because freeing
everything that qualifies would empty a cluster for a request that needed
one node. And never a replica with no reported usage: after a router restart
that is all of them, and unknown is not idle.

Writing the test found a gap the feature had rather than a bug in the test.
Two on-demand models on a cluster with room for one would take turns
evicting each other, at a model load per cycle — worse for both than one of
them waiting. A replica is now safe from eviction for a few minutes after it
starts serving, which bounds that.

Which in turn found a latent bug: deriveState stamped ReadyAt with
time.Now() inside a package that has an injectable clock, so the new
lifetime rule could not be tested at all — the fake clock said three minutes
and the wall clock said eight months. Invisible while nothing compared
ReadyAt to anything.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:51:07 +02:00