How long an on-demand model holds a card is a live setting

On one card shared by flux and sf3d, a mesh asked for after a picture
waited out the picture model's five minutes of protection and two of idle.
on_demand.evict_after_idle and on_demand.min_lifetime set those for the
cluster, in force on the leader as they change; unset keeps 2m and 5m.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -26,6 +26,14 @@ APIs change between minors without a deprecation period.
 
 ## Unreleased
 
+### Added
+
+- **How long on-demand models hold a card** is a setting, in force on the
+  leader within seconds: `on_demand.evict_after_idle` (2m unless set) and
+  `on_demand.min_lifetime` (5m). On one card shared by a picture model and
+  a 3D model, 1m each brings the wait between a picture and its mesh to
+  about a minute.
+
 ### Fixed
 
 - The cluster and node pages swapped their live part every few seconds,

docs/reference.md

@@ -205,6 +205,8 @@ rather than repeated.
 | `no_passwords` | `bool` | [OIDC](#oidc) |
 | `mirror` | `string` | [Offline](#offline) |
 | `checksums` | `string` | [Offline](#offline) |
+| `evict_after_idle` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
+| `min_lifetime` | `time.Duration` | [OnDemandSettings](#ondemandsettings) |
 | `nvidia_smi` | `string` | [Options](#options) |
 | `ib_root` | `string` | [Options](#options) |
 | `prefer_nvml` | `bool` | [Options](#options) |
@@ -346,6 +348,7 @@ rather than repeated.
 | `serving` | `ServingSettings` | [Server](#server) |
 | `firewall` | `FirewallSettings` | [Server](#server) |
 | `presets` | `PresetSettings` | [Server](#server) |
+| `on_demand` | `OnDemandSettings` | [Server](#server) |
 | `agents` | `AgentSettings` | [Server](#server) |
 | `firewall_networks` | `[]string` | [Server](#server) |
 | `backup_recipients` | `[]string` | [Server](#server) |
@@ -1912,6 +1915,25 @@ kept by llmgw bundle import and llmgw bundle trust; every engine file
 a node fetches from the mirror is checked against it. Read again
 whenever it changes. Empty is /etc/llmgw/offline.sha256.
 
+## OnDemandSettings
+
+### `evict_after_idle`
+
+`time.Duration`
+
+EvictAfterIdle is how long an on-demand replica must go unasked-for
+before a model that wants its card may have it. Zero keeps 2m.
+
+### `min_lifetime`
+
+`time.Duration`
+
+MinLifetime is how long a replica that has just come up is safe from
+that, so two models do not take turns paying for loads. Zero keeps
+5m. On one card shared by a picture model and a 3D model, 1m each
+brings the wait between a picture and its mesh to about a minute.
+Both are in force on the leader within seconds.
+
 ## Options
 
 ### `nvidia_smi`
@@ -2984,6 +3006,15 @@ Presets is how the models that come with llmgw reach the catalog.
 
 Takes a [PresetSettings](#presetsettings) block.
 
+### `on_demand`
+
+`OnDemandSettings`
+
+OnDemand is how models that load when asked for take turns on the
+cards.
+
+Takes a [OnDemandSettings](#ondemandsettings) block.
+
 ### `agents`
 
 `AgentSettings`

internal/cli/clustersettings.go

@@ -245,6 +245,8 @@ var liveSettings = []string{
 	"agents",
 	// Read by the leader's presets pass each time it runs.
 	"presets",
+	// Handed to the reconciler as they change.
+	"on_demand",
 	// Read by server init, node add and backups, not by the server.
 	"backup_recipients", "backup_copy_to",
 }

internal/cli/server.go

@@ -264,6 +264,7 @@ func newServerCmd() *cobra.Command {
 				agentSvc = svc
 				// Stopped by the operators (llmgw stop): nothing placed.
 				ctl.Reconciler().SetStopped(cfg.Serving.Stopped)
+				ctl.Reconciler().SetEvictionTimes(cfg.OnDemand.EvictAfterIdle, cfg.OnDemand.MinLifetime)
 				// Whether nodes keep llmgw's firewall, as the settings say now.
 				svc.SetFirewall(func() *llmgwv1.Firewall {
 					c := current.Load()
@@ -778,6 +779,9 @@ func newServerCmd() *cobra.Command {
 							fwPub.Kick()
 						}
 					}
+					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "on_demand") }) && admin != nil {
+						admin.Reconciler().SetEvictionTimes(next.OnDemand.EvictAfterIdle, next.OnDemand.MinLifetime)
+					}
 					if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
 						// A moment apart per node, so off this goroutine.
 						go agentSvc.BroadcastAgentSettings()

internal/config/config.go

@@ -161,6 +161,10 @@ type Server struct {
 	// Presets is how the models that come with llmgw reach the catalog.
 	Presets PresetSettings `koanf:"presets"`
 
+	// OnDemand is how models that load when asked for take turns on the
+	// cards.
+	OnDemand OnDemandSettings `koanf:"on_demand"`
+
 	// Agents are the nodes' agent settings: agent.yaml's keys but those an
 	// agent needs to start, for every node (default) and for one
 	// (nodes.<id>). Sent to the agents as they change; an agent whose

internal/config/ondemand.go

@@ -0,0 +1,18 @@
+package config
+
+import "time"
+
+// OnDemandSettings is how models that load when asked for take turns on the
+// cards: on a site with fewer cards than such models, the wait between one
+// and the next.
+type OnDemandSettings struct {
+	// EvictAfterIdle is how long an on-demand replica must go unasked-for
+	// before a model that wants its card may have it. Zero keeps 2m.
+	EvictAfterIdle time.Duration `koanf:"evict_after_idle"`
+	// MinLifetime is how long a replica that has just come up is safe from
+	// that, so two models do not take turns paying for loads. Zero keeps
+	// 5m. On one card shared by a picture model and a 3D model, 1m each
+	// brings the wait between a picture and its mesh to about a minute.
+	// Both are in force on the leader within seconds.
+	MinLifetime time.Duration `koanf:"min_lifetime"`
+}

internal/control/reconciler/evict.go

@@ -120,7 +120,7 @@ func (r *Reconciler) evictable(wantedBy string) []evictionCandidate {
 			continue
 		}
 		idle := now.Sub(rep.LastUsedAt)
-		if idle < r.cfg.EvictionGrace {
+		if idle < r.evictionGrace() {
 			continue
 		}
 		// A replica that has only just started is left alone however idle it
@@ -128,7 +128,7 @@ func (r *Reconciler) evictable(wantedBy string) []evictionCandidate {
 		// otherwise take turns evicting each other, paying a model load each
 		// time — and a load is expensive enough that waiting is strictly
 		// better than winning.
-		if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
+		if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.minLifetime() {
 			continue
 		}
 		nodes := map[string]struct{}{}
@@ -156,3 +156,34 @@ func (r *Reconciler) EvictionReason(model string) string {
 	}
 	return "nothing idle enough to stop"
 }
+
+// evictionGrace is how long an on-demand replica must go unasked-for before
+// another model may have its card: the cluster's setting, else the
+// configuration's.
+func (r *Reconciler) evictionGrace() time.Duration {
+	if d := time.Duration(r.evictGrace.Load()); d > 0 {
+		return d
+	}
+	return r.cfg.EvictionGrace
+}
+
+// minLifetime is how long a replica is safe from eviction once it is up.
+func (r *Reconciler) minLifetime() time.Duration {
+	if d := time.Duration(r.minLife.Load()); d > 0 {
+		return d
+	}
+	return r.cfg.MinLifetime
+}
+
+// SetEvictionTimes puts the cluster's on_demand settings in force: how long
+// an idle on-demand replica keeps its card, and how long a new one is safe.
+// Zero keeps the configuration's.
+//
+// On one card shared by several models these are the wait between them: a
+// 3D job after a picture waited out five minutes of a picture model it no
+// longer needed.
+func (r *Reconciler) SetEvictionTimes(afterIdle, minLifetime time.Duration) {
+	r.evictGrace.Store(int64(afterIdle))
+	r.minLife.Store(int64(minLifetime))
+	r.Notify()
+}

internal/control/reconciler/health.go

@@ -205,9 +205,9 @@ func (r *Reconciler) WaitFor(model string) (heldBy string, eta time.Duration, ok
 			// Evicted sooner, when it is no higher a class than the model
 			// waiting; one put first is waited for to its idle timeout.
 			if catalog.ClassRank(other.EffectiveClass()) <= catalog.ClassRank(m.EffectiveClass()) {
-				evict := rep.LastUsedAt.Add(r.cfg.EvictionGrace)
-				if !rep.ReadyAt.IsZero() && rep.ReadyAt.Add(r.cfg.MinLifetime).After(evict) {
-					evict = rep.ReadyAt.Add(r.cfg.MinLifetime)
+				evict := rep.LastUsedAt.Add(r.evictionGrace())
+				if !rep.ReadyAt.IsZero() && rep.ReadyAt.Add(r.minLifetime()).After(evict) {
+					evict = rep.ReadyAt.Add(r.minLifetime())
 				}
 				if evict.Before(h.release) {
 					h.release = evict

internal/control/reconciler/preempt.go

@@ -249,7 +249,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
 			aboveMin := i < above
 			tier := -1
 			switch {
-			case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.cfg.EvictionGrace:
+			case rank < myRank && other.Scaling.OnDemand() && idleFor(rep, now) >= r.evictionGrace():
 				tier = 0
 			case rank < myRank && aboveMin:
 				tier = 2
@@ -275,7 +275,7 @@ func (r *Reconciler) victims(m catalog.Model) []victim {
 			}
 			// Just started: stopping it throws away a model load that has
 			// only now been paid for.
-			if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.cfg.MinLifetime {
+			if !rep.ReadyAt.IsZero() && now.Sub(rep.ReadyAt) < r.minLifetime() {
 				continue
 			}
 			if len(nodesOf(rep)) > 1 {

internal/control/reconciler/reconciler.go

@@ -264,6 +264,9 @@ type Reconciler struct {
 	// because it is state about this moment and not about what the model is.
 	// A model nobody has wanted has no entry and is simply not running.
 	wanted map[string]time.Time
+	// evictGrace and minLife are the cluster's on_demand settings, as
+	// durations; zero keeps cfg's.
+	evictGrace, minLife atomic.Int64
 	// coldStart is how long each model's last replica took from placed to
 	// ready, for telling a request waiting on it about how long it waits.
 	coldStart map[string]time.Duration

internal/control/reconciler/reconciler_test.go

@@ -1763,6 +1763,31 @@ func mustResolve(t *testing.T, h *harness, name string) catalog.Model {
 	return m
 }
 
+func TestTheClustersOnDemandSettingsShortenTheWaitForACard(t *testing.T) {
+	// One card, a picture model and a 3D model: a mesh after a picture
+	// waited out five minutes of the picture model's protection.
+	h := newHarness(t, testModel(1), gpuNode("1", 4))
+	h.r.cat.Store(twoOnDemandModels(t))
+	h.r.Want("llama")
+	h.r.Reconcile(t.Context())
+	h.markReady()
+	reps := h.r.Replicas()
+	h.r.RecordUsage(map[string]time.Time{reps[0].ID: h.clock.now()})
+	h.advance(90 * time.Second)
+	if got := h.r.evictable("second"); len(got) != 0 {
+		t.Fatalf("offered at 90 s with the defaults: %+v", got)
+	}
+	h.r.SetEvictionTimes(30*time.Second, time.Minute)
+	if got := h.r.evictable("second"); len(got) != 1 {
+		t.Errorf("not offered at 90 s with 30 s idle and 1 min lifetime: %+v", got)
+	}
+	// Zero goes back to the configuration's.
+	h.r.SetEvictionTimes(0, 0)
+	if got := h.r.evictable("second"); len(got) != 0 {
+		t.Errorf("offered after the settings were cleared: %+v", got)
+	}
+}
+
 func TestAFreshlyStartedReplicaIsNotEvicted(t *testing.T) {
 	// What stops two models on a cluster with room for one from taking turns
 	// evicting each other, paying a model load each time. A load is expensive

internal/docgen/reference_gen.go

@@ -260,6 +260,10 @@ var Reference = []Block{
 		{Key: "mirror", Type: "string", Doc: "Mirror is the mirror's file address, as nodes reach it:\nhttp://10.0.0.5:8081 — the port model files are served on, not the\nmanagement API. Set, the cluster is offline: nodes install engines\nfrom it and never from their publishers, releases come from it unless\nreleases names another place, and no registry is asked whether a\nmodel has newer weights. Python engines (vLLM, SGLang) come from PyPI,\nwhich no bundle carries; on an offline node they are installed by hand.", Nested: ""},
 		{Key: "checksums", Type: "string", Doc: "Checksums is the file of engine checksums this control plane trusts,\nkept by llmgw bundle import and llmgw bundle trust; every engine file\na node fetches from the mirror is checked against it. Read again\nwhenever it changes. Empty is /etc/llmgw/offline.sha256.", Nested: ""},
 	}},
+	{Package: "config", Name: "OnDemandSettings", Title: "OnDemandSettings", Doc: "", Fields: []Field{
+		{Key: "evict_after_idle", Type: "time.Duration", Doc: "EvictAfterIdle is how long an on-demand replica must go unasked-for\nbefore a model that wants its card may have it. Zero keeps 2m.", Nested: ""},
+		{Key: "min_lifetime", Type: "time.Duration", Doc: "MinLifetime is how long a replica that has just come up is safe from\nthat, so two models do not take turns paying for loads. Zero keeps\n5m. On one card shared by a picture model and a 3D model, 1m each\nbrings the wait between a picture and its mesh to about a minute.\nBoth are in force on the leader within seconds.", Nested: ""},
+	}},
 	{Package: "discovery", Name: "Options", Title: "Options", Doc: "", Fields: []Field{
 		{Key: "nvidia_smi", Type: "string", Doc: "NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.\nWorth configuring on nodes where the driver lives outside the default\npaths, which containerized driver installs often do.", Nested: ""},
 		{Key: "ib_root", Type: "string", Doc: "IBRoot is the sysfs InfiniBand directory.", Nested: ""},
@@ -442,6 +446,7 @@ var Reference = []Block{
 		{Key: "serving", Type: "ServingSettings", Doc: "Serving stops and starts the whole cluster's service (llmgw stop, llmgw\nstart).", Nested: "config.ServingSettings"},
 		{Key: "firewall", Type: "FirewallSettings", Doc: "Firewall turns llmgw's firewall on every machine on or off.", Nested: "config.FirewallSettings"},
 		{Key: "presets", Type: "PresetSettings", Doc: "Presets is how the models that come with llmgw reach the catalog.", Nested: "config.PresetSettings"},
+		{Key: "on_demand", Type: "OnDemandSettings", Doc: "OnDemand is how models that load when asked for take turns on the\ncards.", Nested: "config.OnDemandSettings"},
 		{Key: "agents", Type: "AgentSettings", Doc: "Agents are the nodes' agent settings: agent.yaml's keys but those an\nagent needs to start, for every node (default) and for one\n(nodes.<id>). Sent to the agents as they change; an agent whose\nsettings changed starts again on them, its engines serving\nthroughout.", Nested: "config.AgentSettings"},
 		{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
 		{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},

internal/docgen/render.go

@@ -36,6 +36,7 @@ var Sources = []string{
 	"internal/config/firewall.go",
 	"internal/config/serving.go",
 	"internal/config/presets.go",
+	"internal/config/ondemand.go",
 	"internal/config/agents.go",
 	"internal/mail/mail.go",
 	"internal/notify/notify.go",