Files
llmgw/internal/config/ondemand.go
T
Ilker ManapandClaude Opus 5.5 1e7086e13e How long an on-demand model holds a card is a live setting
On one card shared by flux and sf3d, a mesh asked for after a picture
waited out the picture model's five minutes of protection and two of idle.
on_demand.evict_after_idle and on_demand.min_lifetime set those for the
cluster, in force on the leader as they change; unset keeps 2m and 5m.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-14 08:58:38 +02:00

19 lines
816 B
Go

package config
import "time"
// OnDemandSettings is how models that load when asked for take turns on the
// cards: on a site with fewer cards than such models, the wait between one
// and the next.
type OnDemandSettings struct {
// EvictAfterIdle is how long an on-demand replica must go unasked-for
// before a model that wants its card may have it. Zero keeps 2m.
EvictAfterIdle time.Duration `koanf:"evict_after_idle"`
// MinLifetime is how long a replica that has just come up is safe from
// that, so two models do not take turns paying for loads. Zero keeps
// 5m. On one card shared by a picture model and a 3D model, 1m each
// brings the wait between a picture and its mesh to about a minute.
// Both are in force on the leader within seconds.
MinLifetime time.Duration `koanf:"min_lifetime"`
}