Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s

On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 01:21:57 +02:00
1 parent 5e1b6ecea0
commit 37a80a9b44
17 files changed
+740 -24

No files matched your search

+18 -2
View File
@@ -25,6 +25,9 @@ type Summary struct {
// Unplaceable names models the scheduler currently cannot satisfy. This is
// the line that turns "something is wrong" into a place to look.
Unplaceable []string `json:"unplaceable,omitempty"`
// Parked are models that failed every time they started and are no
// longer retried, with the last error.
Parked []ParkedModel `json:"parked,omitempty"`
// IdleReplicas are serving replicas nothing has asked for in a while, which
// on a fleet with more models than GPUs is the first place to find room.
IdleReplicas []IdleReplica `json:"idle_replicas,omitempty"`
@@ -201,17 +204,30 @@ func (a *API) Summary() Summary {
for _, m := range a.Models() {
s.Models.Total++
if m.Ready >= m.Min {
if m.Parked {
s.Parked = append(s.Parked, ParkedModel{Model: m.Name, Failures: m.Failures, Reason: m.Reason})
}
// Against what it should have now, not its minimum: an on-demand
// model's minimum is 0, so one that was asked for and fitted nowhere
// used to count as satisfied.
if m.Ready >= m.Desired && !m.Parked {
s.Models.Satisfied++
continue
}
s.Models.Short++
// Short of replicas and unable to place more is the actionable case;
// short and still starting is just a cluster doing its job.
if m.Total <= m.Ready {
if m.Unplaceable || (m.Total <= m.Ready && !m.Parked) {
s.Unplaceable = append(s.Unplaceable, m.Name)
}
}
sort.Strings(s.Unplaceable)
return s
}
// ParkedModel is a model no longer retried, and why.
type ParkedModel struct {
Model string `json:"model"`
Failures int `json:"failures"`
Reason string `json:"reason"`
}