Park a model that fails every time it starts, and say so everywhere
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m34s

On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not
know — was started 155 times in eight hours, failing the same way each time,
and phi-4, which fitted on no node, logged "no placement" 250 times in two
days. Neither showed up anywhere an operator looked: the backoff never gave
up, no metric said a model was failing or unplaceable, and the one alert for
a model with nothing serving could not fire, because its series was published
only while a replica was ready.

After five failures in a row a model is parked: not placed again until its
definition changes — a new source or profile — or `llmgw ctl retry` is run.
Status, the model list and the admin API show it with its last error. The
placement warning is logged when its reason changes or every ten minutes.

A request for a parked or unplaceable model is answered at once with why,
rather than "loading; retry shortly", which a client retries forever. Each
request still clears the previous placement result and triggers a fresh
attempt, so a model that fits once room is made is not refused on a stale
answer; the router waits for that attempt, not for its full activation
window.

Metrics now cover model health (desired, ready at 0 for what should run,
consecutive start failures, parked, unplaceable), engine installs per node,
leadership, and certificate expiry read from disk at each scrape, with alerts
for each. `ctl status` counts a model against what it should have now, so an
on-demand model that was asked for and fitted nowhere no longer reads as
satisfied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 01:21:57 +02:00
1 parent 5e1b6ecea0
commit 37a80a9b44
17 files changed
+740 -24

No files matched your search

+80 -4
View File
@@ -42,10 +42,86 @@ groups:
annotations:
summary: "{{ $labels.model }} has no replica ready"
description: >-
The model is placed and nothing is serving it. Ten minutes, because
a large model on CPU legitimately takes minutes to load. A model
that is deliberately stopped publishes no series at all, so this
cannot fire for one nobody asked for.
The model should be running and nothing is serving it. Ten minutes,
because a large model on CPU legitimately takes minutes to load. The
series is published at 0 for every model that should be running —
before v0.10.4 it was published only while a replica was ready, so
this could not fire at all. A model deliberately stopped, or an
on-demand one nobody is asking for, publishes nothing.
- alert: ModelParked
expr: llmgw_model_parked == 1
labels: { severity: critical }
annotations:
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
description: >-
Five replicas in a row failed at start. Before this existed a model
whose weights its engine could not load was started 155 times over
eight hours. `llmgw ctl status` has the last error; once it is dealt
with, `llmgw ctl retry <model>`. Changing the model's definition
also retries it.
- alert: ModelFailingToStart
expr: llmgw_model_start_failures >= 2
for: 5m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
description: >-
On its way to being parked. The engine's own words are in
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
the control plane's log for the last one.
- alert: ModelCannotBePlaced
expr: llmgw_model_unplaceable == 1
for: 10m
labels: { severity: warning }
annotations:
summary: "{{ $labels.model }} is wanted and fits on no node"
description: >-
Short of replicas, and the last attempt found nowhere with the
memory, cards or engine it needs. `llmgw ctl explain-placement
<model>` says what each node lacks. For an on-demand model, requests
for it are told this rather than asked to retry.
- alert: EngineInstallFailed
expr: llmgw_node_engine_install{state="failed"} == 1
for: 15m
labels: { severity: warning }
annotations:
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
description: >-
The node takes no model needing it until it does. The agent retries
on its own; `llmgw ctl nodes` and the node's logs on the cluster
page say why it failed.
- alert: ControlPlaneLeadership
expr: sum(llmgw_control_plane_leader) != 1
for: 2m
labels: { severity: critical }
annotations:
summary: "{{ $value }} control planes believe they are placing work"
description: >-
None means nothing is placed or replaced; two means both are, each
into GPUs the other believes free. The lease is in the store: check
it is reachable from every control plane.
- alert: CertificateExpiresSoon
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
labels: { severity: warning }
annotations:
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
description: >-
Certificates last a year and nothing renews them. When the agent
server's runs out every agent refuses to connect at once. Reissue
with `llmgw ctl cert` and restart the control plane.
- alert: CertificateExpiresVerySoon
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
labels: { severity: critical }
annotations:
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
description: See CertificateExpiresSoon; a week is not long.
- alert: PoolSaturated
expr: llmgw_pool_saturation > 1