Park a model that fails every time it starts, and say so everywhere
On the live cluster kev-0.5b — an architecture the nodes' llama.cpp did not know — was started 155 times in eight hours, failing the same way each time, and phi-4, which fitted on no node, logged "no placement" 250 times in two days. Neither showed up anywhere an operator looked: the backoff never gave up, no metric said a model was failing or unplaceable, and the one alert for a model with nothing serving could not fire, because its series was published only while a replica was ready. After five failures in a row a model is parked: not placed again until its definition changes — a new source or profile — or `llmgw ctl retry` is run. Status, the model list and the admin API show it with its last error. The placement warning is logged when its reason changes or every ten minutes. A request for a parked or unplaceable model is answered at once with why, rather than "loading; retry shortly", which a client retries forever. Each request still clears the previous placement result and triggers a fresh attempt, so a model that fits once room is made is not refused on a stale answer; the router waits for that attempt, not for its full activation window. Metrics now cover model health (desired, ready at 0 for what should run, consecutive start failures, parked, unplaceable), engine installs per node, leadership, and certificate expiry read from disk at each scrape, with alerts for each. `ctl status` counts a model against what it should have now, so an on-demand model that was asked for and fitted nowhere no longer reads as satisfied. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
5e1b6ecea0
commit
37a80a9b44
17 files changed
+740
-24
No files matched your search
@@ -42,10 +42,86 @@ groups:
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} has no replica ready"
|
||||
description: >-
|
||||
The model is placed and nothing is serving it. Ten minutes, because
|
||||
a large model on CPU legitimately takes minutes to load. A model
|
||||
that is deliberately stopped publishes no series at all, so this
|
||||
cannot fire for one nobody asked for.
|
||||
The model should be running and nothing is serving it. Ten minutes,
|
||||
because a large model on CPU legitimately takes minutes to load. The
|
||||
series is published at 0 for every model that should be running —
|
||||
before v0.10.4 it was published only while a replica was ready, so
|
||||
this could not fire at all. A model deliberately stopped, or an
|
||||
on-demand one nobody is asking for, publishes nothing.
|
||||
|
||||
- alert: ModelParked
|
||||
expr: llmgw_model_parked == 1
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} failed every time it started and is no longer retried"
|
||||
description: >-
|
||||
Five replicas in a row failed at start. Before this existed a model
|
||||
whose weights its engine could not load was started 155 times over
|
||||
eight hours. `llmgw ctl status` has the last error; once it is dealt
|
||||
with, `llmgw ctl retry <model>`. Changing the model's definition
|
||||
also retries it.
|
||||
|
||||
- alert: ModelFailingToStart
|
||||
expr: llmgw_model_start_failures >= 2
|
||||
for: 5m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} has failed to start {{ $value }} times in a row"
|
||||
description: >-
|
||||
On its way to being parked. The engine's own words are in
|
||||
`llmgw ctl logs <node> --source <shard>` for the next attempt, or in
|
||||
the control plane's log for the last one.
|
||||
|
||||
- alert: ModelCannotBePlaced
|
||||
expr: llmgw_model_unplaceable == 1
|
||||
for: 10m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} is wanted and fits on no node"
|
||||
description: >-
|
||||
Short of replicas, and the last attempt found nowhere with the
|
||||
memory, cards or engine it needs. `llmgw ctl explain-placement
|
||||
<model>` says what each node lacks. For an on-demand model, requests
|
||||
for it are told this rather than asked to retry.
|
||||
|
||||
- alert: EngineInstallFailed
|
||||
expr: llmgw_node_engine_install{state="failed"} == 1
|
||||
for: 15m
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.engine }} {{ $labels.version }} will not install on {{ $labels.node }}"
|
||||
description: >-
|
||||
The node takes no model needing it until it does. The agent retries
|
||||
on its own; `llmgw ctl nodes` and the node's logs on the cluster
|
||||
page say why it failed.
|
||||
|
||||
- alert: ControlPlaneLeadership
|
||||
expr: sum(llmgw_control_plane_leader) != 1
|
||||
for: 2m
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $value }} control planes believe they are placing work"
|
||||
description: >-
|
||||
None means nothing is placed or replaced; two means both are, each
|
||||
into GPUs the other believes free. The lease is in the store: check
|
||||
it is reachable from every control plane.
|
||||
|
||||
- alert: CertificateExpiresSoon
|
||||
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 30 * 86400
|
||||
labels: { severity: warning }
|
||||
annotations:
|
||||
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
||||
description: >-
|
||||
Certificates last a year and nothing renews them. When the agent
|
||||
server's runs out every agent refuses to connect at once. Reissue
|
||||
with `llmgw ctl cert` and restart the control plane.
|
||||
|
||||
- alert: CertificateExpiresVerySoon
|
||||
expr: (llmgw_certificate_expiry_timestamp_seconds - time()) < 7 * 86400
|
||||
labels: { severity: critical }
|
||||
annotations:
|
||||
summary: "{{ $labels.cert }} ({{ $labels.subject }}) expires in {{ $value | humanizeDuration }}"
|
||||
description: See CertificateExpiresSoon; a week is not long.
|
||||
|
||||
- alert: PoolSaturated
|
||||
expr: llmgw_pool_saturation > 1
|
||||
|
||||
Reference in new issue
Block a user