Add capacity classes and preempt lower ones to place a model
Release / build (push) Successful in 1m7s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 3m46s

A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-30 18:02:59 +02:00
1 parent 990d82d801
commit ac24e204a6
16 files changed
+720 -12

No files matched your search

+11
View File
@@ -84,6 +84,17 @@ groups:
<model>` says what each node lacks. For an on-demand model, requests
for it are told this rather than asked to retry.
- alert: ModelPreempted
expr: increase(llmgw_model_preempted_total[15m]) > 0
labels: { severity: info }
annotations:
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
description: >-
Capacity ran short and a model of a higher class took this one's
room. Expected now and then; if it keeps firing for the same model,
the cluster is too small for what is asked of it at these classes.
`llmgw ctl models` shows who it gave way to.
- alert: EngineInstallFailed
expr: llmgw_node_engine_install{state="failed"} == 1
for: 15m