Add capacity classes and preempt lower ones to place a model
A model's class (critical, standard, opportunistic; unset follows its scaling) decides what may be stopped to make room for it: a lower class for a higher, and within a class only what another grew into under load, for a model below its minimum. Victims are tried on a copy of the inventory and stopped only if they make the room. Preempted replicas are drained with a short grace and their model is held back until the room is used. ctl model resources and class move under ctl model. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
990d82d801
commit
ac24e204a6
16 files changed
+720
-12
No files matched your search
@@ -84,6 +84,17 @@ groups:
|
||||
<model>` says what each node lacks. For an on-demand model, requests
|
||||
for it are told this rather than asked to retry.
|
||||
|
||||
- alert: ModelPreempted
|
||||
expr: increase(llmgw_model_preempted_total[15m]) > 0
|
||||
labels: { severity: info }
|
||||
annotations:
|
||||
summary: "{{ $labels.model }} ({{ $labels.class }}) gave up replicas to a higher class"
|
||||
description: >-
|
||||
Capacity ran short and a model of a higher class took this one's
|
||||
room. Expected now and then; if it keeps firing for the same model,
|
||||
the cluster is too small for what is asked of it at these classes.
|
||||
`llmgw ctl models` shows who it gave way to.
|
||||
|
||||
- alert: EngineInstallFailed
|
||||
expr: llmgw_node_engine_install{state="failed"} == 1
|
||||
for: 15m
|
||||
|
||||
Reference in new issue
Block a user