Busy is not unfit: rank files by the hardware the cluster has, and let a model wait for its cards
Ranking what was free first made every file come out on the CPU on a cluster whose one card was in use. The plan now ranks by hardware alone, free now only breaking a tie; a model served onto cards that are busy is added and waits for them, the job ending as waiting rather than failed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
c2be429554
commit
7cb9d4fa01
7 files changed
+118
-17
No files matched your search
+4
-1
@@ -195,7 +195,10 @@ to restart it. Another control plane keeps its own copy.
|
||||
one card, on several of a node's cards, across several nodes' cards over a
|
||||
fabric (original weights only), or on a CPU node — what each card or node
|
||||
would hold, and whether that hardware is free now. The file that suits the
|
||||
cluster best is first, marked ★; one that fits nowhere says so. Choosing a
|
||||
cluster best is first, marked ★; one that fits nowhere says so. Busy is not
|
||||
unfit: which file is best follows the hardware the cluster has, not what it
|
||||
is running, so a file for a card that is in use is still offered for it —
|
||||
served, it waits for the card, or with a fallback runs elsewhere until then. Choosing a
|
||||
file offers every way it fits, best first, and what to do when that one is
|
||||
busy: the next best way, written as a lower-priority profile so placement
|
||||
falls back to it, or wait. A vision model offers its projector too, fetched
|
||||
|
||||
Reference in new issue
Block a user