Fill in the memory a model needs, instead of leaving the field blank
CI / test (push) Successful in 1m41s
CI / test (push) Successful in 1m41s
Serving a model from the mirror page left "Needs RAM (GB)" optional and empty. Empty writes no constraint, and a CPU profile has no card to be short of, so every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never told what the model needed. The mirror knows the size of the weights, so the field arrives filled in: weights plus half again for the KV pool, plus two for the engine. The half is there because the cache grows with the same things the weights do, layers and heads. It is an estimate and the field is still editable — what it is not is blank. A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every model to the one large machine would be a different way of getting placement wrong. An unknown size fills in nothing rather than the floor of two, which would exclude no node while reading as though somebody had measured it. And when the estimate is larger than any node in the fleet, the page says so, with the largest node's memory. Better read there than discovered by a node being OOM-killed twenty minutes into a load. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
2acd794022
commit
e0ea4f6be9
3 files changed
+133
-8
No files matched your search
@@ -110,8 +110,8 @@ reachable from this machine.</p>
|
||||
</div>
|
||||
<div class="field-row">
|
||||
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">optional</span>
|
||||
<input type="text" name="min_memory_gb" placeholder="8"></label>
|
||||
<label>Needs RAM (GB) <span class="opt">estimated from the weights</span>
|
||||
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
|
||||
<label>Only nodes labelled <span class="opt">key / value, optional</span>
|
||||
<input type="text" name="selector_key" placeholder="role" size="8">
|
||||
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
|
||||
@@ -120,9 +120,13 @@ reachable from this machine.</p>
|
||||
</form>
|
||||
<p class="note">The context pool is what the engine is given; llama.cpp divides
|
||||
it across its parallel slots, so the number one request may use is the smaller
|
||||
one above. A node label keeps a large model off the small machines — a CPU
|
||||
profile has no card to be short of, so without one every ready node looks like
|
||||
a candidate and the first one picked is the one that runs out of memory.</p>
|
||||
one above.</p>
|
||||
<p class="note">The memory figure is the weights plus half again for the KV pool
|
||||
plus two for the engine — an estimate, and editable. It is filled in rather than
|
||||
left blank because a CPU profile has no card to be short of: with no figure,
|
||||
every ready node is a candidate and the first one picked is whichever the
|
||||
scheduler reached, not whichever can hold the model. A node label does the same
|
||||
job by name if you would rather pin it.</p>
|
||||
{{end}}
|
||||
</td></tr>
|
||||
{{end}}
|
||||
|
||||
Reference in new issue
Block a user