Fill in the memory a model needs, instead of leaving the field blank
CI / test (push) Successful in 1m41s

Serving a model from the mirror page left "Needs RAM (GB)" optional and empty.
Empty writes no constraint, and a CPU profile has no card to be short of, so
every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB
machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never
told what the model needed.

The mirror knows the size of the weights, so the field arrives filled in:
weights plus half again for the KV pool, plus two for the engine. The half is
there because the cache grows with the same things the weights do, layers and
heads. It is an estimate and the field is still editable — what it is not is
blank.

A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every
model to the one large machine would be a different way of getting placement
wrong.

An unknown size fills in nothing rather than the floor of two, which would
exclude no node while reading as though somebody had measured it.

And when the estimate is larger than any node in the fleet, the page says so,
with the largest node's memory. Better read there than discovered by a node
being OOM-killed twenty minutes into a load.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 16:38:49 +02:00
1 parent 2acd794022
commit e0ea4f6be9
3 files changed
+133 -8

No files matched your search

+9 -5
View File
@@ -110,8 +110,8 @@ reachable from this machine.</p>
</div>
<div class="field-row">
<label>Profile name <input type="text" name="profile" value="{{.Profile}}"></label>
<label>Needs RAM (GB) <span class="opt">optional</span>
<input type="text" name="min_memory_gb" placeholder="8"></label>
<label>Needs RAM (GB) <span class="opt">estimated from the weights</span>
<input type="text" name="min_memory_gb" value="{{.MinMemoryGB}}"></label>
<label>Only nodes labelled <span class="opt">key / value, optional</span>
<input type="text" name="selector_key" placeholder="role" size="8">
<input type="text" name="selector_value" placeholder="cpu-large" size="10"></label>
@@ -120,9 +120,13 @@ reachable from this machine.</p>
</form>
<p class="note">The context pool is what the engine is given; llama.cpp divides
it across its parallel slots, so the number one request may use is the smaller
one above. A node label keeps a large model off the small machines — a CPU
profile has no card to be short of, so without one every ready node looks like
a candidate and the first one picked is the one that runs out of memory.</p>
one above.</p>
<p class="note">The memory figure is the weights plus half again for the KV pool
plus two for the engine — an estimate, and editable. It is filled in rather than
left blank because a CPU profile has no card to be short of: with no figure,
every ready node is a candidate and the first one picked is whichever the
scheduler reached, not whichever can hold the model. A node label does the same
job by name if you would rather pin it.</p>
{{end}}
</td></tr>
{{end}}