Fill in the memory a model needs, instead of leaving the field blank

Serving a model from the mirror page left "Needs RAM (GB)" optional and empty.
Empty writes no constraint, and a CPU profile has no card to be short of, so
every ready node became a candidate — and a 4.68 GB model went onto a 7.9 GB
machine and left it 3.5 GB for a KV pool. Placement was not wrong; it was never
told what the model needed.

The mirror knows the size of the weights, so the field arrives filled in:
weights plus half again for the KV pool, plus two for the engine. The half is
there because the cache grows with the same things the weights do, layers and
heads. It is an estimate and the field is still editable — what it is not is
blank.

A 1.12 GB model asks for 3 GB, so the small nodes stay usable. Pinning every
model to the one large machine would be a different way of getting placement
wrong.

An unknown size fills in nothing rather than the floor of two, which would
exclude no node while reading as though somebody had measured it.

And when the estimate is larger than any node in the fleet, the page says so,
with the largest node's memory. Better read there than discovered by a node
being OOM-killed twenty minutes into a load.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 16:38:49 +02:00
1 parent 2acd794022
commit e0ea4f6be9
3 files changed
+133 -8

No files matched your search

+59
View File
@@ -32,6 +32,14 @@ type serveForm struct {
// be written later. Named for the hardware rather than the model, because
// a profile is about where a model runs.
Profile string
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
//
// Filled in rather than left blank. Blank writes no constraint at all, and
// a CPU profile has no card to be short of, so every ready node becomes a
// candidate: a 4.7 GB model went onto a 7.9 GB machine and left it with
// 3.5 GB for a KV pool. Nothing was wrong with placement — it was never
// told what the model needed.
MinMemoryGB int
// Refusal is set when this version cannot be served as it stands, and says
// why. Offered and then failing at placement is worse than not offered.
@@ -90,6 +98,12 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
"same model runs on the CPU nodes"
}
}
f.MinMemoryGB = memoryFor(st.BytesTotal, st.BytesDone)
if biggest := biggestNode(fleet); biggest > 0 && f.MinMemoryGB > biggest {
f.Warning = strings.TrimSuffix(f.Warning, ".") +
fmt.Sprintf("%sthe largest node has %d GB and this needs about %d",
sep(f.Warning), biggest, f.MinMemoryGB)
}
if cat != nil && cat.Declared(st.Model) {
f.Already = true
f.Refusal = "a model called " + st.Model +
@@ -226,3 +240,48 @@ func positive(v, what string) (int, error) {
}
return n, nil
}
// memoryFor estimates the host RAM a model needs, from the size of its weights.
//
// Weights plus half again, plus two. The half covers the KV pool, which grows
// with the same things the weights do — layers and heads — and the two are the
// engine's own footprint and the page cache it will not get back. It is an
// estimate and the field it fills is editable; what it is not is blank, which
// is what put a 4.7 GB model on a 7.9 GB node.
//
// The honest number comes from running it: `llmgw ctl bench` measures what a
// replica can actually do, and that is a different question from what it needs
// to start.
func memoryFor(total, done int64) int {
size := total
if size <= 0 {
size = done
}
if size <= 0 {
return 0
}
gb := float64(size) / (1 << 30)
return int(gb*1.5) + 2
}
// biggestNode is the most memory any single node reports, or 0 when none says.
func biggestNode(fleet Source) int {
if fleet == nil {
return 0
}
most := 0
for _, n := range fleet.Nodes() {
if n.MemoryGB > most {
most = n.MemoryGB
}
}
return most
}
// sep joins a second clause onto a warning that may be empty.
func sep(existing string) string {
if existing == "" {
return ""
}
return ", and "
}