Ilker ManapandClaude Opus 5
3081c6f69b
Placement had no idea how much memory a node has or a model needs. A CPU profile has no card to be free, so eligibleNodes accepted every ready node: Qwen2.5-Coder 14B, whose working set is about 24 GB, would have been placed just as happily on one of the 8 GB nodes, which reports itself perfectly healthy right up until the OOM killer arrives. node_selector was yesterday's stopgap and it is the operator restating by hand something the system can know. Agents now report MemTotal, profiles declare min_memory_gb, and placement filters on it, naming what was short — "has 8 GB of memory, needs 24" is actionable where "no candidates were considered" is not. The label selector stays, because "this node is for coding models" is a real thing to want to say that memory does not express. Two deliberate non-exclusions. A profile with no requirement fits anywhere, which is every profile that exists today. And a node that reports no memory is still usable: an agent older than the field says nothing, and letting a control plane upgrade quietly empty half a fleet is a worse failure than the one this prevents. ctl nodes grows a MEM column showing "-" rather than 0 for a node that did not say, because an old agent is not a small machine. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
52 lines
1.4 KiB
Go
52 lines
1.4 KiB
Go
package discovery
|
|
|
|
import (
|
|
"bufio"
|
|
"os"
|
|
"strconv"
|
|
"strings"
|
|
)
|
|
|
|
// MemoryTotalMB reports the machine's total RAM.
|
|
//
|
|
// It exists because placement had no idea how much a node could hold. A CPU
|
|
// profile has no card to be free, so every ready node looked like a candidate:
|
|
// a 14B model whose working set is 24 GB would have been placed just as
|
|
// happily on a node with 8 GB, which reports itself perfectly healthy right up
|
|
// until the OOM killer arrives. Labels were the stopgap; this is the fact they
|
|
// were standing in for.
|
|
//
|
|
// Read from /proc/meminfo, which is Linux-only — every node is. On anything
|
|
// else it returns 0, which callers treat as "unknown" and therefore no
|
|
// constraint, exactly as an older agent that does not report it at all.
|
|
func MemoryTotalMB() int { return memTotalMBFrom("/proc/meminfo") }
|
|
|
|
func memTotalMBFrom(path string) int {
|
|
f, err := os.Open(path)
|
|
if err != nil {
|
|
return 0
|
|
}
|
|
defer f.Close()
|
|
|
|
sc := bufio.NewScanner(f)
|
|
for sc.Scan() {
|
|
rest, ok := strings.CutPrefix(sc.Text(), "MemTotal:")
|
|
if !ok {
|
|
continue
|
|
}
|
|
fields := strings.Fields(rest)
|
|
if len(fields) == 0 {
|
|
return 0
|
|
}
|
|
kb, err := strconv.Atoi(fields[0])
|
|
if err != nil {
|
|
return 0
|
|
}
|
|
// MemTotal is what the kernel can hand out, already short of the
|
|
// installed total by the firmware's reservations. That is the right
|
|
// number for deciding what fits.
|
|
return kb / 1024
|
|
}
|
|
return 0
|
|
}
|