Ilker ManapandClaude Opus 5.5
6403e61f4b
CI / test (push) Successful in 9m31s
Each agent measures one core's memory rate beside the whole machine's; the estimate takes a replica's threads at about one core's rate each, up to the machine's, calibrated by llama-bench on the PoC. The serve form shows it beside a card's, warns when slow, and asks for the CPU on purpose when a reply would take minutes; ctl model plan, the console and serve jobs say it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
128 lines
4.7 KiB
Go
128 lines
4.7 KiB
Go
package modelplan
|
|
|
|
import (
|
|
"math"
|
|
"regexp"
|
|
"strconv"
|
|
)
|
|
|
|
// How fast a model answers on a CPU, from its size and the node's memory.
|
|
//
|
|
// Writing a token reads every weight it uses once, so on a CPU the memory
|
|
// decides the speed: a token takes the bytes read over the rate llama.cpp
|
|
// reads them at, plus a little that does not shrink with the model.
|
|
//
|
|
// The rate is the threads': each reads at about what one core reads alone,
|
|
// a little less with every one added, up to what the whole machine reads.
|
|
// On a server of many memory channels a replica given four threads reads at
|
|
// a fraction of the machine's rate, and the machine's rate alone overstated
|
|
// it several times. Measured on the PoC's node05 on 2026-11-03 (8 vCPU,
|
|
// 43.9 GB/s with every CPU, 9.7 with one; llama-bench, 64 tokens):
|
|
//
|
|
// threads 1 2 4 8
|
|
// qwen2.5-coder-7b 1.9 3.6 6.3 9.3 tok/s 4.7 GB: 8.9 17.0 30.7 45.6 GB/s read
|
|
// qwen2.5-1.5b 8.3 15.2 25.4 37.0 tok/s 1.1 GB: 9.7 18.3 32.6 —
|
|
//
|
|
// so 0.95 of one core's rate times threads^0.9, up to the machine's, and
|
|
// 5 ms a token besides. Against what was measured before (2026-10-26):
|
|
// node01's 1.5b on two threads, 14.5 tok/s, is taken for 15.0; node05's 14b
|
|
// on four, 3.3, for 3.5. One reply at a time; several at once share the same
|
|
// reads and each is slower. The PoC's host has far fewer cores than its VMs,
|
|
// so these are what a busy host gives, and a machine of its own does better.
|
|
|
|
// cpuReadShare is the part of the whole machine's rate llama.cpp reads at,
|
|
// for a node whose one-core rate is not known: an agent older than its
|
|
// measurement.
|
|
const cpuReadShare = 0.68
|
|
|
|
// coreShare and threadExp are how threads add up: each about one core's
|
|
// rate, a little less with every one added.
|
|
const (
|
|
coreShare = 0.95
|
|
threadExp = 0.9
|
|
)
|
|
|
|
// cpuTokenFixed is a token's cost that does not grow with the model.
|
|
const cpuTokenFixed = 0.005
|
|
|
|
// CPUTokensPerSecond is about how many tokens a second one reply gets on a
|
|
// CPU whose memory reads bwGBs, from a model whose tokens each read bytes.
|
|
// 0 when either is unknown. Use CPUTokensPerSecondOn where the threads and
|
|
// one core's rate are known.
|
|
func CPUTokensPerSecond(bytes int64, bwGBs float64) float64 {
|
|
if bytes <= 0 || bwGBs <= 0 {
|
|
return 0
|
|
}
|
|
return 1 / (float64(bytes)/(bwGBs*1e9*cpuReadShare) + cpuTokenFixed)
|
|
}
|
|
|
|
// CPUReadGBs is the rate a replica given threads reads its weights at, on a
|
|
// node whose memory reads allGBs with every CPU and coreGBs with one. Without
|
|
// one core's rate or the threads, the share of the whole that small machines
|
|
// gave.
|
|
func CPUReadGBs(allGBs, coreGBs float64, threads int) float64 {
|
|
if allGBs <= 0 {
|
|
return 0
|
|
}
|
|
if coreGBs <= 0 || threads <= 0 {
|
|
return allGBs * cpuReadShare
|
|
}
|
|
return min(coreShare*coreGBs*math.Pow(float64(threads), threadExp), allGBs)
|
|
}
|
|
|
|
// CPUTokensPerSecondOn is about how many tokens a second one reply gets
|
|
// from a model whose tokens each read bytes, on threads of a node whose
|
|
// memory reads allGBs with every CPU and coreGBs with one. 0 when unknown.
|
|
func CPUTokensPerSecondOn(bytes int64, allGBs, coreGBs float64, threads int) float64 {
|
|
read := CPUReadGBs(allGBs, coreGBs, threads)
|
|
if bytes <= 0 || read <= 0 {
|
|
return 0
|
|
}
|
|
return 1 / (float64(bytes)/(read*1e9) + cpuTokenFixed)
|
|
}
|
|
|
|
// CPUThreads is how many threads a CPU replica runs on: the cores its
|
|
// profile reserves, else half the node's, which is what the agent gives
|
|
// llama.cpp then.
|
|
func CPUThreads(cpuCores, nodeCPUs int) int {
|
|
if cpuCores > 0 {
|
|
return cpuCores
|
|
}
|
|
return max(nodeCPUs/2, 1)
|
|
}
|
|
|
|
// Slow speeds: under the first a reply is a wait somebody notices, under the
|
|
// second one of a few hundred tokens takes minutes.
|
|
const (
|
|
CPUSlowTPS = 15
|
|
CPUTooSlowTPS = 5
|
|
)
|
|
|
|
// moeRead is what a token of a model of experts reads over its active
|
|
// share.
|
|
const moeRead = 1.6
|
|
|
|
// activeRE finds a model of experts' active parameters in its name, the
|
|
// way they are written: Qwen3-30B-A3B, Ling-flash-A2.5B.
|
|
var activeRE = regexp.MustCompile(`(?i)(?:^|[-_./])a(\d+(?:\.\d+)?)b(?:$|[-_.])`)
|
|
|
|
// ActiveShare is the part of a model's weights one token reads: for a model
|
|
// of experts named with its active parameters (…-30B-A3B) that over its
|
|
// total, otherwise 1 — and 1.6 times that, which is what Qwen3-Coder-30B-A3B
|
|
// read on cuda01's card (144 tok/s from 18.6 GB): the shared layers, the
|
|
// embedding and the routing are read for every token whatever the experts.
|
|
func ActiveShare(name string, params int64) float64 {
|
|
m := activeRE.FindStringSubmatch(name)
|
|
if m == nil || params <= 0 {
|
|
return 1
|
|
}
|
|
b, err := strconv.ParseFloat(m[1], 64)
|
|
if err != nil || b <= 0 {
|
|
return 1
|
|
}
|
|
if s := moeRead * b * 1e9 / float64(params); s < 1 {
|
|
return s
|
|
}
|
|
return 1
|
|
}
|