How fast a model would answer on a CPU, before it is fetched
Each agent measures its node's memory read rate when it starts: every CPU sums a share of a buffer past any cache, a tenth of free memory at most, the best of four passes, in a fraction of a second; reported in Host (mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it uses once, so one reply's speed is about that rate over those bytes: modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token, calibrated on the PoC against llama.cpp's predicted_per_second (node01 24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s, qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active part (ActiveShare). The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that holds the model; a repository's page gives it for each GGUF by its size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2a45068461
commit
84444e71c8
20 files changed
+388
-7
No files matched your search
@@ -248,6 +248,9 @@ type NodeView struct {
|
||||
MemoryGB int `json:"memory_gb,omitempty"`
|
||||
// CPUCount is the machine's CPUs, 0 when the agent did not report them.
|
||||
CPUCount int `json:"cpu_count,omitempty"`
|
||||
// MemBandwidthGBs is how fast its memory reads, in GB/s, measured when
|
||||
// its agent started; 0 when not reported. A model's speed on the CPU.
|
||||
MemBandwidthGBs float64 `json:"mem_bandwidth_gbs,omitempty"`
|
||||
// Engines are the inference engines installed here, as the agent found
|
||||
// them. Absent means the agent is older than the probe and said nothing,
|
||||
// which is not the same as having none — placement treats it that way too,
|
||||
@@ -703,6 +706,7 @@ func (a *API) Nodes() []NodeView {
|
||||
v.EngineBuilds[e] = b
|
||||
}
|
||||
}
|
||||
v.MemBandwidthGBs = n.Host.MemBandwidthGBs
|
||||
if h := n.Host; h.OSID != "" {
|
||||
v.OS = strings.TrimSpace(h.OSID + " " + h.OSVersion + " " + h.Arch)
|
||||
if h.Arch == "amd64" && !h.HasCPUFlag("avx2") {
|
||||
|
||||
Reference in new issue
Block a user