Each agent measures its node's memory read rate when it starts: every CPU sums a share of a buffer past any cache, a tenth of free memory at most, the best of four passes, in a fraction of a second; reported in Host (mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it uses once, so one reply's speed is about that rate over those bytes: modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token, calibrated on the PoC against llama.cpp's predicted_per_second (node01 24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s, qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active part (ActiveShare). The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that holds the model; a repository's page gives it for each GGUF by its size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2a45068461
commit
84444e71c8
20 files changed
+388
-7
No files matched your search
@@ -93,6 +93,9 @@ type exploreRow struct {
|
||||
New bool
|
||||
// Updated is how long ago the repository changed, in the page's language.
|
||||
Updated string
|
||||
// CPUSpeed is about how many tokens a second one reply would get on the
|
||||
// CPU of CPUNode, the fastest node that holds it; empty when unknown.
|
||||
CPUSpeed, CPUNode string
|
||||
}
|
||||
|
||||
// ParamsLabel is the parameter count as people say it: 7.6B, 494M.
|
||||
@@ -311,6 +314,9 @@ func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
|
||||
e.Hidden++
|
||||
continue
|
||||
}
|
||||
if fit != nil {
|
||||
row.CPUSpeed, row.CPUNode = fit.rowSpeed(m)
|
||||
}
|
||||
if e.Here && row.Verdict.Checked && !row.Verdict.Servable {
|
||||
e.Hidden++
|
||||
continue
|
||||
|
||||
Reference in new issue
Block a user