How fast a model would answer on a CPU, before it is fetched
CI / test (push) Failing after 6m37s

Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).

The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-27 00:09:47 +01:00
1 parent 2a45068461
commit 84444e71c8
20 files changed
+388 -7

No files matched your search

+9
View File
@@ -154,6 +154,15 @@ Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
see everything. On a model's page a file too big for the cluster is shown
without a **Fetch** button, with what it would need.
The **CPU tok/s** column is about how fast one reply would come on the CPU of
the fastest node whose memory holds the model at Q4_K_M; a model's page gives
it for each GGUF file by its own size. Every node's agent measures how fast its
memory reads when it starts (it is in the agent's log as `memory_gbs`), and a
token on a CPU reads every weight once: the estimate is that speed over the
model's size — for a model of experts named like `30B-A3B`, its active part.
On the test cluster it came within 15% of what llama.cpp measured. Several
replies at once share the reads and each is slower.
1. Click a model to read its card.
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
**Fetch**. It is downloaded into the mirror once, for every node.