Each agent measures its node's memory read rate when it starts: every CPU sums a share of a buffer past any cache, a tenth of free memory at most, the best of four passes, in a fraction of a second; reported in Host (mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it uses once, so one reply's speed is about that rate over those bytes: modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token, calibrated on the PoC against llama.cpp's predicted_per_second (node01 24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s, qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active part (ActiveShare). The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that holds the model; a repository's page gives it for each GGUF by its size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
2a45068461
commit
84444e71c8
20 files changed
+388
-7
No files matched your search
@@ -154,6 +154,15 @@ Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
|
||||
see everything. On a model's page a file too big for the cluster is shown
|
||||
without a **Fetch** button, with what it would need.
|
||||
|
||||
The **CPU tok/s** column is about how fast one reply would come on the CPU of
|
||||
the fastest node whose memory holds the model at Q4_K_M; a model's page gives
|
||||
it for each GGUF file by its own size. Every node's agent measures how fast its
|
||||
memory reads when it starts (it is in the agent's log as `memory_gbs`), and a
|
||||
token on a CPU reads every weight once: the estimate is that speed over the
|
||||
model's size — for a model of experts named like `30B-A3B`, its active part.
|
||||
On the test cluster it came within 15% of what llama.cpp measured. Several
|
||||
replies at once share the reads and each is slower.
|
||||
|
||||
1. Click a model to read its card.
|
||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||
**Fetch**. It is downloaded into the mirror once, for every node.
|
||||
|
||||
Reference in new issue
Block a user