How fast a model would answer on the cards, as this cluster would lay it out
CI / test (push) Successful in 9m1s

For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.

Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-28 00:19:36 +01:00
1 parent e70d6caa2f
commit f2c6f12cfb
13 files changed
+449 -14

No files matched your search

+3
View File
@@ -96,6 +96,8 @@ type exploreRow struct {
// CPUSpeed is about how many tokens a second one reply would get on the
// CPU of CPUNode, the fastest node that holds it; empty when unknown.
CPUSpeed, CPUNode string
// GPUSpeed is the same on the cards, laid out as GPULayout says.
GPUSpeed, GPULayout string
}
// ParamsLabel is the parameter count as people say it: 7.6B, 494M.
@@ -316,6 +318,7 @@ func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
}
if fit != nil {
row.CPUSpeed, row.CPUNode = fit.rowSpeed(m)
row.GPUSpeed, row.GPULayout = fit.rowGPU(l, m)
}
if e.Here && row.Verdict.Checked && !row.Verdict.Servable {
e.Hidden++