How fast a model would answer on the cards, as this cluster would lay it out
CI / test (push) Successful in 9m1s
CI / test (push) Successful in 9m1s
For each layout modelplan.Options offers here — one card, tensor parallel across a node's cards, llama.cpp's layer split, a pipeline across nodes on a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes a token reads over each card's bandwidth, by its model name (a table, SXM and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on NVLink, 25 us and up on PCIe), turns between cards for a layer split, and a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a repository's files show the fastest, with the layout; mixed classes are judged class by class, FP8 on 8.9 cards only. Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB, 62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB, 144 measured, which says a model of experts reads 1.6 times its active share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics are not measured here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e70d6caa2f
commit
f2c6f12cfb
13 files changed
+449
-14
No files matched your search
@@ -163,6 +163,19 @@ model's size — for a model of experts named like `30B-A3B`, its active part.
|
||||
On the test cluster it came within 15% of what llama.cpp measured. Several
|
||||
replies at once share the reads and each is slower.
|
||||
|
||||
**GPU tok/s** is the same on the cards, for the layout this cluster would
|
||||
give the model — the fewest cards that hold it, of each class, the fastest of
|
||||
those; its tooltip says which ("2× h100-80g, NVLink, vLLM", "2 nodes × 1×
|
||||
a100-80g over ib0 (InfiniBand 200 Gb/s), vLLM"). A card's memory bandwidth is
|
||||
known from its model name (an H100 SXM and an H100 PCIe differ by half); a card
|
||||
not in the table gets no figure. Tensor parallel across a node's cards divides
|
||||
what each reads, and adds two all-reduces a layer, short over NVLink and
|
||||
longer over PCIe; llama.cpp across cards splits by layer, so the cards take
|
||||
turns and a reply is no faster than on one; a pipeline across nodes takes turns
|
||||
too, and adds a hop over the fabric for each node. One card's figures were
|
||||
measured on the test cluster's RTX 6000; tensor parallel and the fabric are
|
||||
from what NCCL is known to take, not measured here.
|
||||
|
||||
1. Click a model to read its card.
|
||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||
**Fetch**. It is downloaded into the mirror once, for every node.
|
||||
|
||||
Reference in new issue
Block a user