How fast a model would answer on the cards, as this cluster would lay it out
For each layout modelplan.Options offers here — one card, tensor parallel across a node's cards, llama.cpp's layer split, a pipeline across nodes on a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes a token reads over each card's bandwidth, by its model name (a table, SXM and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on NVLink, 25 us and up on PCIe), turns between cards for a layer split, and a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a repository's files show the fastest, with the layout; mixed classes are judged class by class, FP8 on 8.9 cards only. Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB, 62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB, 144 measured, which says a model of experts reads 1.6 times its active share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics are not measured here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e70d6caa2f
commit
f2c6f12cfb
13 files changed
+449
-14
No files matched your search
@@ -302,6 +302,7 @@
|
||||
{{if .Rows}}
|
||||
<table class="rows explore">
|
||||
<tr><th>{{t "explore.col.repo"}}</th><th class="num">{{t "explore.col.size"}}</th><th>{{t "explore.col.arch"}}</th><th class="num">{{t "explore.col.context"}}</th>
|
||||
<th class="num" title="{{t "explore.col.gpu_title"}}">{{t "explore.col.gpu"}}</th>
|
||||
<th class="num" title="{{t "explore.col.cpu_title"}}">{{t "explore.col.cpu"}}</th>
|
||||
<th class="num">{{t "explore.col.downloads"}}</th><th class="num">{{t "explore.col.likes"}}</th><th class="num">{{t "explore.col.updated"}}</th></tr>
|
||||
{{range .Rows}}<tr class="explore-row">
|
||||
@@ -311,6 +312,7 @@
|
||||
<td class="num">{{.ParamsLabel}}</td>
|
||||
<td>{{if .Architecture}}<span{{if not .Verdict.Servable}} class="cannot"{{end}} title="{{.Verdict.Reason}}">{{.Architecture}}{{if .Verdict.Checked}}{{if .Verdict.Servable}} ✓{{else}} ✗{{end}}{{end}}</span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{.ContextLabel}}</td>
|
||||
<td class="num">{{if .GPUSpeed}}<span title="{{.GPULayout}}">{{.GPUSpeed}}</span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{if .CPUSpeed}}<span title="{{t "explore.cpu_on" .CPUNode}}">{{.CPUSpeed}}</span>{{else}}—{{end}}</td>
|
||||
<td class="num">{{compact .Downloads}}</td>
|
||||
<td class="num">{{compact .Likes}}</td>
|
||||
@@ -470,6 +472,7 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
{{with .Companions}}<span class="opt" title="{{join .}}">{{t "card.companions" (len .)}}</span>{{end}}
|
||||
{{if eq .Kind "adapter"}}<span class="tag new" title="{{t "card.adapter_title" .Arch}}">{{t "card.adapter" .Arch}}</span>
|
||||
{{else if eq .Kind "model"}}<span class="opt">{{if .Arch}}{{t "card.model_arch" .Arch}}{{else}}{{t "card.model"}}{{end}}</span>{{end}}
|
||||
{{if .GPUSpeed}}<span class="opt" title="{{t "explore.col.gpu_title"}}">{{t "card.gpu_speed" .GPUSpeed .GPULayout}}</span>{{end}}
|
||||
{{if .CPUSpeed}}<span class="opt" title="{{t "explore.col.cpu_title"}}">{{t "card.cpu_speed" .CPUSpeed .CPUNode}}</span>{{end}}</td>
|
||||
<td class="num">{{bytes .Bytes}}</td>
|
||||
<td>{{with .SHA256}}<code title="{{.}}">{{slice . 0 12}}</code>{{else}}—{{end}}</td>
|
||||
|
||||
Reference in new issue
Block a user