Commit Graph
5 Commits
Author SHA1 Message Date
Ilker Manap d6f8f433f4 CHANGELOG: one list
CI / test (push) Successful in 9m18s
2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 6a870a8c15 The explorer's card figures: MIG, NVLink, GH200, FP8 and FP4
CI / test (push) Successful in 9m13s
From review. A MIG instance is given its share of the card's bandwidth,
not the whole card's. NVLink between a layout's cards means all of them
joined to each other, the largest such set on one ready node, not a count
of cards with any peer: two bridged pairs are not four on NVLink. GH200 is
matched before H200. An FP8 checkpoint is listed for cards of 8.0 and up,
which vLLM runs with Marlin; 8.9 is the agent's bound for --quantization
fp8. NVFP4 repositories tagged fp4 are sized at four bits.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:38:38 +01:00
Ilker ManapandClaude Opus 5.5 f2c6f12cfb How fast a model would answer on the cards, as this cluster would lay it out
CI / test (push) Successful in 9m1s
For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.

Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 00:19:36 +01:00
Ilker ManapandClaude Opus 5.5 84444e71c8 How fast a model would answer on a CPU, before it is fetched
CI / test (push) Failing after 6m37s
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).

The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 00:09:47 +01:00
Ilker ManapandClaude Opus 5.5 2a45068461 The explorer lists what this cluster's hardware holds
CI / test (push) Successful in 8m58s
Judged by the nodes as they are now, through modelplan.Options, the
planner that serving a model uses: GGUF at Q4_K_M from the parameter count,
safetensors at the size the registry's dtype breakdown gives (packed AWQ,
GPTQ and MXFP4 at four bits), FP8 only where a card has compute capability
8.9, MLX safetensors never. A picture model is judged against one card. A
cluster with cards asks the registry for safetensors too, a hundred at a
time; one without asks for GGUF only. The page says what it judged by: the
largest card, its capability, the most RAM on one node.

On a repository's page a file set no node holds is marked with what it
needs, against what the best node offers, and has no Fetch button.

Tried against the PoC's seven nodes and Hugging Face's trending hundred:
63 shown, the rest too big for one 24 GB Turing card or FP8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 18:30:16 +01:00