Judged by the nodes as they are now, through modelplan.Options, the planner that serving a model uses: GGUF at Q4_K_M from the parameter count, safetensors at the size the registry's dtype breakdown gives (packed AWQ, GPTQ and MXFP4 at four bits), FP8 only where a card has compute capability 8.9, MLX safetensors never. A picture model is judged against one card. A cluster with cards asks the registry for safetensors too, a hundred at a time; one without asks for GGUF only. The page says what it judged by: the largest card, its capability, the most RAM on one node. On a repository's page a file set no node holds is marked with what it needs, against what the best node offers, and has no Fetch button. Tried against the PoC's seven nodes and Hugging Face's trending hundred: 63 shown, the rest too big for one 24 GB Turing card or FP8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
e84631ce3a
commit
2a45068461
11 files changed
+458
-18
No files matched your search
@@ -146,7 +146,13 @@ agent is stopped.
|
||||
## 4. Add a model
|
||||
|
||||
Open `/admin/mirror` and choose **Explore models**. It lists what is popular on
|
||||
Hugging Face, showing only what your nodes can run.
|
||||
Hugging Face, showing only what your nodes can run: judged by the cluster as it
|
||||
is now — the largest card and its compute capability, the most memory on one
|
||||
node — so a model too big for them, in a form those cards lack (FP8 needs 8.9),
|
||||
or in neither GGUF nor safetensors is left out, and the page says how many.
|
||||
Without GPUs it lists GGUF only. Untick **only what these nodes can run** to
|
||||
see everything. On a model's page a file too big for the cluster is shown
|
||||
without a **Fetch** button, with what it would need.
|
||||
|
||||
1. Click a model to read its card.
|
||||
2. Pick one of its files, usually a `q4_k_m` for a first try, and click
|
||||
|
||||
Reference in new issue
Block a user