Fetch a sharded model in one action, not one shard at a time
CI / test (push) Successful in 1m39s

The explorer listed every weight file with its own button. For a repository of
alternatives that is right — q4_k_m or q8_0, you want one — and for a sharded
model it is fifteen clicks to fetch one thing, with nothing stopping you from
losing count and leaving a version with fourteen of its parts.

Files are grouped into sets now. A run of "-00002-of-00004" is one row and one
button; everything else stays its own row.

Not "fetch all the weights", which would be wrong half the time: two safetensors
files are halves of one thing and useless apart, two GGUF files are usually
alternatives and fetching both spends gigabytes on a copy nobody asked for. The
set is the unit that is whole.

A repository can hold both the split and the joined copy of the same
quantisation — the real Qwen coder repos do — so a shard set keeps the -of-
notation in its name rather than collapsing onto the name of the whole file
beside it.

A set publishes no single checksum. Several files have several, and showing one
of them invites comparing it against the wrong thing.

The form posts every source in the set, and a failure part way says how many
already started: a set that fails on its fourth shard has three arriving, and
reading that as "nothing happened" sends somebody to fetch them all again.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 15:32:27 +02:00
1 parent ed530133b6
commit a588646b7c
5 files changed
+277 -31

No files matched your search

+16 -10
View File
@@ -120,18 +120,20 @@ directory, or give the mirror a token.</p>
{{if .Repo}}
<h3>{{.Repo.ID}} <span class="opt">at {{.Repo.ShortRevision}}</span></h3>
{{if .RepoFiles}}
{{if .RepoSets}}
<table>
<tr><th>File</th><th>Size</th><th>Published checksum</th><th></th></tr>
{{range .RepoFiles}}<tr>
<td><code>{{.Path}}</code></td>
<tr><th>What</th><th>Size</th><th>Published checksum</th><th></th></tr>
{{range .RepoSets}}<tr>
<td><code>{{.Name}}</code>
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}</td>
<td class="num">{{bytes .Bytes}}</td>
<td>{{if .SHA256}}<code title="{{.SHA256}}">{{slice .SHA256 0 12}}</code>{{else}}—{{end}}</td>
<td>{{with .SHA256}}<code title="{{.}}">{{slice . 0 12}}</code>{{else}}—{{end}}</td>
<td>
<form method="post" action="/admin/mirror/fetch" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
{{range .Sources}}<input type="hidden" name="source" value="{{.URL}}">
<input type="hidden" name="file" value="{{.Path}}">
{{end}}
<input type="text" name="model" value="{{.Model}}" size="24">
<input type="text" name="version" value="{{.Version}}" size="10">
<button type="submit">Fetch</button>
@@ -140,10 +142,14 @@ directory, or give the mirror a token.</p>
</tr>{{end}}
</table>
<p class="note">Only the weights are listed; a repository is mostly a README and
a tokenizer. The source URL is pinned to <code>{{.Repo.ShortRevision}}</code>
rather than to a branch, so what arrives is what is listed here even if the
branch moves in between — and the checksum above is what the registry publishes,
to compare with what the mirror computes once it has the file.</p>
a tokenizer. Shards of one file are one row and one button — a fifteen-shard
model is one decision, not fifteen. Two separate files are two rows because they
are usually alternatives: one quantisation or the other, not both.</p>
<p class="note">The source URLs are pinned to
<code>{{.Repo.ShortRevision}}</code> rather than to a branch, so what arrives is
what is listed here even if the branch moves in between — and a single file's
checksum above is what the registry publishes, to compare with what the mirror
computes once it has it.</p>
{{else}}
<p class="empty">No weight files in {{.Repo.ID}}.</p>
{{end}}