Serve a model made of several files
CI / test (push) Successful in 1m42s

The mirror would happily hold a four-shard model that nothing could serve. A
catalog entry names one source, the agent fetched one URL, and so a version of
several files had no way to be described — which is every safetensors model,
and every GGUF too large to ship whole.

A source ending in a slash names a version rather than a file. The agent fetches
every file in it into one directory and hands the engine the directory: vLLM
reads the shards and the config out of the folder, and llama.cpp is given the
first part and finds the rest beside it.

The trailing slash is the whole signal. A flag beside the source would be a
second place for the two to disagree, and a URL already distinguishes a
directory from a file everywhere else.

The mirror answers a version with a manifest instead of a file listing. A plain
file server gives HTML there, and something that has to know what a version
consists of would be left parsing it. It is on the byte listener, not the
management API: fetching weights is the one thing every node does, and making
it depend on the administrative port would undo the split that keeps that port
closeable.

Built from the directory rather than a file written beside it. A manifest on
disk is a second record of what is there, and the two disagree the first time
somebody copies a file in by hand — which the mirror explicitly allows.

The whole set or nothing. A directory with three of four shards loads as a
broken model rather than failing to load, so a completion marker is written
last and an interrupted fetch starts the missing files again rather than being
found and served. Files already there at the right size are not fetched twice.

Progress is reported under the version's address, not each file's: the shards
waiting on it asked for the version, and progress under a file's own URL
reaches nobody.

A filename from the manifest is refused if it contains a separator. The mirror
is a place an operator copies files into by hand, so what it reports is not
automatically safe to join onto a path.

An HTML listing where a manifest was expected says so. "invalid character '<'"
sends somebody looking at the wrong thing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 21:07:26 +02:00
1 parent c95cfbe426
commit dfbb7557dd
10 files changed
+554 -23

No files matched your search

+21 -9
View File
@@ -32,6 +32,8 @@ type serveForm struct {
// be written later. Named for the hardware rather than the model, because
// a profile is about where a model runs.
Profile string
// Files is how many files the version holds, above one for a set.
Files int
// MinMemoryGB is the host RAM to ask for, estimated from the weights.
//
// Filled in rather than left blank. Blank writes no constraint at all, and
@@ -63,16 +65,26 @@ func planServe(st mirror.Status, files []mirror.Status, base string,
Profile: "cpu",
}
// A catalogue entry names one source. A version made of several files has
// no single one, so an entry for it would be an entry for one shard — which
// loads as a broken model rather than failing to load.
// A version of several files is served as the version, not as one of them.
// The source is the directory — a trailing slash — and the agent fetches
// every file in it and hands the engine the folder.
if len(files) > 1 {
f.Refusal = fmt.Sprintf(
"this version is %d files, and a catalogue entry names one source. "+
"Serve a single-file build (a GGUF), or give the engine the "+
"repository id so it fetches the shards itself",
len(files))
return f
f.Source = strings.TrimSuffix(
mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
f.Files = len(files)
// From whichever file says, because a version is a mixture only when
// somebody has put one there by hand. The weights decide the engine.
for _, one := range files {
if e := engineFor(one.File); e != "" {
f.Engine = e
break
}
}
var total int64
for _, one := range files {
total += one.BytesDone
}
st.BytesTotal = total
}
switch f.Engine {