Serve a model made of several files
CI / test (push) Successful in 1m42s

The mirror would happily hold a four-shard model that nothing could serve. A
catalog entry names one source, the agent fetched one URL, and so a version of
several files had no way to be described — which is every safetensors model,
and every GGUF too large to ship whole.

A source ending in a slash names a version rather than a file. The agent fetches
every file in it into one directory and hands the engine the directory: vLLM
reads the shards and the config out of the folder, and llama.cpp is given the
first part and finds the rest beside it.

The trailing slash is the whole signal. A flag beside the source would be a
second place for the two to disagree, and a URL already distinguishes a
directory from a file everywhere else.

The mirror answers a version with a manifest instead of a file listing. A plain
file server gives HTML there, and something that has to know what a version
consists of would be left parsing it. It is on the byte listener, not the
management API: fetching weights is the one thing every node does, and making
it depend on the administrative port would undo the split that keeps that port
closeable.

Built from the directory rather than a file written beside it. A manifest on
disk is a second record of what is there, and the two disagree the first time
somebody copies a file in by hand — which the mirror explicitly allows.

The whole set or nothing. A directory with three of four shards loads as a
broken model rather than failing to load, so a completion marker is written
last and an interrupted fetch starts the missing files again rather than being
found and served. Files already there at the right size are not fetched twice.

Progress is reported under the version's address, not each file's: the shards
waiting on it asked for the version, and progress under a file's own URL
reaches nobody.

A filename from the manifest is refused if it contains a separator. The mirror
is a place an operator copies files into by hand, so what it reports is not
automatically safe to join onto a path.

An HTML listing where a manifest was expected says so. "invalid character '<'"
sends somebody looking at the wrong thing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-27 21:07:26 +02:00
1 parent c95cfbe426
commit dfbb7557dd
10 files changed
+554 -23

No files matched your search

+46 -1
View File
@@ -1,6 +1,7 @@
package cli
import (
"encoding/json"
"fmt"
"log/slog"
"net/http"
@@ -264,7 +265,7 @@ func newMirrorServeCmd(dir *string) *cobra.Command {
// by me.
files := &http.Server{
Addr: listen,
Handler: readOnly(http.FileServer(http.Dir(*dir))),
Handler: readOnly(withManifest(*dir, http.FileServer(http.Dir(*dir)))),
ReadHeaderTimeout: 10 * time.Second,
}
admin := &http.Server{
@@ -353,3 +354,47 @@ func catalogSources(cmd *cobra.Command, server string) ([]string, error) {
}
return out, nil
}
// withManifest answers a request for a version with the list of files in it.
//
// A directory otherwise gets FileServer's HTML index, and something that has to
// know what a version consists of would be left parsing it. A model is
// routinely several files — sharded safetensors, or a GGUF split across parts —
// and a catalog entry names one source, so without a way to ask "what is in
// this version" a sharded model can be mirrored and never served.
//
// On the byte listener rather than the management API on purpose. Fetching
// weights is the one thing every node does, and making it depend on the
// administrative port would undo the split that keeps that port closeable.
func withManifest(root string, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
model, version, ok := versionPath(r.URL.Path)
if !ok {
next.ServeHTTP(w, r)
return
}
m, err := mirror.ManifestFor(root, model, version)
if err != nil {
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(m)
})
}
// versionPath matches "/model/version/" and nothing else.
//
// The trailing slash is the whole signal: "/m/v/file.gguf" is a file and
// "/m/v/" is the set it belongs to. Requiring it means a source URL says which
// of the two it means without a flag beside it.
func versionPath(p string) (model, version string, ok bool) {
if !strings.HasSuffix(p, "/") {
return "", "", false
}
parts := strings.Split(strings.Trim(p, "/"), "/")
if len(parts) != 2 || parts[0] == "" || parts[1] == "" {
return "", "", false
}
return parts[0], parts[1], true
}