Serve vision models with their projector and original weights on vLLM; group the mirror by version

model serve takes --projector, fetched beside the weights and loaded by
llama.cpp with --mmproj, and --format safetensors with --gpu-class and --tp,
which fetches the weights with their config and tokenizer and runs them on
vLLM. The console offers both, F switching between GGUF and original
weights. The mirror view is one row per version with its files on Enter.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 00:42:59 +02:00
1 parent 7aefb236d6
commit a16097cfe8
12 files changed
+509 -81

No files matched your search

+19
View File
@@ -76,6 +76,25 @@ is not an update.
A model from `server.yaml` is reported but not changed: its source is edited A model from `server.yaml` is reported but not changed: its source is edited
there. there.
### Models of several files: vision projectors, original weights
- A vision model's projector (`mmproj`) is fetched beside its GGUF weights
and loaded with them: `llmgw ctl model serve REPO --projector FILE`, or the
choice offered by the console. The model is marked as taking images. Nodes
load it once their agent is this version or later.
- `--format safetensors` serves a repository's original weights, with their
config.json, tokenizer and shard index, on vLLM across `--tp` GPUs of
`--gpu-class` (the fleet's one class when there is one). In the console,
**F** on Serve a model switches to them.
- The mirror keeps where it fetched each file beside it, and reports that
and the checksum after it restarts. The console's Mirror view shows one
row per version of a model, with its files on Enter.
- An update watches every file of a model, the projector and the tokenizer
included; one whose files the mirror cannot trace is found on the
registry by name, commit and size, or given with `llmgw ctl model origin`.
- `llmgw_model_update_available` is published, and `ModelUpdateAvailable`
alerts on it.
### The manual, in the binary ### The manual, in the binary
`llmgw docs` reads every document — installation, operation, the `llmgw docs` reads every document — installation, operation, the
+19 -9
View File
@@ -111,9 +111,13 @@ which): requests, tokens, errors and spend.
team or project is guaranteed and may at most hold, and what it holds now. team or project is guaranteed and may at most hold, and what it holds now.
**A** adds one; on a row, change or remove it. **A** adds one; on a row, change or remove it.
**Mirror** is what the model mirror holds, how far each download has got, and **Mirror** is what the model mirror holds, one row for each version of each
which models use each file. **A** downloads a URL into it; on a row, where model — a file, or the set of files a model of several is: shards, a vision
nodes fetch it from, or remove it — with a warning when a model still uses it. projector, a config and tokenizer — with how far its download has got and
which models use it. **A** downloads a URL into it; on a row, its files with
their checksums and where each was fetched from, the source a model
definition names for it, or remove it — with a warning when a model still
uses it.
**Logs** shows a node's agent log, newest at the bottom, errors and warnings **Logs** shows a node's agent log, newest at the bottom, errors and warnings
coloured. coloured.
@@ -181,12 +185,18 @@ to restart it. Another control plane keeps its own copy.
certificate and starts its agent (`sudo llmgw node add`). certificate and starts its agent (`sudo llmgw node add`).
- **Replicate the store** — moves the store onto two more machines under - **Replicate the store** — moves the store onto two more machines under
Patroni (offered only while it is not replicated). Patroni (offered only while it is not replicated).
- **Serve a model** — Hugging Face's GGUF models: trending, most downloaded, - **Serve a model** — Hugging Face's models: trending, most downloaded,
most liked or newest (**S** changes which), or **/** to search. Enter on most liked or newest (**S** changes which), or **/** to search. **F**
one lists its files with their sizes; the one chosen is fetched into the switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
mirror, checked, added to the catalog and started, and a replica is asked original safetensors weights, which vLLM runs on GPUs. Enter on a GGUF
to say hello (`llmgw ctl model serve`). `mirror.registry` points it at repository lists its files with their sizes; a vision model offers its
another registry, for a site without the internet. projector too, which is fetched beside it and loaded with it so the model
reads images. Enter on original weights offers them with their config and
tokenizer, a GPU class and how many GPUs a replica spans. What is chosen is
fetched into the mirror, checked, added to the catalog and started, and a
replica is asked to say hello (`llmgw ctl model serve`).
`mirror.registry` points it at another registry, for a site without the
internet.
- **Upgrade the cluster** to a release: this machine, then every node in turn - **Upgrade the cluster** to a release: this machine, then every node in turn
(`sudo llmgw ctl upgrade`). (`sudo llmgw ctl upgrade`).
- **Restart all replicas**, one at a time. - **Restart all replicas**, one at a time.
+29 -8
View File
@@ -93,7 +93,7 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
bin := l.bin() bin := l.bin()
p := s.Profile p := s.Profile
model, err := ggufIn(p.ModelPath) model, projector, err := ggufIn(p.ModelPath)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -120,6 +120,11 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
if p.ServedModelName != "" { if p.ServedModelName != "" {
args = append(args, "--alias", p.ServedModelName) args = append(args, "--alias", p.ServedModelName)
} }
// A vision model's projector, fetched beside it: what turns an image in
// a request into something the language model reads.
if projector != "" && !hasFlag(p.Args, "--mmproj", "-mm") {
args = append(args, "--mmproj", projector)
}
if !hasFlag(p.Args, "--parallel", "-np") { if !hasFlag(p.Args, "--parallel", "-np") {
n := l.cfg.Parallel n := l.cfg.Parallel
if n <= 0 { if n <= 0 {
@@ -203,30 +208,46 @@ func hasFlag(args []string, names ...string) bool {
// //
// A path that is already a file is returned untouched, which is every // A path that is already a file is returned untouched, which is every
// single-file model and every hand-staged one. // single-file model and every hand-staged one.
func ggufIn(path string) (string, error) { //
// A vision projector in the directory — an mmproj file — is not the model:
// it is returned second, to be loaded beside it.
func ggufIn(path string) (model, projector string, err error) {
st, err := os.Stat(path) st, err := os.Stat(path)
if err != nil || !st.IsDir() { if err != nil || !st.IsDir() {
// Not a directory, or not there at all — the second is the engine's to // Not a directory, or not there at all — the second is the engine's to
// report, with the message it would have given anyway. // report, with the message it would have given anyway.
return path, nil //nolint:nilerr // a missing path is the engine's error to give return path, "", nil //nolint:nilerr // a missing path is the engine's error to give
} }
entries, err := os.ReadDir(path) entries, err := os.ReadDir(path)
if err != nil { if err != nil {
return "", err return "", "", err
} }
var names []string var names, projectors []string
for _, e := range entries { for _, e := range entries {
if !e.IsDir() && strings.HasSuffix(strings.ToLower(e.Name()), ".gguf") { lower := strings.ToLower(e.Name())
if e.IsDir() || !strings.HasSuffix(lower, ".gguf") {
continue
}
if strings.Contains(lower, "mmproj") {
projectors = append(projectors, e.Name())
} else {
names = append(names, e.Name()) names = append(names, e.Name())
} }
} }
if len(names) == 0 { if len(names) == 0 {
return "", fmt.Errorf("%s holds no .gguf file", path) if len(projectors) > 0 {
return "", "", fmt.Errorf("%s holds a vision projector and no model", path)
}
return "", "", fmt.Errorf("%s holds no .gguf file", path)
} }
// Sorted, so a split model's first part comes first: the parts are named // Sorted, so a split model's first part comes first: the parts are named
// "-00001-of-00004" and lexical order is shard order. // "-00001-of-00004" and lexical order is shard order.
sort.Strings(names) sort.Strings(names)
return filepath.Join(path, names[0]), nil if len(projectors) > 0 {
sort.Strings(projectors)
projector = filepath.Join(path, projectors[0])
}
return filepath.Join(path, names[0]), projector, nil
} }
// UseInstalled points the engine at a build the agent installed and checked. // UseInstalled points the engine at a build the agent installed and checked.
+29 -4
View File
@@ -204,7 +204,7 @@ func TestASetDirectoryResolvesToItsFirstPart(t *testing.T) {
} }
} }
got, err := ggufIn(dir) got, _, err := ggufIn(dir)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -221,7 +221,7 @@ func TestAPlainFilePathIsUntouched(t *testing.T) {
if err := os.WriteFile(p, []byte("x"), 0o644); err != nil { if err := os.WriteFile(p, []byte("x"), 0o644); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if got, _ := ggufIn(p); got != p { if got, _, _ := ggufIn(p); got != p {
t.Errorf("expected the path unchanged, got %s", got) t.Errorf("expected the path unchanged, got %s", got)
} }
} }
@@ -231,7 +231,7 @@ func TestADirectoryWithNoWeightsSaysSo(t *testing.T) {
if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o644); err != nil { if err := os.WriteFile(filepath.Join(dir, "config.json"), []byte("{}"), 0o644); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := ggufIn(dir); err == nil { if _, _, err := ggufIn(dir); err == nil {
t.Error("a directory with no .gguf in it is not a model llama.cpp can load") t.Error("a directory with no .gguf in it is not a model llama.cpp can load")
} }
} }
@@ -240,7 +240,7 @@ func TestAMissingPathIsLeftToTheEngine(t *testing.T) {
// The engine's own "no such file" names the path and the flag it came // The engine's own "no such file" names the path and the flag it came
// from, which is more use than anything this could say. // from, which is more use than anything this could say.
p := "/nonexistent/model.gguf" p := "/nonexistent/model.gguf"
got, err := ggufIn(p) got, _, err := ggufIn(p)
if err != nil || got != p { if err != nil || got != p {
t.Errorf("expected it passed through, got %q %v", got, err) t.Errorf("expected it passed through, got %q %v", got, err)
} }
@@ -361,3 +361,28 @@ func TestLlamaCppSpreadsAReplicaOverItsNodesGPUsByLayer(t *testing.T) {
t.Error("a one-GPU replica was given a split mode") t.Error("a one-GPU replica was given a split mode")
} }
} }
func TestAVisionProjectorIsLoadedBesideTheModelNotAsIt(t *testing.T) {
dir := t.TempDir()
// "mmproj" sorts before the model's name; it must still not be taken for it.
for _, n := range []string{"gemma-3-4b-it-Q4_K_M.gguf", "mmproj-gemma-3-4b-it-f16.gguf"} {
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
t.Fatal(err)
}
}
cmds, err := (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, Profile: Profile{ModelPath: dir}})
if err != nil {
t.Fatal(err)
}
args := strings.Join(cmds[0].Args, " ")
if !strings.Contains(args, "--model "+filepath.Join(dir, "gemma-3-4b-it-Q4_K_M.gguf")) ||
!strings.Contains(args, "--mmproj "+filepath.Join(dir, "mmproj-gemma-3-4b-it-f16.gguf")) {
t.Errorf("%s", args)
}
// One the profile names itself is left as it is.
cmds, _ = (&LlamaCpp{}).Commands(ShardSpec{HTTPPort: 8080, Profile: Profile{ModelPath: dir,
Args: []string{"--mmproj", "/elsewhere.gguf"}}})
if strings.Count(strings.Join(cmds[0].Args, " "), "--mmproj") != 1 {
t.Errorf("two projectors: %v", cmds[0].Args)
}
}
+18 -1
View File
@@ -8,6 +8,7 @@ import (
"github.com/spf13/cobra" "github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve" "gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
) )
@@ -24,15 +25,27 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
"that the nodes' llama.cpp can load it and sizes its memory from its own\n" + "that the nodes' llama.cpp can load it and sizes its memory from its own\n" +
"header, adds it to the catalog, waits for a replica, and asks it to say\n" + "header, adds it to the catalog, waits for a replica, and asks it to say\n" +
"hello. Each step is shown as it happens; the first that fails stops it.\n\n" + "hello. Each step is shown as it happens; the first that fails stops it.\n\n" +
"A vision model's projector (--projector) is fetched beside it and loaded with\n" +
"it. --format safetensors serves the original weights instead, with their\n" +
"config.json and tokenizer, on vLLM across --tp GPUs of --gpu-class.\n\n" +
" llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" + " llmgw ctl model serve bartowski/Qwen2.5-3B-Instruct-GGUF\n" +
" llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" + " llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct-GGUF --quant Q5_K_M --name qwen-7b\n" +
" llmgw ctl model serve owner/big-GGUF --on-demand", " llmgw ctl model serve owner/big-GGUF --on-demand\n" +
" llmgw ctl model serve google/gemma-3-4b-it-GGUF --projector mmproj-model-f16.gguf\n" +
" llmgw ctl model serve Qwen/Qwen2.5-7B-Instruct --format safetensors --gpu-class l40s-48g",
Args: cobra.ExactArgs(1), Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
req.Repo = args[0] req.Repo = args[0]
if onDemand { if onDemand {
req.Replicas = 0 req.Replicas = 0
} }
// One kind of GPU in the fleet: that one, unless told.
if req.Format == "safetensors" && req.GPUClass == "" {
var sum adminapi.Summary
if err := client().get(cmd.Context(), "/api/v1/summary", &sum); err == nil && len(sum.GPUs) == 1 {
req.GPUClass = sum.GPUs[0].Class
}
}
body, _ := json.Marshal(req) body, _ := json.Marshal(req)
var job modelserve.Job var job modelserve.Job
if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil { if err := client().post(cmd.Context(), "/api/v1/models/serve", body, &job); err != nil {
@@ -44,6 +57,10 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
f := c.Flags() f := c.Flags()
f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)") f.StringVar(&req.Quant, "quant", "", "the quantisation to serve (default Q4_K_M)")
f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant") f.StringVar(&req.File, "file", "", "the file to serve, instead of choosing by --quant")
f.StringVar(&req.Projector, "projector", "", "a vision model's mmproj file, fetched and loaded beside it")
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
f.StringVar(&req.GPUClass, "gpu-class", "", "for safetensors: the GPUs to run on (default: the fleet's one class)")
f.IntVar(&req.TP, "tp", 1, "for safetensors: how many GPUs one replica spans")
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)") f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable") f.StringSliceVar(&req.Aliases, "alias", nil, "other names clients may use; repeatable")
f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models") f.StringVar(&req.Description, "description", "", "what it is for, shown to clients in /v1/models")
+9 -3
View File
@@ -292,7 +292,7 @@ func (m consoleModel) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
case consoleMirrorMsg: case consoleMirrorMsg:
m.mirror = msg m.mirror = msg
case consoleHFMsg: case consoleHFMsg:
if msg.query == m.hf.query && msg.sort == m.hf.sort { if msg.query == m.hf.query && msg.sort == m.hf.sort && msg.st == m.hf.safetensors {
m.hf.models, m.hf.err = msg.models, msg.err m.hf.models, m.hf.err = msg.models, msg.err
if m.hf.models == nil && msg.err == nil { if m.hf.models == nil && msg.err == nil {
m.hf.models = []hf.Model{} m.hf.models = []hf.Model{}
@@ -447,7 +447,7 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
case cQuotas: case cQuotas:
m.menuD = m.quotaMenu(m.quotas[m.row]) m.menuD = m.quotaMenu(m.quotas[m.row])
case cMirror: case cMirror:
m.menuD = m.mirrorMenu(m.mirror.res.Models[m.row]) m.menuD = m.mirrorMenu(m.mirrorGroups()[m.row])
case cServe: case cServe:
return m, m.fetchRepo(m.hf.models[m.row].ID) return m, m.fetchRepo(m.hf.models[m.row].ID)
case cAccounts: case cAccounts:
@@ -464,6 +464,12 @@ func (m consoleModel) key(k tea.KeyMsg) (tea.Model, tea.Cmd) {
if m.menu == cServe { if m.menu == cServe {
m.hfSearchDialog() m.hfSearchDialog()
} }
case "f":
if m.menu == cServe {
m.hf.safetensors = !m.hf.safetensors
m.hf.models, m.hf.err, m.row = nil, nil, 0
return m, m.fetchHF()
}
case "s": case "s":
if m.menu == cServe { if m.menu == cServe {
m.hf.sort = (m.hf.sort + 1) % len(hfSorts) m.hf.sort = (m.hf.sort + 1) % len(hfSorts)
@@ -824,7 +830,7 @@ func (m consoleModel) rowCount() int {
case cQuotas: case cQuotas:
return len(m.quotas) return len(m.quotas)
case cMirror: case cMirror:
return len(m.mirror.res.Models) return len(m.mirrorGroups())
case cServe: case cServe:
return len(m.hf.models) return len(m.hf.models)
case cAccounts: case cAccounts:
+88 -15
View File
@@ -11,6 +11,7 @@ import (
tea "github.com/charmbracelet/bubbletea" tea "github.com/charmbracelet/bubbletea"
"gitea.mynodes.xyz/ilker/llmgw/internal/config" "gitea.mynodes.xyz/ilker/llmgw/internal/config"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf" "gitea.mynodes.xyz/ilker/llmgw/internal/hf"
) )
@@ -24,15 +25,19 @@ var hfSorts = []struct {
}{{"trending", hf.SortTrending}, {"most downloaded", hf.SortDownloads}, {"most liked", hf.SortLikes}, {"newest", hf.SortCreated}} }{{"trending", hf.SortTrending}, {"most downloaded", hf.SortDownloads}, {"most liked", hf.SortLikes}, {"newest", hf.SortCreated}}
type hfState struct { type hfState struct {
query string query string
sort int sort int
models []hf.Model // safetensors lists original weights, for vLLM on GPUs, rather than
err error // GGUF files for llama.cpp.
safetensors bool
models []hf.Model
err error
} }
type consoleHFMsg struct { type consoleHFMsg struct {
query string query string
sort int sort int
st bool
models []hf.Model models []hf.Model
err error err error
} }
@@ -56,18 +61,21 @@ func (m consoleModel) registry() *hf.Client {
} }
func (m consoleModel) fetchHF() tea.Cmd { func (m consoleModel) fetchHF() tea.Cmd {
c, q, sort := m.registry(), m.hf.query, m.hf.sort c, q, sort, st := m.registry(), m.hf.query, m.hf.sort, m.hf.safetensors
return func() tea.Msg { return func() tea.Msg {
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second) ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel() defer cancel()
hq := hf.Query{Search: q, Sort: hfSorts[sort].sort, GGUF: true, Limit: 50} hq := hf.Query{Search: q, Sort: hfSorts[sort].sort, GGUF: !st, Limit: 50}
if st {
hq.Filters = []string{"safetensors"}
}
// Browsing: models that chat. A search finds the rest too, an // Browsing: models that chat. A search finds the rest too, an
// embedding model included. // embedding model included.
if q == "" { if q == "" {
hq.Task = "text-generation" hq.Task = "text-generation"
} }
got, err := c.Explore(ctx, hq) got, err := c.Explore(ctx, hq)
return consoleHFMsg{query: q, sort: sort, models: got, err: err} return consoleHFMsg{query: q, sort: sort, st: st, models: got, err: err}
} }
} }
@@ -85,6 +93,17 @@ func (m *consoleModel) fetchRepo(id string) tea.Cmd {
// repoMenu offers a repository's GGUF files, smallest first as listed. // repoMenu offers a repository's GGUF files, smallest first as listed.
func (m consoleModel) repoMenu(r hf.Repo) *menuDialog { func (m consoleModel) repoMenu(r hf.Repo) *menuDialog {
var items []menuItem var items []menuItem
if m.hf.safetensors {
title := r.ID
set, err := modelserve.SafetensorsSet(r)
if err != nil {
return &menuDialog{title: title, items: []menuItem{{err.Error(), func(*consoleModel) tea.Cmd { return nil }}}}
}
return &menuDialog{title: title, items: []menuItem{{
fmt.Sprintf("Original weights for vLLM: %s in %d files", bytesShort(set.Bytes), len(set.Files)),
func(m *consoleModel) tea.Cmd { m.serveVLLMForm(r, set); return nil },
}}}
}
for _, set := range hf.Group(r.Files) { for _, set := range hf.Group(r.Files) {
if !strings.HasSuffix(strings.ToLower(set.Files[0].Path), ".gguf") { if !strings.HasSuffix(strings.ToLower(set.Files[0].Path), ".gguf") {
continue continue
@@ -124,13 +143,19 @@ func (m *consoleModel) serveForm(r hf.Repo, set hf.FileSet) {
note += ", licence " + r.License note += ", licence " + r.License
} }
note += "\nThe mirror fetches it and checks it, it joins the catalog, and a\nreplica is started and asked to say hello. The output is shown as it goes." note += "\nThe mirror fetches it and checks it, it joins the catalog, and a\nreplica is started and asked to say hello. The output is shown as it goes."
fields := []formField{
textF("name", hf.ModelName(r.ID), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
textF("context length", "4096", "what one request may use"),
textF("description", "", "optional: shown to clients in /v1/models"),
}
// A vision model: its projector, loaded beside it, is what reads images.
if proj := modelserve.Projectors(r); len(proj) > 0 {
fields = append(fields, choiceF("images", append(proj, "none: text only"), proj[0]))
fields[len(fields)-1].hint = "the vision projector, fetched and loaded beside the model"
}
m.openForm(&formDialog{title: "Serve " + r.ID, note: note, m.openForm(&formDialog{title: "Serve " + r.ID, note: note,
fields: []formField{ fields: fields,
textF("name", hf.ModelName(r.ID), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
textF("context length", "4096", "what one request may use"),
textF("description", "", "optional: shown to clients in /v1/models"),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) { submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
n, err := strconv.Atoi(v[1]) n, err := strconv.Atoi(v[1])
if err != nil || n < 0 { if err != nil || n < 0 {
@@ -147,6 +172,50 @@ func (m *consoleModel) serveForm(r hf.Repo, set hf.FileSet) {
if v[3] != "" { if v[3] != "" {
args = append(args, "--description="+v[3]) args = append(args, "--description="+v[3])
} }
if len(v) > 4 && !strings.HasPrefix(v[4], "none") {
args = append(args, "--projector="+v[4])
}
return m.runAfter(args...), nil
}})
}
func (m *consoleModel) serveVLLMForm(r hf.Repo, set hf.FileSet) {
server := "--server=" + m.client.base
var classes []string
for _, g := range m.data.sum.GPUs {
classes = append(classes, g.Class)
}
if len(classes) == 0 {
m.dialog, m.dialogTitle = "error", "No GPUs"
m.dialogText = "Original weights run on vLLM, on GPUs, and this fleet reports none.\n" +
"F goes back to GGUF models, which run on the CPU.\n\n" + sDim.Render("any key")
return
}
m.openForm(&formDialog{title: "Serve " + r.ID,
note: fmt.Sprintf("%s in %d files: the weights, config.json and the tokenizer.\n"+
"Fetched into the mirror and checked, added, and run by vLLM on the GPUs chosen.", bytesShort(set.Bytes), len(set.Files)),
fields: []formField{
textF("name", hf.ModelName(r.ID), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
textF("context length", "8192", "what one request may use; vLLM's --max-model-len"),
textF("description", "", "optional: shown to clients in /v1/models"),
choiceF("GPU class", classes, classes[0]),
textF("GPUs per replica", "1", "tensor parallel: the cards one replica spans"),
},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
for i, what := range map[int]string{1: "replicas", 2: "the context length", 5: "GPUs per replica"} {
if n, err := strconv.Atoi(v[i]); err != nil || n < 0 || (i == 5 && n < 1) {
return nil, fmt.Errorf("%s is a whole number", what)
}
}
args := []string{"ctl", server, "model", "serve", r.ID, "--format=safetensors", "--gpu-class=" + v[4],
"--tp=" + v[5], "--name=" + v[0], "--replicas=" + v[1], "--context-length=" + v[2]}
if v[1] == "0" {
args = append(args, "--on-demand")
}
if v[3] != "" {
args = append(args, "--description="+v[3])
}
return m.runAfter(args...), nil return m.runAfter(args...), nil
}}) }})
} }
@@ -164,7 +233,7 @@ func (m *consoleModel) hfSearchDialog() {
func (m consoleModel) hfPanel(pw int) (string, string, string) { func (m consoleModel) hfPanel(pw int) (string, string, string) {
keys := "→ or Enter open · ↑↓ choose · Q leave" keys := "→ or Enter open · ↑↓ choose · Q leave"
if m.inPanel { if m.inPanel {
keys = "↑↓ row · Enter choose a file · / search · S order · ← back · Q leave" keys = "↑↓ row · Enter choose · / search · S order · F GGUF or original weights · ← back · Q leave"
} }
what := hfSorts[m.hf.sort].label what := hfSorts[m.hf.sort].label
if m.hf.query != "" { if m.hf.query != "" {
@@ -173,7 +242,11 @@ func (m consoleModel) hfPanel(pw int) (string, string, string) {
if m.hf.query == "" { if m.hf.query == "" {
what = "text generation, " + what what = "text generation, " + what
} }
head := "\n GGUF models on Hugging Face: " + what + "\n\n" kind := "GGUF models, for llama.cpp"
if m.hf.safetensors {
kind = "Original weights, for vLLM on GPUs"
}
head := "\n " + kind + " on Hugging Face: " + what + "\n\n"
switch { switch {
case m.hf.err != nil: case m.hf.err != nil:
return "Serve from Hugging Face", head + " " + sBad.Render(m.hf.err.Error()) + return "Serve from Hugging Face", head + " " + sBad.Render(m.hf.err.Error()) +
+82 -25
View File
@@ -62,26 +62,75 @@ func (m *consoleModel) mirrorCall(title string, do func(context.Context, *mirror
} }
} }
// usedBy names the catalog's models whose source is this file. // mirrorGroup is one version of one model on the mirror: a file, or the set
func (m consoleModel) usedBy(base string, s mirror.Status) []string { // of files a model of several is fetched as.
u := mirror.Entry{Model: s.Model, Version: s.Version, File: s.File}.URL(base) type mirrorGroup struct {
dir := strings.TrimSuffix(mirror.Entry{Model: s.Model, Version: s.Version}.URL(base), "/") + "/" model, version string
files []mirror.Status
}
func (g mirrorGroup) bytes() (done, total int64) {
for _, f := range g.files {
done += f.BytesDone
total += max(f.BytesTotal, f.BytesDone)
}
return done, total
}
// source is what a model's definition names for it: the file, or the
// directory of a set.
func (g mirrorGroup) source(base string) string {
if len(g.files) == 1 {
return mirror.Entry{Model: g.model, Version: g.version, File: g.files[0].File}.URL(base)
}
return strings.TrimSuffix(mirror.Entry{Model: g.model, Version: g.version}.URL(base), "/") + "/"
}
func (m consoleModel) mirrorGroups() []mirrorGroup {
var out []mirrorGroup
at := map[string]int{}
for _, st := range m.mirror.res.Models {
k := st.Model + "/" + st.Version
i, ok := at[k]
if !ok {
i = len(out)
at[k] = i
out = append(out, mirrorGroup{model: st.Model, version: st.Version})
}
out[i].files = append(out[i].files, st)
}
return out
}
// usedBy names the catalog's models that run from a version's files.
func (m consoleModel) usedBy(base string, g mirrorGroup) []string {
dir := strings.TrimSuffix(mirror.Entry{Model: g.model, Version: g.version}.URL(base), "/") + "/"
var names []string var names []string
for _, md := range m.data.models { for _, md := range m.data.models {
if md.Source == u || strings.HasPrefix(md.Source, dir) { if strings.HasPrefix(md.Source, dir) {
names = append(names, md.Name) names = append(names, md.Name)
} }
} }
return names return names
} }
func (m consoleModel) mirrorMenu(s mirror.Status) *menuDialog { func (m consoleModel) mirrorMenu(g mirrorGroup) *menuDialog {
what := s.Model + "/" + s.Version what := g.model + "/" + g.version
used := m.usedBy(m.mirror.res.Base, s) used := m.usedBy(m.mirror.res.Base, g)
items := []menuItem{{"Where nodes fetch it", func(m *consoleModel) tea.Cmd { items := []menuItem{{"Its files, and where nodes fetch them", func(m *consoleModel) tea.Cmd {
var b strings.Builder
for _, f := range g.files {
sum := ""
if f.SHA256 != "" {
sum = sDim.Render(" sha256 " + f.SHA256[:min(12, len(f.SHA256))])
}
fmt.Fprintf(&b, "%-50s %9s%s\n", truncate(f.File, 50), bytesShort(max(f.BytesTotal, f.BytesDone)), sum)
if f.Source != "" {
b.WriteString(sDim.Render(" from "+truncate(f.Source, 90)) + "\n")
}
}
m.dialog, m.dialogTitle = "error", what m.dialog, m.dialogTitle = "error", what
m.dialogText = mirror.Entry{Model: s.Model, Version: s.Version, File: s.File}.URL(m.mirror.res.Base) + m.dialogText = b.String() + "\nA model's source: " + g.source(m.mirror.res.Base) + "\n\n" + sDim.Render("any key")
"\n\n" + sDim.Render("the source to give a model definition · any key")
return nil return nil
}}} }}}
items = append(items, menuItem{"Remove from the mirror…", func(m *consoleModel) tea.Cmd { items = append(items, menuItem{"Remove from the mirror…", func(m *consoleModel) tea.Cmd {
@@ -93,7 +142,7 @@ func (m consoleModel) mirrorMenu(s mirror.Status) *menuDialog {
m.confirmD = &confirmDialog{title: "Remove " + what, text: text, m.confirmD = &confirmDialog{title: "Remove " + what, text: text,
yes: func(m *consoleModel) tea.Cmd { yes: func(m *consoleModel) tea.Cmd {
return m.mirrorCall("Remove "+what, func(ctx context.Context, c *mirror.Client) (string, error) { return m.mirrorCall("Remove "+what, func(ctx context.Context, c *mirror.Client) (string, error) {
removed, err := c.Remove(ctx, s.Model, s.Version) removed, err := c.Remove(ctx, g.model, g.version)
return fmt.Sprintf("Removed %d file(s).", len(removed)), err return fmt.Sprintf("Removed %d file(s).", len(removed)), err
}) })
}} }}
@@ -139,24 +188,32 @@ func (m consoleModel) mirrorPanel(pw int) (string, string, string) {
} }
head := "\n " + sDim.Render("Nodes fetch from "+m.mirror.res.Base) + "\n\n" head := "\n " + sDim.Render("Nodes fetch from "+m.mirror.res.Base) + "\n\n"
var rows [][]string var rows [][]string
for _, s := range m.mirror.res.Models { for _, g := range m.mirrorGroups() {
state := string(s.State) done, total := g.bytes()
switch s.State { state := "present"
case mirror.Fetching: for _, f := range g.files {
if p, ok := s.Percent(); ok { switch f.State {
state = sWarn.Render(fmt.Sprintf("fetching %d%%", p)) case mirror.Fetching:
} else { state = sWarn.Render("fetching")
state = sWarn.Render("fetching " + bytesShort(s.BytesDone)) if total > 0 {
state = sWarn.Render(fmt.Sprintf("fetching %d%%", done*100/total))
}
case mirror.Failed:
state = sBad.Render("failed")
}
if f.State != mirror.Present {
break
} }
case mirror.Failed:
state = sBad.Render("failed")
} }
size := bytesShort(s.BytesDone) what := g.files[0].File
used := strings.Join(m.usedBy(m.mirror.res.Base, s), ", ") if len(g.files) > 1 {
what = fmt.Sprintf("%d files", len(g.files))
}
used := strings.Join(m.usedBy(m.mirror.res.Base, g), ", ")
if used == "" { if used == "" {
used = sDim.Render("unused") used = sDim.Render("unused")
} }
rows = append(rows, []string{s.Model, s.Version, s.File, state, size, used}) rows = append(rows, []string{g.model, g.version, what, state, bytesShort(total), used})
} }
if len(rows) == 0 { if len(rows) == 0 {
return "Mirror", head + " " + sDim.Render("Empty. A downloads one."), keys return "Mirror", head + " " + sDim.Render("Empty. A downloads one."), keys
+136 -9
View File
@@ -87,6 +87,15 @@ type Request struct {
Quant string `json:"quant,omitempty"` Quant string `json:"quant,omitempty"`
// File picks it by name instead. // File picks it by name instead.
File string `json:"file,omitempty"` File string `json:"file,omitempty"`
// Projector is a vision model's mmproj file in the repository, fetched
// beside the weights and loaded with them; empty serves text alone.
Projector string `json:"projector,omitempty"`
// Format is "gguf", run by llama.cpp, or "safetensors": the original
// weights with their config and tokenizer, run by vLLM on GPUs of
// GPUClass, across TP of them.
Format string `json:"format,omitempty"`
GPUClass string `json:"gpu_class,omitempty"`
TP int `json:"tp,omitempty"`
// Name is the catalog name; the repository suggests one otherwise. // Name is the catalog name; the repository suggests one otherwise.
Name string `json:"name,omitempty"` Name string `json:"name,omitempty"`
Aliases []string `json:"aliases,omitempty"` Aliases []string `json:"aliases,omitempty"`
@@ -179,6 +188,20 @@ func (s *Server) Start(req Request) (Job, error) {
if req.Replicas < 0 { if req.Replicas < 0 {
return Job{}, errors.New("replicas cannot be negative") return Job{}, errors.New("replicas cannot be negative")
} }
switch req.Format {
case "", "gguf":
req.Format = "gguf"
case "safetensors":
if req.GPUClass == "" {
return Job{}, errors.New("safetensors weights run on vLLM, on GPUs: name the GPU class, as `llmgw ctl status` lists it")
}
if req.Projector != "" {
return Job{}, errors.New("a projector is for a GGUF model; vLLM reads a vision model's own files")
}
req.TP = max(req.TP, 1)
default:
return Job{}, fmt.Errorf("the format is gguf or safetensors, not %q", req.Format)
}
if req.Quant == "" && req.File == "" { if req.Quant == "" && req.File == "" {
req.Quant = "Q4_K_M" req.Quant = "Q4_K_M"
} }
@@ -251,11 +274,28 @@ func (s *Server) run(j *Job) {
s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo)) s.fail(j, "registry", fmt.Errorf("%s is gated: it needs an account's approval the mirror does not have", req.Repo))
return return
} }
set, err := pick(repo, req) pickSet := pick
if req.Format == "safetensors" {
pickSet = pickSafetensors
}
set, err := pickSet(repo, req)
if err != nil { if err != nil {
s.fail(j, "registry", err) s.fail(j, "registry", err)
return return
} }
if req.Projector != "" {
i := slices.IndexFunc(repo.Files, func(f hf.File) bool {
return (f.Path == req.Projector || mirror.LastPathElement(f.Path) == req.Projector) &&
strings.Contains(strings.ToLower(f.Path), "mmproj")
})
if i < 0 {
s.fail(j, "registry", fmt.Errorf("%s has no vision projector %q; it has: %s", repo.ID, req.Projector,
strings.Join(Projectors(repo), ", ")))
return
}
set.Files = append(set.Files, repo.Files[i])
set.Bytes += repo.Files[i].Bytes
}
name := req.Name name := req.Name
if name == "" { if name == "" {
name = hf.ModelName(repo.ID) name = hf.ModelName(repo.ID)
@@ -274,10 +314,20 @@ func (s *Server) run(j *Job) {
} }
// 3. Whether the nodes' llama.cpp loads it, and how much memory it needs. // 3. Whether the nodes' llama.cpp loads it, and how much memory it needs.
if req.Format == "safetensors" {
// vLLM reads the model's shape from its config.json, which came
// with the weights; the GPUs it needs are the class and width asked.
s.step(j, "check", Done, fmt.Sprintf("%.1f GB of weights for vLLM on %d × %s", gb(set.Bytes), req.TP, req.GPUClass))
s.addAndStart(ctx, j, repo, set, name, source, catalog.Profile{
Name: "gpu", Engine: "vllm", GPUClass: req.GPUClass, TP: req.TP, Weight: 1, Priority: 100,
Args: []string{"--max-model-len", strconv.Itoa(req.ContextLength)},
})
return
}
s.step(j, "check", Running, "reading its header from the mirror") s.step(j, "check", Running, "reading its header from the mirror")
var header gguf.Header var header gguf.Header
if !strings.HasSuffix(source, "/") { if u := headerURL(source, set); u != "" {
v := gguf.Check(ctx, source, s.deps.LlamaVersion) v := gguf.Check(ctx, u, s.deps.LlamaVersion)
if !v.Servable { if !v.Servable {
s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason)) s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run this: %s", s.deps.LlamaVersion, v.Reason))
return return
@@ -302,16 +352,24 @@ func (s *Server) run(j *Job) {
} }
} }
s.addAndStart(ctx, j, repo, set, name, source, catalog.Profile{
Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100,
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores,
})
}
// addAndStart puts the model in the catalog with its profile, waits for a
// replica and asks it to answer.
func (s *Server) addAndStart(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSet, name, source string, prof catalog.Profile) {
req := j.Request
// 4. The catalog. // 4. The catalog.
m := catalog.Model{ m := catalog.Model{
Name: name, Aliases: req.Aliases, Description: req.Description, Source: source, Name: name, Aliases: req.Aliases, Description: req.Description, Source: source,
RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024, RoutingProfile: "chat", ContextLength: req.ContextLength, MaxOutput: 1024,
Profiles: []catalog.Profile{{ Profiles: []catalog.Profile{prof},
Name: "cpu", Engine: "llamacpp", Weight: 1, Priority: 100, Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
Args: []string{"--ctx-size", strconv.Itoa(req.CtxPool)}, MinMemoryGB: memGB, CPUCores: cores, Capabilities: catalog.Capabilities{Vision: req.Projector != ""},
}}, Origin: &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
Scaling: catalog.Scaling{Min: req.Replicas, Max: max(req.Replicas, 1)},
Origin: &catalog.Origin{Repo: repo.ID, Revision: repo.Revision, Files: originFiles(set)},
} }
s.step(j, "catalog", Running, "adding "+name) s.step(j, "catalog", Running, "adding "+name)
if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil { if err := s.deps.Catalog.Add(ctx, m, req.By); err != nil {
@@ -593,3 +651,72 @@ func orUnknown(arch string) string {
} }
return arch return arch
} }
// Projectors are a repository's vision projectors: the mmproj files a vision
// model loads beside its weights.
func Projectors(repo hf.Repo) []string {
var out []string
for _, f := range repo.Files {
if strings.HasSuffix(strings.ToLower(f.Path), ".gguf") && strings.Contains(strings.ToLower(f.Path), "mmproj") {
out = append(out, f.Path)
}
}
return out
}
// headerURL is the file whose header says what the model is: the source
// itself, or in a directory the first part of the weights, not a projector.
func headerURL(source string, set hf.FileSet) string {
if !strings.HasSuffix(source, "/") {
return source
}
var names []string
for _, f := range set.Files {
name := mirror.LastPathElement(f.Path)
lower := strings.ToLower(name)
if strings.HasSuffix(lower, ".gguf") && !strings.Contains(lower, "mmproj") {
names = append(names, name)
}
}
if len(names) == 0 {
return ""
}
slices.Sort(names)
return source + names[0]
}
// SafetensorsSet is what serving a repository's original weights fetches.
func SafetensorsSet(repo hf.Repo) (hf.FileSet, error) { return pickSafetensors(repo, Request{}) }
// pickSafetensors chooses a repository's original weights and the files an
// engine reads beside them: config.json, the tokenizer, the shard index.
func pickSafetensors(repo hf.Repo, _ Request) (hf.FileSet, error) {
var weights []hf.File
for _, f := range repo.Files {
if !strings.Contains(f.Path, "/") && strings.HasSuffix(f.Path, ".safetensors") {
weights = append(weights, f)
}
}
sets := hf.Group(weights)
if len(sets) == 0 {
return hf.FileSet{}, fmt.Errorf("%s has no safetensors weights at its top", repo.ID)
}
// Sharded "model-00001-of-00004" or a single "model.safetensors"; a
// repository with a second, consolidated copy keeps it under another name.
set := sets[0]
for _, st := range sets {
if strings.HasPrefix(st.Name, "model") {
set = st
break
}
}
companions := hf.Companions(repo.Files)
if !slices.ContainsFunc(companions, func(f hf.File) bool { return f.Path == "config.json" }) {
return hf.FileSet{}, fmt.Errorf("%s has no config.json, which vLLM reads the model's shape from", repo.ID)
}
for _, f := range companions {
set.Files = append(set.Files, f)
set.Bytes += f.Bytes
}
return set, nil
}
@@ -6,6 +6,7 @@ import (
"encoding/binary" "encoding/binary"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"slices"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -223,3 +224,68 @@ func TestARepositoryThatIsNotOneIsRefused(t *testing.T) {
} }
} }
} }
func TestAVisionModelIsServedWithItsProjectorBesideIt(t *testing.T) {
files := []hf.File{
{Path: "gemma-3-4b-it-Q4_K_M.gguf", Bytes: 2 << 30, SHA256: "aa"},
{Path: "mmproj-gemma-3-4b-it-f16.gguf", Bytes: 800 << 20, SHA256: "bb"},
}
s, m, c, _ := setup(t, files, map[string]string{"gemma-3-4b-it-Q4_K_M.gguf": "aa", "mmproj-gemma-3-4b-it-f16.gguf": "bb"})
job, err := s.Start(Request{Repo: "google/gemma-3-4b-it-GGUF", Quant: "Q4_K_M",
Projector: "mmproj-gemma-3-4b-it-f16.gguf", Replicas: 1})
if err != nil {
t.Fatal(err)
}
if j := wait(t, s, job.ID); j.State != Done {
t.Fatalf("%s %+v", j.Error, j.Steps)
}
if len(m.got) != 2 {
t.Fatalf("fetched %+v", m.got)
}
got := c.added[0]
if !strings.HasSuffix(got.Source, "/") || !got.Capabilities.Vision || len(got.Origin.Files) != 2 {
t.Errorf("%s vision=%v origin=%+v", got.Source, got.Capabilities.Vision, got.Origin)
}
// A projector the repository does not have is named with those it has.
job, _ = s.Start(Request{Repo: "google/gemma-3-4b-it-GGUF", Quant: "Q4_K_M", Projector: "mmproj-x.gguf", Name: "g2"})
if j := wait(t, s, job.ID); j.State != Failed || !strings.Contains(j.Error, "mmproj-gemma-3-4b-it-f16.gguf") {
t.Errorf("%+v", j)
}
}
func TestOriginalWeightsAreServedOnVLLMWithTheirConfigAndTokenizer(t *testing.T) {
files := []hf.File{
{Path: "model-00001-of-00002.safetensors", Bytes: 4 << 30, SHA256: "a1"},
{Path: "model-00002-of-00002.safetensors", Bytes: 3 << 30, SHA256: "a2"},
{Path: "model.safetensors.index.json", Bytes: 30000},
{Path: "config.json", Bytes: 700},
{Path: "tokenizer.json", Bytes: 7 << 20},
{Path: "tokenizer_config.json", Bytes: 7000},
{Path: "README.md", Bytes: 9000},
{Path: "original/consolidated.safetensors", Bytes: 7 << 30},
}
s, m, c, _ := setup(t, files, map[string]string{"model-00001-of-00002.safetensors": "a1", "model-00002-of-00002.safetensors": "a2"})
if _, err := s.Start(Request{Repo: "Qwen/Qwen2.5-7B-Instruct", Format: "safetensors"}); err == nil {
t.Error("safetensors with no GPU class was taken")
}
job, err := s.Start(Request{Repo: "Qwen/Qwen2.5-7B-Instruct", Format: "safetensors", GPUClass: "l40s-48g", Replicas: 1})
if err != nil {
t.Fatal(err)
}
if j := wait(t, s, job.ID); j.State != Done {
t.Fatalf("%s %+v", j.Error, j.Steps)
}
var got []string
for _, r := range m.got {
got = append(got, r.File)
}
if len(got) != 6 || slices.Contains(got, "README.md") || slices.Contains(got, "consolidated.safetensors") {
t.Errorf("fetched %v", got)
}
mod := c.added[0]
p := mod.Profiles[0]
if !strings.HasSuffix(mod.Source, "/") || p.Engine != "vllm" || p.GPUClass != "l40s-48g" || p.TP != 1 ||
!slices.Contains(p.Args, "--max-model-len") {
t.Errorf("%s %+v", mod.Source, p)
}
}
+3 -3
View File
@@ -509,9 +509,9 @@ func (s *Server) runUpdate(j *Job) {
// 3. Whether the nodes load it, and whether it still fits what is // 3. Whether the nodes load it, and whether it still fits what is
// reserved for it. // reserved for it.
s.step(j, "check", Running, "reading its header from the mirror") s.step(j, "check", Running, "reading its header from the mirror")
detail := "a set of shards; not read" detail := "no header to read"
if !strings.HasSuffix(source, "/") { if u := headerURL(source, set); u != "" {
v := gguf.Check(ctx, source, s.deps.LlamaVersion) v := gguf.Check(ctx, u, s.deps.LlamaVersion)
if !v.Servable { if !v.Servable {
s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run the new weights: %s", s.deps.LlamaVersion, v.Reason)) s.fail(j, "check", fmt.Errorf("llama.cpp %s on the nodes cannot run the new weights: %s", s.deps.LlamaVersion, v.Reason))
return return
+11 -4
View File
@@ -68,6 +68,10 @@ type Model struct {
// GGUF is what the registry read from the repository's weights, absent // GGUF is what the registry read from the repository's weights, absent
// for one that has none. // for one that has none.
GGUF *GGUFInfo `json:"gguf"` GGUF *GGUFInfo `json:"gguf"`
// Safetensors is the registry's summary of original weights.
Safetensors *struct {
Total int64 `json:"total"`
} `json:"safetensors,omitempty"`
// Siblings are the repository's files, when asked for with WithFiles. // Siblings are the repository's files, when asked for with WithFiles.
Siblings []struct { Siblings []struct {
RFilename string `json:"rfilename"` RFilename string `json:"rfilename"`
@@ -84,10 +88,13 @@ type GGUFInfo struct {
// Params is the parameter count, 0 when the registry does not know it. // Params is the parameter count, 0 when the registry does not know it.
func (m Model) Params() int64 { func (m Model) Params() int64 {
if m.GGUF == nil { switch {
return 0 case m.GGUF != nil:
return m.GGUF.Total
case m.Safetensors != nil:
return m.Safetensors.Total
} }
return m.GGUF.Total return 0
} }
// Architecture is the GGUF architecture, empty when unknown. // Architecture is the GGUF architecture, empty when unknown.
@@ -172,7 +179,7 @@ type Query struct {
// exploreFields is everything a listing shows. Asking for any field with // exploreFields is everything a listing shows. Asking for any field with
// expand returns only the fields asked for, so all of them are named. // expand returns only the fields asked for, so all of them are named.
var exploreFields = []string{"downloads", "likes", "pipeline_tag", "tags", "lastModified", var exploreFields = []string{"downloads", "likes", "pipeline_tag", "tags", "lastModified",
"createdAt", "gated", "private", "trendingScore", "gguf"} "createdAt", "gated", "private", "trendingScore", "gguf", "safetensors"}
// Explore lists repositories: the trending ones, the newest, or a search. // Explore lists repositories: the trending ones, the newest, or a search.
// //