Refuse weights the nodes' llama.cpp cannot load, before fetching them

kev-0.5b declared architecture "ggmlc", which llama.cpp does not know.
It was fetched, catalogued and placed, and failed on every node — while
the answer was in the first kilobytes of the file.

internal/gguf reads a GGUF header (v1-v3) up to general.architecture,
over HTTP with a Range request, and judges it against the architectures
llama.cpp b10991 lists in LLM_ARCH_NAMES (152, copied from the source at
that tag); a vision projector is not a model on its own. An unknown
build is not refused, only not vouched for, and a header that cannot be
fetched does not refuse anything.

Checked in three places: the registry listing (from the architecture
Hugging Face reports with expand[]=gguf, so one request per repository)
shows it and withholds the fetch button; a fetch reads the header and
refuses unless "fetch anyway"; and the model service refuses to add
such a model, which covers the serve form and ctl model add.

Tried on real files: the mirror's qwen2.5-1.5b (qwen2) and bge-m3
(bert), and Qwen2.5-0.5B on Hugging Face through its redirect.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-09-28 20:44:36 +02:00
1 parent c609805e4b
commit f13b18149a
12 files changed
+782 -3

No files matched your search

+27
View File
@@ -5,6 +5,8 @@ import (
"crypto/tls"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"log/slog"
"net"
"net/http"
@@ -150,6 +152,7 @@ func newServerCmd() *cobra.Command {
// rather than accepting a model that vanishes on restart.
if st := rt.Store(); st != nil {
svc := modelsvc.New(cfg.Models, st, admin, log)
svc.SetCheck(servableCheck(cfg.WantedEngines()["llamacpp"]))
// Rebuild the catalog from configuration plus what was
// stored, and put that in force — otherwise models added
// before the last restart would be silently absent until
@@ -535,3 +538,27 @@ func parseEndpointFlag(spec string) (config.Endpoint, error) {
Weight: 1,
}, nil
}
// servableCheck refuses a model whose llama.cpp weights the nodes' build
// cannot load, read from the weights' own header before anything is placed.
func servableCheck(llamaVersion string) func(context.Context, catalog.Model) error {
return func(ctx context.Context, m catalog.Model) error {
if llamaVersion == "" {
return nil
}
usesLlama := false
for _, p := range m.Profiles {
if p.Engine == "llamacpp" {
usesLlama = true
}
}
url, ok := sourcecheck.HTTPSource(m.Source)
if !usesLlama || !ok || !gguf.IsGGUF(url) {
return nil
}
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
}
return nil
}
}