Compare commits

...
5 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 0c7934ba96 v0.10.6
Release / build (push) Successful in 1m11s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m51s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 08:52:01 +02:00
Ilker ManapandClaude Opus 5.5 e40b9ac202 Load LoRA adapters at scale zero
Release / build (push) Successful in 1m17s
CI / test (push) Successful in 4m0s
Release / notes (push) Successful in 5s
--lora with --lora-init-without-apply leaves an adapter's scale at 1,
and llama-server applies it at that scale to any request without a
"lora" list of its own: every plain request for the model was served
with its adapters on. Found live on qwen2.5-coder-7b-instruct. Each
adapter is now loaded with --lora-scaled FILE:0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:39:12 +02:00
Ilker ManapandClaude Opus 5.5 323d172538 Attach LoRA adapters from the command line
Release / build (push) Successful in 1m14s
Release / notes (push) Successful in 4s
CI / test (push) Successful in 3m52s
llmgw ctl model adapters lists a model's adapters and adds or removes
them through POST /api/v1/models/{name}/adapters, which reads each new
file first and refuses one that is not an adapter for the model's
architecture, as the mirror page's card does.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:30:13 +02:00
Ilker ManapandClaude Opus 5.5 6b1fa65f59 Format kind_live_test.go
Release / build (push) Successful in 1m12s
Release / notes (push) Successful in 3s
CI / test (push) Successful in 4m3s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:20:35 +02:00
Ilker ManapandClaude Opus 5.5 663770226e Find LoRA adapters on Hugging Face and attach them from the mirror page
CI / test (push) Failing after 33s
Release / build (push) Successful in 1m3s
Release / notes (push) Successful in 3s
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-01 07:19:08 +02:00
24 changed files with 848 additions and 37 deletions

No files matched your search

+28 -4
View File
@@ -20,10 +20,17 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.10.6 — 2026-10-01
Changes since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.
Published first as release candidates v0.10.6-rc.1 to rc.8; rc.5 failed its
format check and has no binaries.
Tried on the PoC cluster before release: LoRA adapters asked for by name
through the gateway, two on one model, each answering as trained and the model
itself unchanged; a saturated pool spilling to the next (8 of 12 concurrent
requests on the preferred replica, 4 on the other); and a bronze request
displaced from a full queue by a gold one, with 503 and Retry-After 5. Not tried
live: a model served from an external provider, which needs a provider key.
### Capacity from the admin pages
@@ -46,7 +53,7 @@ What v0.10.5 made settable from the command line is now on the pages too:
A llama.cpp model can carry `adapters`, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (`--lora-init-without-apply`); a request that names an
applied to nothing (`--lora-scaled FILE:0`); a request that names an
adapter goes to the model's replicas with `"lora": [{"id": i, "scale": s}]`,
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in `/v1/models`, and follow key allowlists on
@@ -54,6 +61,23 @@ their own — a key allowed an adapter does not reach the model under it. Adapte
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.
### Finding and attaching adapters from the mirror page
- **Explore** has a *LoRA adapters* view: adapters for any model, or — from a
held model's card, *find adapters on Hugging Face* — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original.
- **Opening a repository** reads the first bytes of each GGUF file and marks it
a *LoRA adapter* or a *model*. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says.
- **Held adapter files** are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
- **`llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]`**
lists and changes them from the command line, through
`POST /api/v1/models/{name}/adapters`, with the same checks.
### Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere: `external: {url,
+6
View File
@@ -694,6 +694,12 @@ follow key allowlists on their own: a key allowed `coder-sql` reaches that
adapter and not the model under it, while a key allowed the model reaches its
adapters too. Adding or removing an adapter is a change to the model; running
replicas load the new set when restarted (`llmgw ctl restart --model M`).
Besides the mirror page's model card, adapters are attached from the command
line, which reads the file first and refuses one that is not an adapter for
this model's architecture:
llmgw ctl model adapters coder --add coder-sql=http://mirror:8081/sql-lora/v1/sql.gguf@0.8
llmgw ctl model adapters coder --remove coder-sql
The adapter file must be a GGUF LoRA made for this model — llama.cpp's
`convert_lora_to_gguf.py` makes one from a PEFT adapter — and not a model with
+6 -6
View File
@@ -146,12 +146,12 @@ func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
}
// Loaded at start, applied per request: a request that names an adapter
// carries "lora": [{"id": i, "scale": s}], and one that names none is the
// base model, since every adapter starts at scale zero.
if len(p.Adapters) > 0 {
args = append(args, "--lora-init-without-apply")
for _, a := range p.Adapters {
args = append(args, "--lora", a.Path)
}
// base model. Each is loaded at scale zero, not with --lora and
// --lora-init-without-apply: that leaves an adapter's scale at 1, and
// llama-server applies it at that scale to every request with no "lora"
// of its own — every request for the model itself.
for _, a := range p.Adapters {
args = append(args, "--lora-scaled", a.Path+":0")
}
args = append(args, p.Args...)
+2 -2
View File
@@ -315,8 +315,8 @@ func TestAdaptersAreLoadedAtStartAndAppliedToNoneByDefault(t *testing.T) {
t.Fatal(err)
}
args := strings.Join(cmds[0].Args, " ")
if !strings.Contains(args, "--lora-init-without-apply --lora /w/sql.gguf --lora /w/docs.gguf") {
t.Errorf("args %s: both adapters, in order, loaded unapplied", args)
if !strings.Contains(args, "--lora-scaled /w/sql.gguf:0 --lora-scaled /w/docs.gguf:0") {
t.Errorf("args %s: both adapters, in order, loaded at scale zero", args)
}
plain, _ := (&LlamaCpp{}).Commands(llamaSpec(nil))
if strings.Contains(strings.Join(plain[0].Args, " "), "--lora") {
+70 -1
View File
@@ -5,6 +5,9 @@ import (
"fmt"
"os"
"os/user"
"slices"
"strconv"
"strings"
"time"
"github.com/knadh/koanf/parsers/yaml"
@@ -31,7 +34,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
}
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
newTargetCmd(client, asJSON),
newTargetCmd(client, asJSON), newAdaptersCmd(client),
newModelServeCmd(client))
return c
}
@@ -264,3 +267,69 @@ func newClassCmd(client func() *ctlClient) *cobra.Command {
},
}
}
func newAdaptersCmd(client func() *ctlClient) *cobra.Command {
var add, remove []string
c := &cobra.Command{
Use: "adapters <model> [--add NAME=URL[@SCALE]] [--remove NAME]",
Short: "List, add or remove a model's LoRA adapters",
Long: "An adapter is asked for by its NAME as though it were a model, and is served\n" +
"by this model's replicas with it applied. URL is the adapter file, usually in\n" +
"the mirror; it must be a GGUF LoRA adapter for this model's architecture,\n" +
"which is checked before it is attached. Replicas load a changed set when\n" +
"restarted: llmgw ctl restart --model MODEL.",
Args: cobra.ExactArgs(1),
RunE: func(cmd *cobra.Command, args []string) error {
name := args[0]
var models []adminapi.ModelView
if err := client().get(cmd.Context(), "/api/v1/models", &models); err != nil {
return err
}
var current []catalog.Adapter
found := false
for _, m := range models {
if m.Name == name {
current, found = m.Adapters, true
}
}
if !found {
return fmt.Errorf("no model %q", name)
}
out := cmd.OutOrStdout()
if len(add) == 0 && len(remove) == 0 {
if len(current) == 0 {
fmt.Fprintf(out, "%s has no adapters\n", name)
}
for _, a := range current {
fmt.Fprintf(out, "%s\tscale %g\t%s\n", a.Name, a.EffectiveScale(), a.Source)
}
return nil
}
next := slices.DeleteFunc(slices.Clone(current), func(a catalog.Adapter) bool { return slices.Contains(remove, a.Name) })
for _, spec := range add {
n, rest, ok := strings.Cut(spec, "=")
if !ok || n == "" || rest == "" {
return fmt.Errorf("--add %q is not NAME=URL[@SCALE]", spec)
}
ad := catalog.Adapter{Name: n, Source: rest}
if i := strings.LastIndex(rest, "@"); i > 0 {
if s, err := strconv.ParseFloat(rest[i+1:], 64); err == nil {
ad.Source, ad.Scale = rest[:i], s
}
}
next = append(next, ad)
}
body, _ := json.Marshal(adminapi.AdaptersRequest{Adapters: next})
var res adminapi.AdaptersRequest
if err := client().post(cmd.Context(), "/api/v1/models/"+name+"/adapters", body, &res); err != nil {
return err
}
fmt.Fprintf(out, "%s serves %d adapter(s); restart its replicas to load them: llmgw ctl restart --model %s\n",
name, len(res.Adapters), name)
return nil
},
}
c.Flags().StringArrayVar(&add, "add", nil, "NAME=URL[@SCALE]; repeatable")
c.Flags().StringArrayVar(&remove, "remove", nil, "an adapter's name; repeatable")
return c
}
@@ -0,0 +1,71 @@
package adminapi
import (
"bytes"
"encoding/binary"
"net/http"
"net/http/httptest"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
)
// ggufStart is the start of a GGUF file with these string keys.
func ggufStart(kv ...string) []byte {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(len(kv) / 2))
for i := 0; i < len(kv); i += 2 {
str(kv[i])
w(uint32(8)) // string
str(kv[i+1])
}
b.Write(make([]byte, 4096))
return b.Bytes()
}
func TestOnlyAnAdapterForTheModelsArchitectureIsAttached(t *testing.T) {
files := map[string][]byte{
"/base.gguf": ggufStart("general.architecture", "qwen2"),
"/model.gguf": ggufStart("general.architecture", "qwen2", "general.type", "model"),
"/llama.gguf": ggufStart("general.architecture", "llama", "general.type", "adapter"),
"/lora.gguf": ggufStart("general.architecture", "qwen2", "general.type", "adapter"),
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusPartialContent)
_, _ = w.Write(files[r.URL.Path])
}))
defer srv.Close()
models := []catalog.Model{{
Name: "coder", Source: srv.URL + "/base.gguf",
Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 8}},
}}
for _, tc := range []struct {
file, want string
code int
}{
{"/model.gguf", "not a LoRA adapter", http.StatusBadRequest},
{"/llama.gguf", "for llama models", http.StatusBadRequest},
{"/lora.gguf", "", http.StatusOK},
} {
a, _ := setup(t, models)
fm := &fakeModels{}
a.SetModels(fm)
mux := http.NewServeMux()
a.Register(mux)
req := httptest.NewRequest(http.MethodPost, "/api/v1/models/coder/adapters",
strings.NewReader(`{"adapters":[{"name":"fix","source":"`+srv.URL+tc.file+`"}]}`))
rec := httptest.NewRecorder()
mux.ServeHTTP(rec, req)
if rec.Code != tc.code || !strings.Contains(rec.Body.String(), tc.want) {
t.Errorf("%s: %d %s; want %d mentioning %q", tc.file, rec.Code, rec.Body.String(), tc.code, tc.want)
}
}
}
+64 -4
View File
@@ -20,6 +20,7 @@ import (
"unicode"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -62,6 +63,7 @@ func (a *API) Register(mux *http.ServeMux) {
mux.HandleFunc("POST /api/v1/models/{name}/scale", a.scaleModel)
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
a.registerQuotas(mux)
mux.HandleFunc("POST /api/v1/models/{name}/adapters", a.setAdapters)
mux.HandleFunc("GET /api/v1/models/{name}/target", a.target)
mux.HandleFunc("POST /api/v1/models/{name}/class", func(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
@@ -318,9 +320,11 @@ type ModelView struct {
Owner string `json:"owner,omitempty"`
// External is where a model served outside the cluster goes, provider
// host and its name for the model; empty for one served here.
External string `json:"external,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
External string `json:"external,omitempty"`
// Adapters are the LoRA adapters it serves.
Adapters []catalog.Adapter `json:"adapters,omitempty"`
Preempted int `json:"preempted,omitempty"`
HeldFor string `json:"held_for,omitempty"`
}
// ReplicaView is a replica as `ctl replicas` shows it.
@@ -574,7 +578,7 @@ func (a *API) Models() []ModelView {
Desired: h.Desired, Failures: h.Failures,
Parked: h.Parked, Reason: h.Reason, Unplaceable: h.Unplaceable,
Class: m.EffectiveClass(), Owner: m.Owner, Preempted: h.Preempted, HeldFor: h.HeldFor,
External: externalLabel(m),
External: externalLabel(m), Adapters: m.Adapters,
}
for _, p := range m.Profiles {
v.Profiles = append(v.Profiles, p.Name)
@@ -657,6 +661,7 @@ type Models interface {
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
}
// ResourcesRequest sets what a model's profiles reserve of a node.
@@ -1461,3 +1466,58 @@ func externalLabel(m catalog.Model) string {
}
return host + " · " + m.External.Model
}
// AdaptersRequest replaces a model's LoRA adapters with these.
type AdaptersRequest struct {
Adapters []catalog.Adapter `json:"adapters"`
}
// setAdapters replaces a model's adapters, after reading each new file to
// check it is an adapter for the model's architecture: one that is not is a
// replica that fails to start, found minutes later on a node.
func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
if a.models == nil {
writeError(w, http.StatusNotImplemented, "this control plane has no store")
return
}
var req AdaptersRequest
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<16)).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "the request is not valid JSON: "+err.Error())
return
}
name := r.PathValue("name")
m, ok := a.ctl.Catalog().Resolve(name)
if !ok {
writeError(w, http.StatusBadRequest, "no model "+name)
return
}
base, baseErr := gguf.KindFromURL(r.Context(), nil, m.Source)
have := map[string]bool{}
for _, ad := range m.Adapters {
have[ad.Source] = true
}
for _, ad := range req.Adapters {
if have[ad.Source] {
continue // already checked when it was attached
}
h, err := gguf.KindFromURL(r.Context(), nil, ad.Source)
switch {
case err != nil:
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
return
case !h.IsAdapter():
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
return
case baseErr == nil && h.Architecture != base.Architecture:
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
ad.Name, h.Architecture, m.Name, base.Architecture))
return
}
}
out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
if err != nil {
writeError(w, http.StatusBadRequest, err.Error())
return
}
writeJSON(w, http.StatusOK, AdaptersRequest{Adapters: out.Adapters})
}
+4
View File
@@ -258,6 +258,10 @@ type fakeModels struct {
err error
}
func (f *fakeModels) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Adapters: as}, f.err
}
func (f *fakeModels) SetOwner(_ context.Context, name, owner, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Owner: owner}, f.err
}
+17
View File
@@ -271,6 +271,23 @@ func (s *Service) SetOwner(ctx context.Context, name, owner, by string) (catalog
return final, err
}
// SetAdapters replaces the LoRA adapters a model serves. Replicas load the
// new set when they next start.
func (s *Service) SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error) {
final, err := s.update(ctx, name, by, "change its adapters", func(m *catalog.Model) error {
m.Adapters = adapters
return nil
})
if err == nil {
names := make([]string, 0, len(adapters))
for _, a := range adapters {
names = append(names, a.Name)
}
s.log.Info("model adapters set", "model", name, "adapters", names, "by", by)
}
return final, err
}
// update changes one model added at runtime and keeps the change.
//
// Built before anything is written, like Add: catalog.New is what applies the
+41
View File
@@ -25,6 +25,10 @@ type Header struct {
Version uint32
Architecture string
Name string
// Type is general.type: "model", or "adapter" for a LoRA adapter, which
// is applied to a model rather than run on its own. Empty in files made
// before the key existed, which are models.
Type string
// The shape the KV cache is sized from, from the architecture's own keys
// (llama.block_count, qwen2.attention.head_count_kv, ...). Zero when the
@@ -235,6 +239,8 @@ func Read(r io.Reader) (Header, error) {
h.Architecture = v
case "general.name":
h.Name = v
case "general.type":
h.Type = v
}
if h.Architecture != "" {
switch strings.TrimPrefix(key, h.Architecture+".") {
@@ -290,6 +296,41 @@ func FromURL(ctx context.Context, client *http.Client, url string) (Header, erro
return Read(io.LimitReader(resp.Body, headerBytes))
}
// IsAdapter reports whether the file is a LoRA adapter rather than a model.
func (h Header) IsAdapter() bool { return h.Type == "adapter" }
// adapterProbe is how much of a file is read to tell an adapter from a model:
// general.type comes before the tokenizer, which is what makes a model's
// header megabytes long.
const adapterProbe = 256 << 10
// KindFromURL reads just enough of a GGUF file to say what it is — its type
// and architecture — over HTTP. A header cut short by the probe is fine: the
// keys asked for come first.
func KindFromURL(ctx context.Context, client *http.Client, url string) (Header, error) {
if client == nil {
client = &http.Client{Timeout: 30 * time.Second}
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return Header{}, err
}
req.Header.Set("Range", fmt.Sprintf("bytes=0-%d", adapterProbe-1))
resp, err := client.Do(req)
if err != nil {
return Header{}, err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusPartialContent {
return Header{}, fmt.Errorf("%s answered %s", url, resp.Status)
}
h, err := Read(io.LimitReader(resp.Body, adapterProbe))
if h.Architecture != "" {
return h, nil // what was asked for was read, whatever came after
}
return h, err
}
// IsGGUF reports whether a source names a GGUF file, or a directory of them.
func IsGGUF(source string) bool {
s := strings.ToLower(strings.TrimSuffix(source, "/"))
+31
View File
@@ -0,0 +1,31 @@
package gguf
import (
"os"
"testing"
)
// Run by hand against the real adapter this was written for:
//
// LLMGW_LIVE_GGUF=1 go test ./internal/gguf -run LiveAdapter -v
func TestLiveAdapterIsAnAdapter(t *testing.T) {
if os.Getenv("LLMGW_LIVE_GGUF") == "" {
t.Skip("set LLMGW_LIVE_GGUF to read a real adapter from Hugging Face")
}
h, err := KindFromURL(t.Context(), nil,
"https://huggingface.co/ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF/resolve/main/LoRA-Qwen2.5-Coder-7B-Instruct-f16.gguf")
if err != nil {
t.Fatal(err)
}
if !h.IsAdapter() || h.Architecture != "qwen2" {
t.Errorf("%+v", h)
}
m, err := KindFromURL(t.Context(), nil,
"https://huggingface.co/bartowski/Qwen2.5-0.5B-Instruct-GGUF/resolve/main/Qwen2.5-0.5B-Instruct-Q4_K_M.gguf")
if err != nil {
t.Fatal(err)
}
if m.IsAdapter() || m.Architecture != "qwen2" {
t.Errorf("a model read as %+v", m)
}
}
+58
View File
@@ -0,0 +1,58 @@
package gguf
import (
"bytes"
"encoding/binary"
"net/http"
"net/http/httptest"
"testing"
)
// adapterHeader is the start of a LoRA adapter as llama.cpp's converter
// writes it, followed by padding that stands in for its tensors.
func adapterHeader(kind string, pad int) []byte {
var b bytes.Buffer
w := func(v any) { _ = binary.Write(&b, binary.LittleEndian, v) }
str := func(s string) { w(uint64(len(s))); b.WriteString(s) }
b.WriteString("GGUF")
w(uint32(3))
w(uint64(0))
w(uint64(3))
str("general.architecture")
w(uint32(tString))
str("qwen2")
str("general.type")
w(uint32(tString))
str(kind)
str("adapter.type")
w(uint32(tString))
str("lora")
b.Write(make([]byte, pad))
return b.Bytes()
}
func TestAnAdapterIsToldFromAModelByItsFirstBytes(t *testing.T) {
var asked string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
asked = r.Header.Get("Range")
w.WriteHeader(http.StatusPartialContent)
if r.URL.Path == "/adapter.gguf" {
_, _ = w.Write(adapterHeader("adapter", 1<<20))
return
}
_, _ = w.Write(header("qwen2"))
}))
defer srv.Close()
h, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/adapter.gguf")
if err != nil || !h.IsAdapter() || h.Architecture != "qwen2" {
t.Errorf("an adapter read as %+v, %v", h, err)
}
if asked != "bytes=0-262143" {
t.Errorf("asked for %q: the probe should read the start only", asked)
}
m, err := KindFromURL(t.Context(), srv.Client(), srv.URL+"/model.gguf")
if err != nil || m.IsAdapter() || m.Architecture != "qwen2" {
t.Errorf("a model read as %+v, %v", m, err)
}
}
+8 -1
View File
@@ -156,6 +156,10 @@ type Query struct {
// applies it, so a page of small models is a full page.
MaxParams int64
Limit int
// Filters are further registry tags every result must carry, such as
// "lora", or "base_model:adapter:Qwen/Qwen2.5-7B-Instruct" for the
// adapters made for one model.
Filters []string
}
// exploreFields is everything a listing shows. Asking for any field with
@@ -188,7 +192,10 @@ func (c *Client) Explore(ctx context.Context, q Query) ([]Model, error) {
v.Set("pipeline_tag", q.Task)
}
if q.GGUF {
v.Set("filter", "gguf")
v.Add("filter", "gguf")
}
for _, f := range q.Filters {
v.Add("filter", f)
}
if q.MaxParams > 0 {
v.Set("num_parameters", fmt.Sprintf("min:0,max:%d", q.MaxParams))
+1 -1
View File
@@ -1013,7 +1013,7 @@ func withModel(body []byte, name string) []byte {
// withAdapter is body asking for model with a LoRA adapter applied, in
// llama.cpp's form: "lora": [{"id": index, "scale": scale}]. Adapters not
// listed are applied at zero, so the list is the whole of it.
// listed stay at zero, the scale the engine loads each one at.
func withAdapter(body []byte, model string, index int, scale float64) []byte {
var m map[string]json.RawMessage
if err := json.Unmarshal(body, &m); err != nil {
+4
View File
@@ -9,6 +9,7 @@ import (
"net/http"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/account"
@@ -76,6 +77,9 @@ type AccountPages struct {
priced Priced
// registry browses a model registry, or is nil where there is no way out.
registry Registry
// kinds remembers what each held GGUF file is, by its address: a held
// version never changes, so it is read once.
kinds sync.Map
// catalog puts a mirrored model into service; fleet says whether anything
// could run it. Both nil where the mirror page is read-only.
catalog CatalogAdmin
+55
View File
@@ -11,6 +11,7 @@ import (
"sort"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
@@ -46,6 +47,10 @@ type Registry interface {
// repoSet is one thing worth fetching: a file, or every shard of one.
type repoSet struct {
// Kind is what the file says it is, read from its first bytes: "adapter"
// for a LoRA adapter, "model", or empty when it was not read. Arch is the
// architecture it names, which an adapter must share with its model.
Kind, Arch string
// Verdict is whether the nodes' llama.cpp can load it, from the
// architecture the registry reports; a set that cannot be served is
// shown with the reason and no fetch button.
@@ -76,6 +81,8 @@ type modelUse struct {
// a replica of its first profile reserves.
Class, EffClass, Owner string
MemoryGB, CPUCores int
// Adapters are the LoRA adapters it serves.
Adapters []catalog.Adapter
// Ready, Wanted and Nodes are how it runs now, for the row itself:
// replicas ready of those it should have, and the nodes they are on.
// Fleet is false where nothing could be asked, so the row says nothing
@@ -105,6 +112,9 @@ type mirrorRow struct {
// Detail is the card opened beneath the row: what runs it, on which
// nodes, and how many replicas it runs.
Detail *heldDetail
// Kind is what a single GGUF file is — "adapter" or "model" — and Arch
// its architecture; empty when not read.
Kind, Arch string
// Open is the row's ?open= value.
Open string
@@ -299,6 +309,7 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
}))
mux.HandleFunc("POST /admin/mirror/replicas", p.mirrorAction(m, p.setModel))
mux.HandleFunc("POST /admin/mirror/adapters", p.mirrorAction(m, p.setAdapters))
mux.HandleFunc("POST /admin/mirror/serve", p.mirrorAction(m,
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
@@ -469,6 +480,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
Name: m.Name, Mode: modeOf(m.Scaling), Replicas: max(m.Scaling.Max, 1),
Min: m.Scaling.Min, Max: m.Scaling.Max,
Class: m.Class, EffClass: m.EffectiveClass(), Owner: m.Owner,
Adapters: m.Adapters,
Declared: p.catalog.Declared(m.Name),
}
if len(m.Profiles) > 0 {
@@ -487,6 +499,8 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
}
}
p.heldKinds(r.Context(), d.Rows)
// The one row whose card is open. From the form too, so a change made in
// the card comes back with the card still open to show it.
open := r.FormValue("open")
@@ -494,6 +508,7 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
d.Rows[i].Open = openKey(d.Rows[i].Model, d.Rows[i].Version)
if open != "" && d.Rows[i].Open == open && d.Rows[i].State == mirror.Present {
d.Rows[i].Detail = p.heldDetailFor(r.Context(), d.Rows[i])
d.Rows[i].Detail.addAdapterChoices(d.Rows[i], d.Rows)
}
}
@@ -755,6 +770,46 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
}
d.RepoSets = append(d.RepoSets, rs)
}
p.probeKinds(r.Context(), d.RepoSets)
}
// kindOf reads what a GGUF file is; a variable so tests need no registry.
var kindOf = func(ctx context.Context, url string) (gguf.Header, error) {
return gguf.KindFromURL(ctx, nil, url)
}
// probeKinds reads the start of each single-file GGUF set, to tell LoRA
// adapters from models: a repository tagged "lora" holds a model with the
// adapter merged in as often as an adapter, and only the file says which.
// A few at once and a few in all, since each is a request to the registry.
func (p *AccountPages) probeKinds(ctx context.Context, sets []repoSet) {
const most, together = 12, 6
sem := make(chan struct{}, together)
var wg sync.WaitGroup
n := 0
for i := range sets {
s := &sets[i]
if len(s.Sources) != 1 || !gguf.IsGGUF(s.Sources[0].Path) || n == most {
continue
}
n++
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
cctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if h, err := kindOf(cctx, s.Sources[0].URL); err == nil {
s.Arch = h.Architecture
s.Kind = "model"
if h.IsAdapter() {
s.Kind = "adapter"
}
}
}()
}
wg.Wait()
}
// defaultModelName is the catalog name a repository suggests.
+1
View File
@@ -27,6 +27,7 @@ type CatalogAdmin interface {
SetClass(ctx context.Context, name, class, by string) (catalog.Model, error)
SetOwner(ctx context.Context, name, owner, by string) (catalog.Model, error)
Resources(ctx context.Context, name, profile string, memoryGB, cpuCores int, by string) (catalog.Model, error)
SetAdapters(ctx context.Context, name string, adapters []catalog.Adapter, by string) (catalog.Model, error)
}
// serveForm is the offer to put one mirrored version into service.
+15 -9
View File
@@ -16,15 +16,16 @@ import (
)
type fakeCatalog struct {
added []catalog.Model
declared string
sources map[string][]catalog.Model
scaled []string
lastScale catalog.Scaling
err error
classes []string
owners []string
resources []string
added []catalog.Model
declared string
sources map[string][]catalog.Model
scaled []string
lastScale catalog.Scaling
err error
classes []string
owners []string
resources []string
adapterSets [][]catalog.Adapter
}
func (f *fakeCatalog) Add(_ context.Context, m catalog.Model, _ string) error {
@@ -39,6 +40,11 @@ func (f *fakeCatalog) Sources(context.Context) (map[string][]catalog.Model, erro
return f.sources, nil
}
func (f *fakeCatalog) SetAdapters(_ context.Context, name string, as []catalog.Adapter, _ string) (catalog.Model, error) {
f.adapterSets = append(f.adapterSets, as)
return catalog.Model{Name: name, Adapters: as}, f.err
}
func (f *fakeCatalog) SetClass(_ context.Context, name, class, _ string) (catalog.Model, error) {
f.classes = append(f.classes, name+"="+class)
return catalog.Model{Name: name, Class: class}, f.err
+154 -5
View File
@@ -4,13 +4,19 @@ import (
"context"
"errors"
"fmt"
"html/template"
"net/http"
"net/url"
"slices"
"strconv"
"strings"
"sync"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -37,9 +43,15 @@ type heldDetail struct {
// heldModel is one catalog model running a held version.
type heldModel struct {
modelUse
View adminapi.ModelView
Replicas []heldReplica
Headroom *adminapi.HeadroomView
// Adapters are the model's own; AdapterChoices the held adapter files
// of its architecture not attached yet; FindAdapters the explorer's
// list of adapters made for it.
Adapters []catalog.Adapter
AdapterChoices []adapterChoice
FindAdapters template.URL
View adminapi.ModelView
Replicas []heldReplica
Headroom *adminapi.HeadroomView
// MostNow is how many replicas could always run, Current plus the room
// found; zero when the room was not measured or had no end in sight.
MostNow int
@@ -74,7 +86,7 @@ func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDe
}
if p.fleet == nil {
for _, u := range row.UsedBy {
d.Models = append(d.Models, heldModel{modelUse: u})
d.Models = append(d.Models, heldModel{modelUse: u, Adapters: u.Adapters})
}
return d
}
@@ -88,7 +100,7 @@ func (p *AccountPages) heldDetailFor(ctx context.Context, row mirrorRow) *heldDe
}
hs, _ := p.fleet.(HeadroomSource)
for _, u := range row.UsedBy {
hm := heldModel{modelUse: u, View: views[u.Name]}
hm := heldModel{modelUse: u, View: views[u.Name], Adapters: u.Adapters}
for _, rep := range p.fleet.Replicas(u.Name) {
// Built field by field: a ReplicaView carries the engine's key,
// which is for the admin listener and never for a page.
@@ -282,3 +294,140 @@ func (p *AccountPages) setModel(ctx context.Context, admin store.User, r *http.R
}
return name + " " + strings.Join(done, "; "), nil
}
// heldKinds reads what each held single-file GGUF is — an adapter or a model —
// once per file, remembered after.
func (p *AccountPages) heldKinds(ctx context.Context, rows []mirrorRow) {
var wg sync.WaitGroup
sem := make(chan struct{}, 6)
for i := range rows {
row := &rows[i]
if row.State != mirror.Present || row.Files != 1 || !gguf.IsGGUF(row.File) {
continue
}
if h, ok := p.kinds.Load(row.URL); ok {
row.Kind, row.Arch = kindName(h.(gguf.Header)), h.(gguf.Header).Architecture
continue
}
wg.Add(1)
go func() {
defer wg.Done()
sem <- struct{}{}
defer func() { <-sem }()
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
if h, err := kindOf(cctx, row.URL); err == nil {
p.kinds.Store(row.URL, h)
row.Kind, row.Arch = kindName(h), h.Architecture
}
}()
}
wg.Wait()
}
func kindName(h gguf.Header) string {
if h.IsAdapter() {
return "adapter"
}
return "model"
}
// adapterChoice is a held adapter file that could be attached to a model.
type adapterChoice struct {
URL, Label, Suggest string
}
// addAdapterChoices offers, on each model of a held model's card, the held
// adapter files of the same architecture, and a way to find more.
func (d *heldDetail) addAdapterChoices(row mirrorRow, rows []mirrorRow) {
for i := range d.Models {
hm := &d.Models[i]
hm.FindAdapters = findAdaptersHref(row.Source)
if row.Kind == "adapter" {
continue
}
for _, r := range rows {
if r.Kind != "adapter" || (row.Arch != "" && r.Arch != row.Arch) {
continue
}
attached := false
for _, a := range hm.Adapters {
attached = attached || a.Source == r.URL
}
if !attached {
hm.AdapterChoices = append(hm.AdapterChoices, adapterChoice{URL: r.URL,
Label: r.Model + " " + r.Version + " · " + r.File, Suggest: hm.Name + "-" + r.Model})
}
}
}
}
// findAdaptersHref is the explorer's list of adapters for a held model: those
// made for the model its repository was made from, when it came from the
// registry, and every adapter otherwise.
func findAdaptersHref(source string) template.URL {
const base = "/admin/mirror?tab=explore&view=adapters"
u, err := url.Parse(source)
if err != nil || u.Host != "huggingface.co" {
return base
}
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
if len(parts) < 2 {
return base
}
return template.URL(base + "&from=" + url.QueryEscape(parts[0]+"/"+parts[1]))
}
// setAdapters attaches or detaches one adapter on a model, from its card.
// An adapter is checked before it is attached: a file that is not an adapter,
// or one made for another architecture, is a replica that fails to start.
func (p *AccountPages) setAdapters(ctx context.Context, admin store.User, r *http.Request) (string, error) {
name := r.FormValue("model")
current, err := p.currentModel(ctx, name)
if err != nil {
return "", err
}
adapters := slices.Clone(current.Adapters)
switch r.FormValue("op") {
case "remove":
gone := r.FormValue("name")
adapters = slices.DeleteFunc(adapters, func(a catalog.Adapter) bool { return a.Name == gone })
if len(adapters) == len(current.Adapters) {
return "", fmt.Errorf("%s has no adapter %q", name, gone)
}
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
return "", err
}
return fmt.Sprintf("%s: adapter %s removed; replicas drop it when restarted (llmgw ctl restart --model %s)",
name, gone, name), nil
case "add":
an := strings.TrimSpace(r.FormValue("name"))
src := strings.TrimSpace(r.FormValue("source"))
scale := 1.0
if v := strings.TrimSpace(r.FormValue("scale")); v != "" {
if scale, err = strconv.ParseFloat(v, 64); err != nil || scale <= 0 {
return "", errors.New("the scale is a positive number")
}
}
if an == "" || src == "" {
return "", errors.New("an adapter needs a name and a file")
}
h, err := kindOf(ctx, src)
if err != nil {
return "", fmt.Errorf("the adapter file could not be read: %w", err)
}
if !h.IsAdapter() {
return "", errors.New("that file is a model, not a LoRA adapter")
}
if base, err := kindOf(ctx, current.Source); err == nil && base.Architecture != h.Architecture {
return "", fmt.Errorf("that adapter is for %s models and %s is %s", h.Architecture, name, base.Architecture)
}
adapters = append(adapters, catalog.Adapter{Name: an, Source: src, Scale: scale})
if _, err := p.catalog.SetAdapters(ctx, name, adapters, admin.Email); err != nil {
return "", err
}
return fmt.Sprintf("%s: adapter %s added; ask for it by that name once the replicas are restarted "+
"(llmgw ctl restart --model %s)", name, an, name), nil
}
return "", errors.New("add or remove an adapter")
}
+64
View File
@@ -1,6 +1,7 @@
package web
import (
"context"
"log/slog"
"net/http"
"net/url"
@@ -9,6 +10,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
)
@@ -269,3 +271,65 @@ func TestRoomAHigherClassWouldMakeIsOfferedAndSaid(t *testing.T) {
t.Errorf("three is past even what preemption makes: %s", body)
}
}
func TestAnAdapterHeldInTheMirrorIsAttachedFromItsModelsCard(t *testing.T) {
prev := kindOf
t.Cleanup(func() { kindOf = prev })
kindOf = func(_ context.Context, u string) (gguf.Header, error) {
switch {
case strings.Contains(u, "/sql/"):
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
case strings.Contains(u, "/llama-lora/"):
return gguf.Header{Architecture: "llama", Type: "adapter"}, nil
}
return gguf.Header{Architecture: "qwen2"}, nil
}
cat := coderCatalog()
coder := &cat.sources["http://mirror:8081/coder/v1/c.gguf"][0]
coder.Source = "http://mirror:8081/coder/v1/c.gguf"
coder.Profiles = []catalog.Profile{{Name: "cpu", Engine: "llamacpp", MinMemoryGB: 12}}
m := &fakeMirror{models: []mirror.Status{
{Model: "coder", Version: "v1", File: "c.gguf", State: mirror.Present, BytesDone: 1,
Source: "https://huggingface.co/bartowski/Qwen2.5-Coder-7B-Instruct-GGUF/resolve/abc/c.gguf"},
{Model: "sql", Version: "v1", File: "sql.gguf", State: mirror.Present, BytesDone: 1},
{Model: "llama-lora", Version: "v1", File: "l.gguf", State: mirror.Present, BytesDone: 1},
}}
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Catalog: cat, Fleet: &roomyFleet{more: 1}}, true)
list := mirrorGet(t, mux, "/admin/mirror").Body.String()
if strings.Count(list, ">LoRA adapter</span>") != 2 {
t.Error("the two adapter files are not marked as adapters in the list")
}
card := mirrorGet(t, mux, "/admin/mirror?open=coder:v1").Body.String()
if !strings.Contains(card, `<option value="http://mirror:8081/sql/v1/sql.gguf" data-suggest="coder-sql">`) {
t.Error("the qwen2 adapter is not offered for the qwen2 model")
}
if strings.Contains(card, "llama-lora") && strings.Contains(card, `value="http://mirror:8081/llama-lora/v1/l.gguf"`) {
t.Error("an adapter for another architecture is offered")
}
if !strings.Contains(card, "view=adapters&amp;from=bartowski%2FQwen2.5-Coder-7B-Instruct-GGUF") {
t.Error("finding adapters should start from the model's own repository")
}
post := func(v url.Values) string {
v.Set("csrf", "tok")
v.Set("model", "coder")
v.Set("open", "coder:v1")
return mirrorPost(t, mux, "/admin/mirror/adapters", v).Body.String()
}
body := post(url.Values{"op": {"add"}, "name": {"coder-sql"}, "source": {"http://mirror:8081/sql/v1/sql.gguf"}, "scale": {"0.8"}})
if len(cat.adapterSets) != 1 || len(cat.adapterSets[0]) != 1 || cat.adapterSets[0][0].Scale != 0.8 {
t.Fatalf("attached %+v: %s", cat.adapterSets, body)
}
if !strings.Contains(body, "adapter coder-sql added") {
t.Errorf("the attach was not said: %s", body)
}
for name, src := range map[string]string{"another architecture": "http://mirror:8081/llama-lora/v1/l.gguf",
"a model": "http://mirror:8081/coder/v1/c.gguf"} {
n := len(cat.adapterSets)
out := post(url.Values{"op": {"add"}, "name": {"x"}, "source": {src}})
if len(cat.adapterSets) != n || !strings.Contains(out, `class="error"`) {
t.Errorf("%s was attached", name)
}
}
}
+41 -2
View File
@@ -18,6 +18,9 @@ const (
viewTrending = "trending"
viewNew = "new"
viewDownloads = "downloads"
// viewAdapters lists LoRA adapters, all of them or those made for one
// model.
viewAdapters = "adapters"
)
// newWithin is how recent a repository has to be to count as new.
@@ -31,6 +34,7 @@ const exploreShown = 25
var exploreViews = []struct{ key, label string }{
{viewTrending, "Trending"}, {viewNew, "New this month"}, {viewDownloads, "Most downloaded"},
{viewAdapters, "LoRA adapters"},
}
var exploreTasks = []struct{ key, label, tag string }{
@@ -48,6 +52,9 @@ var exploreSizes = []struct {
// open: a listing, its filters, and links that keep them.
type explorer struct {
View, Task, Size string
// Of is the model whose adapters are listed, in the adapters view; empty
// lists adapters for any model.
Of string
// Here keeps to what the nodes' llama.cpp can load. On by default: a
// trending list is mostly things this cluster cannot run, and finding
// that out one repository at a time is what the explorer is for avoiding.
@@ -123,11 +130,12 @@ func trimZero(s string) string { return strings.TrimSuffix(s, ".0") }
// — the page opened from the menu — it is trending chat models this cluster
// can run.
func exploreState(q url.Values) explorer {
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size")}
e := explorer{View: q.Get("view"), Task: q.Get("task"), Size: q.Get("size"),
Of: strings.Trim(strings.TrimSpace(q.Get("of")), "/")}
// "x" marks a submitted form, so an unticked box reads as off rather than
// as not yet chosen.
e.Here = q.Get("x") != "1" || q.Get("here") == "1"
if !oneOf(e.View, viewTrending, viewNew, viewDownloads) {
if !oneOf(e.View, viewTrending, viewNew, viewDownloads, viewAdapters) {
e.View = viewTrending
// A search with no view asks for what a name usually means, which is
// the one everybody downloads.
@@ -165,6 +173,10 @@ func (e explorer) href(query, key, value string) template.URL {
if key != "" {
v.Set(key, value)
}
// Whose adapters, kept while it is adapters being looked at.
if e.Of != "" && v.Get("view") == viewAdapters {
v.Set("of", e.Of)
}
return template.URL("/admin/mirror?" + v.Encode())
}
@@ -181,6 +193,16 @@ func (e explorer) hfQuery(search string) hf.Query {
// Trending, then kept to the recent: sorted by creation date alone,
// the newest repositories are mostly empty uploads nobody has used.
q.Limit = 100
case viewAdapters:
// Adapters rarely carry a task or a parameter count, so neither is
// asked for; what makes one useful is the model it was made for.
q.Sort = hf.SortDownloads
if e.Of != "" {
q.Filters = []string{"base_model:adapter:" + e.Of}
} else {
q.Filters = []string{"lora"}
}
return q
}
for _, t := range exploreTasks {
if t.key == e.Task {
@@ -198,6 +220,15 @@ func (e explorer) hfQuery(search string) hf.Query {
// exploreList fills in the listing.
func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
e := exploreState(r.URL.Query())
// From a held model's card: its repository, whose adapters are those of
// the model it was made from — a GGUF repository is a quantisation, and
// adapters are made for the original.
if from := strings.TrimSpace(r.URL.Query().Get("from")); e.View == viewAdapters && e.Of == "" && from != "" {
e.Of = from
if repo, err := p.registry.Repo(r.Context(), from); err == nil && len(repo.BaseModel) > 0 {
e.Of = repo.BaseModel[0]
}
}
for _, v := range exploreViews {
e.Views = append(e.Views, exploreLink{v.label, e.href(d.Query, "view", v.key), v.key == e.View})
}
@@ -225,6 +256,14 @@ func (p *AccountPages) exploreList(r *http.Request, d *adminMirrorData) {
if a := m.Architecture(); a != "" && p.llamaVersion != "" {
row.Verdict = gguf.LlamaCpp(p.llamaVersion, a)
}
// Adapters are small and say little about their architecture in a
// listing; whether one fits is read from its file when it is opened.
if e.View == viewAdapters {
if len(e.Rows) < exploreShown {
e.Rows = append(e.Rows, row)
}
continue
}
// An adapter or a fragment, not a model: tens of megabytes of LoRA
// weights listed among the models they need a model to run on.
if e.Here && m.Params() > 0 && m.Params() < minParams {
+52
View File
@@ -1,15 +1,18 @@
package web
import (
"context"
"errors"
"html"
"net/http"
"net/http/httptest"
"net/url"
"os"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
)
@@ -195,3 +198,52 @@ func TestAFetchFromTheListAnswersWithItsOutcomeAlone(t *testing.T) {
t.Errorf("a refusal: %d %s", w.Code, w.Body.String())
}
}
// noProbe answers every file as an unreadable one, for tests that are not
// about what the files are.
func TestMain(m *testing.M) {
kindOf = func(context.Context, string) (gguf.Header, error) {
return gguf.Header{}, errors.New("not probed in tests")
}
os.Exit(m.Run())
}
func TestAdaptersAreListedForAModelAndToldFromModelsByTheirFiles(t *testing.T) {
prev := kindOf
t.Cleanup(func() { kindOf = prev })
kindOf = func(_ context.Context, url string) (gguf.Header, error) {
if strings.Contains(url, "adapter") {
return gguf.Header{Architecture: "qwen2", Type: "adapter"}, nil
}
return gguf.Header{Architecture: "qwen2"}, nil
}
reg := &fakeRegistry{
results: []hf.Model{{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Downloads: 900}},
repo: hf.Repo{ID: "ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF", Revision: "776530df0000",
BaseModel: []string{"Qwen/Qwen2.5-Coder-7B-Instruct"},
Files: []hf.File{{Path: "lora-adapter-f16.gguf", Bytes: 726 << 20}, {Path: "merged-Q4_K_M.gguf", Bytes: 4 << 30}}},
}
mux := mirrorPageWith(t, &fakeMirror{}, reg, true)
// From a held model's card: its repository is looked up, and the
// adapters listed are those of the model it was made from.
body := mirrorGet(t, mux, "/admin/mirror?tab=explore&view=adapters&from=bartowski/Qwen2.5-Coder-7B-Instruct-GGUF").Body.String()
want := "base_model:adapter:Qwen/Qwen2.5-Coder-7B-Instruct"
if len(reg.query.Filters) != 1 || reg.query.Filters[0] != want {
t.Errorf("asked the registry for %v, expected %s", reg.query.Filters, want)
}
if !strings.Contains(body, "LoRA adapters made for <code>Qwen/Qwen2.5-Coder-7B-Instruct</code>") {
t.Error("the page does not say whose adapters these are")
}
if strings.Contains(body, `aria-label="Task"`) {
t.Error("task and size filters shown for adapters, which rarely carry either")
}
repo := mirrorGet(t, mux, "/admin/mirror?repo=ggml-org/LoRA-Qwen2.5-Coder-7B-Instruct-F16-GGUF").Body.String()
if !strings.Contains(repo, "LoRA adapter · qwen2") {
t.Error("the adapter file is not marked as one")
}
if !strings.Contains(repo, `<span class="opt">model · qwen2</span>`) {
t.Error("the merged file is not marked as a model")
}
}
+4
View File
@@ -858,3 +858,7 @@ table.rows td.warn { color: var(--warn); cursor: help; }
.held-card .replicas-form select, .capacity-form select { display: block; margin-top: .25rem; }
.held-card .replicas-form .field-row { flex-wrap: wrap; gap: .75rem 1rem; }
.quota-tree td.q-act form.inline { margin-left: .75rem; }
.held-card .adapters { margin: .75rem 0; }
.held-card .adapters h4 { font-size: .9rem; margin: 0 0 .35rem; display: flex; gap: .75rem; align-items: baseline; }
.held-card .adapter-form .field-row { align-items: flex-end; flex-wrap: wrap; gap: .5rem 1rem; }
.held-card .adapter-form select { display: block; margin-top: .25rem; max-width: 28rem; }
+51 -2
View File
@@ -47,7 +47,8 @@ reachable from this machine.</p>
{{range .Rows}}{{$row0 := .}}<tr{{if $grouped}} class="model-version"{{end}}{{if .Detail}} class="open"{{end}}>
<td>{{if eq (printf "%s" .State) "present"}}<a class="open-held" aria-expanded="{{if .Detail}}true{{else}}false{{end}}"
href="/admin/mirror?tab=held{{if not .Detail}}&amp;open={{.Open}}{{end}}"><strong>{{.Model}}</strong></a>
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span></td>
{{else}}<strong>{{.Model}}</strong>{{end}} <span class="opt">{{.Version}}</span>
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter for {{.Arch}} models: attach it on a model's card">LoRA adapter</span>{{end}}</td>
<td class="file" title="{{.File}}{{if .SHA256}} · sha256 {{.SHA256}}{{end}}">
{{if gt .Files 1}}{{.Files}} files{{else}}<code>{{.File}}</code>{{end}}</td>
<td class="num">
@@ -215,16 +216,25 @@ document.addEventListener('click', function (e) {
{{if and (eq .Tab "explore") .HasRegistry}}
{{if not .Repo}}{{with .Explore}}
<nav class="pills" aria-label="Which models">{{range .Views}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="page"{{end}}>{{.Label}}</a>{{end}}</nav>
{{if eq .View "adapters"}}
<p class="note">{{if .Of}}LoRA adapters made for <code>{{.Of}}</code> ·
<a href="/admin/mirror?tab=explore&amp;view=adapters">all adapters</a>{{else}}LoRA adapters for any model.
From a held model's card, <em>find adapters</em> lists those made for it.{{end}}
An adapter runs on the model it was made for, and only as a GGUF adapter file;
open one to see which of its files are.</p>
{{else}}
<div class="explore-filters">
<nav class="pills small" aria-label="Task">{{range .Tasks}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
<nav class="pills small" aria-label="Size">{{range .Sizes}}<a href="{{.Href}}"{{if .Active}} class="on" aria-current="true"{{end}}>{{.Label}}</a>{{end}}</nav>
</div>
{{end}}
{{end}}
<form method="get" action="/admin/mirror" class="explore-search">
<input type="hidden" name="tab" value="explore">
{{with .Explore}}<input type="hidden" name="view" value="{{.View}}">
<input type="hidden" name="task" value="{{.Task}}">
<input type="hidden" name="size" value="{{.Size}}">
{{if .Of}}<input type="hidden" name="of" value="{{.Of}}">{{end}}
<input type="hidden" name="x" value="1">{{end}}
<input type="search" name="q" value="{{.Query}}" placeholder="Search Hugging Face — qwen coder, gemma, bge…" aria-label="Search Hugging Face">
<label class="check"><input type="checkbox" name="here" value="1" onchange="this.form.submit()"{{if .Explore}}{{if .Explore.Here}} checked{{end}}{{end}}> only what these nodes can run</label>
@@ -405,7 +415,9 @@ version and the old weights stay where they are.</p>
<tr><th>What</th><th class="num">Size</th><th>Published checksum</th><th></th></tr>
{{range .RepoSets}}<tr>
<td><code>{{.Name}}</code>
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}</td>
{{if .Shards}}<span class="opt">{{.Shards}} shards, fetched together</span>{{end}}
{{if eq .Kind "adapter"}}<span class="tag new" title="a LoRA adapter: it runs on a {{.Arch}} model it was made for, not on its own">LoRA adapter · {{.Arch}}</span>
{{else if eq .Kind "model"}}<span class="opt">model{{with .Arch}} · {{.}}{{end}}</span>{{end}}</td>
<td class="num">{{bytes .Bytes}}</td>
<td>{{with .SHA256}}<code title="{{.}}">{{slice . 0 12}}</code>{{else}}—{{end}}</td>
<td>
@@ -489,6 +501,43 @@ computes once it has it.</p>
<ul>{{range .Blocked}}<li>{{.}}</li>{{end}}</ul></details>{{end}}
{{end}}
{{if not .Declared}}{{$mname := .Name}}
<div class="adapters">
<h4>LoRA adapters <a class="note" href="{{.FindAdapters}}">find adapters on Hugging Face →</a></h4>
{{if .Adapters}}<table class="compact rows">
<tr><th>Asked for as</th><th>File</th><th class="num">Scale</th><th></th></tr>
{{range .Adapters}}{{$a := .}}<tr>
<td><code>{{.Name}}</code></td>
<td class="clip" title="{{.Source}}">{{.Source}}</td>
<td class="num">{{.EffectiveScale}}</td>
<td class="actions"><form method="post" action="/admin/mirror/adapters" class="inline">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{$mname}}">
<input type="hidden" name="open" value="{{$row.Open}}">
<input type="hidden" name="op" value="remove">
<input type="hidden" name="name" value="{{$a.Name}}">
<button type="submit" class="link danger">remove</button></form></td>
</tr>{{end}}</table>
{{else}}<p class="note">None. A request for an adapter's name is served by this model's replicas with it applied.</p>{{end}}
{{if .AdapterChoices}}{{$m := .}}
<form method="post" action="/admin/mirror/adapters" class="adapter-form">
<input type="hidden" name="csrf" value="{{$csrf}}">
<input type="hidden" name="model" value="{{.Name}}">
<input type="hidden" name="open" value="{{$row.Open}}">
<input type="hidden" name="op" value="add">
<div class="field-row">
<label>Adapter file <select name="source" required
onchange="this.form.elements.name.value = this.selectedOptions[0].dataset.suggest">
{{range .AdapterChoices}}<option value="{{.URL}}" data-suggest="{{.Suggest}}">{{.Label}}</option>{{end}}
</select></label>
<label>Asked for as <input type="text" name="name" value="{{(index .AdapterChoices 0).Suggest}}" required></label>
<label>Scale <input type="text" name="scale" value="1" size="4"></label>
<button type="submit">Attach</button>
</div>
</form>{{else}}<p class="note">No held adapter file for this model's architecture. Fetch one from Hugging Face first.</p>{{end}}
</div>
{{end}}
{{if .Declared}}
<p class="note">Set in server.yaml, which is read again on every start: change its replicas there.</p>
{{else}}