The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
CI / test (push) Successful in 10m35s
CI / test (push) Successful in 10m35s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4ee9ba54ef
commit
845de8eef4
13 files changed
+513
-21
No files matched your search
@@ -1889,6 +1889,10 @@ served only from a definition written by hand. In order:
|
||||
- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
|
||||
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
|
||||
|
||||
- N7b. The mirror page's Strata tab: every family and size, fetch (GGUF) or serve (packs Strata fetches itself:
|
||||
NVFP4, DeepSeek V4.1). Done 2026-10-30. Community `strata-ple` packs Strata 0.2.7 does not list stay unservable;
|
||||
running them would mean starting the engine past Strata's setup (`--pack`, `--native`, `--ple-gguf`).
|
||||
|
||||
**C. Operating it.**
|
||||
- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
|
||||
- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
|
||||
|
||||
@@ -26,6 +26,16 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **The mirror page's Strata tab.** Every family and size Strata runs, with its download, the RAM it keeps and where
|
||||
it would run here; a GGUF size fetched into the mirror in one press (checked against Hugging Face's checksums),
|
||||
and the packs Strata fetches itself — NVFP4, DeepSeek V4.1 — served at once, their repository the source.
|
||||
`ctl model plan` and `ctl model serve` take those repositories too, and the explorer names a `strata-ple`
|
||||
repository for what it is: part of a pack Strata knows, or one Strata 0.2.7 does not.
|
||||
|
||||
## v0.16.0 — 2026-10-30
|
||||
|
||||
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
|
||||
|
||||
+18
-1
@@ -761,7 +761,24 @@ llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
|
||||
```
|
||||
|
||||
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
|
||||
and the mirror's serve form serves such a version on Strata. A Strata profile
|
||||
and the mirror's serve form serves such a version on Strata.
|
||||
|
||||
The mirror page's **Strata** tab lists every family and size Strata runs, with
|
||||
what each downloads and keeps in RAM and where it would run here:
|
||||
|
||||
- **Fetch into the mirror**, for the GGUF families: the size's files, checked
|
||||
against Hugging Face's checksums; the version's serve form then serves it on
|
||||
Strata.
|
||||
- **Serve**, for the packs Strata fetches itself — the NVFP4 packs
|
||||
(`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` and its NVIDIA
|
||||
sibling, NVIDIA's FP4-Local) and DeepSeek V4.1: the model is added with the
|
||||
repository as its source, and its first start fetches tens of gigabytes onto
|
||||
the node, each file pinned and checked by Strata, before preparing it.
|
||||
`ctl model serve` does the same for these repositories.
|
||||
|
||||
Repositories of architecture `strata-ple` that Strata 0.2.7 does not list —
|
||||
community packs such as `…-Q8_0-Strata` — are named as such in the explorer:
|
||||
Strata's setup refuses them, and llama.cpp cannot read them. A Strata profile
|
||||
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
|
||||
checks against what Strata runs, and reserves the RAM the size keeps its experts
|
||||
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
|
||||
|
||||
@@ -35,6 +35,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// Registry reads a repository.
|
||||
@@ -412,6 +413,40 @@ func (s *Server) run(j *Job) {
|
||||
if req.TP > 0 && req.GPUsPerNode == 0 {
|
||||
req.GPUsPerNode = req.TP
|
||||
}
|
||||
// A pack Strata fetches itself: no mirror; its repository is the source.
|
||||
if _, size, ok := strata.ForFetchedRepo(repo.ID); ok {
|
||||
plan, err := s.plan(ctx, repo, PlanRequest{Repo: repo.ID})
|
||||
if err != nil {
|
||||
s.fail(j, "registry", err)
|
||||
return
|
||||
}
|
||||
set := hf.FileSet{Name: size.Name}
|
||||
layout, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: req.GPUClass}, plan, true, set)
|
||||
if err != nil {
|
||||
s.fail(j, "registry", err)
|
||||
return
|
||||
}
|
||||
name := req.Name
|
||||
if name == "" {
|
||||
name = hf.ModelName(repo.ID)
|
||||
}
|
||||
s.mu.Lock()
|
||||
j.Model = name
|
||||
s.mu.Unlock()
|
||||
if !req.sized {
|
||||
req.ContextLength = strataContext
|
||||
}
|
||||
s.step(j, "registry", Done, fmt.Sprintf("Strata's ready-made %s pack, revision %s; runs on %s",
|
||||
size.Name, repo.ShortRevision(), layout[0].Label()))
|
||||
s.step(j, "mirror", Done, fmt.Sprintf("not through the mirror: Strata fetches about %.0f GB onto the node itself", size.DownloadGB))
|
||||
profiles := profilesFor(layout, req, 0, 0)
|
||||
profiles[0].StartupTimeout = 4 * time.Hour
|
||||
s.mu.Lock()
|
||||
j.Request = req
|
||||
s.mu.Unlock()
|
||||
s.addAndStart(ctx, j, repo, set, name, repo.ID, profiles, hardwareExists(layout))
|
||||
return
|
||||
}
|
||||
// The plan: which file suits the hardware, and how it spreads over it.
|
||||
planCtx := req.CtxPool
|
||||
if req.Format == "safetensors" {
|
||||
|
||||
@@ -98,6 +98,19 @@ func (s *Server) Plan(ctx context.Context, req PlanRequest) (Plan, error) {
|
||||
}
|
||||
|
||||
func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan, error) {
|
||||
// A pack Strata fetches itself: one way, Strata's, whatever its files.
|
||||
if fam, size, ok := strata.ForFetchedRepo(repo.ID); ok {
|
||||
art := modelplan.Artifact{Name: size.Name, Format: "strata", Bytes: int64(size.DownloadGB * 1e9), Quant: size.Name}
|
||||
p := Plan{Repo: repo.ID, Revision: repo.Revision, Format: "strata", Arch: repo.Architecture, License: repo.License,
|
||||
Context: strataContext, Servable: true, Note: "Strata fetches this pack onto the node itself on its first start"}
|
||||
p.Options = modelplan.StrataOptions(fam.Name, []modelplan.Artifact{art}, s.deps.Pool())
|
||||
pa := PlannedArtifact{Artifact: art}
|
||||
if len(p.Options) > 0 {
|
||||
pa.Best = &p.Options[0]
|
||||
}
|
||||
p.Artifacts = []PlannedArtifact{pa}
|
||||
return p, nil
|
||||
}
|
||||
p := Plan{Repo: repo.ID, Revision: repo.Revision, Format: req.Format, Arch: repo.Architecture,
|
||||
License: repo.License, Params: repo.Params, Context: req.Context, Gated: repo.Gated.Is(), Servable: true,
|
||||
Projectors: Projectors(repo)}
|
||||
|
||||
@@ -35,3 +35,18 @@ func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAReadyMadePackIsPlannedOnStrataOnly(t *testing.T) {
|
||||
s := New(Deps{Pool: func() []modelplan.PoolNode {
|
||||
return []modelplan.PoolNode{{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1,
|
||||
HostMemGB: 251, HostFreeGB: 200, Engines: []string{"strata"}}}
|
||||
}})
|
||||
p, err := s.plan(t.Context(), hf.Repo{ID: "Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata", Revision: "abc",
|
||||
Architecture: "strata-ple"}, PlanRequest{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !p.Servable || len(p.Options) != 1 || p.Options[0].StrataFamily != "nvfp4" || p.Options[0].StrataSize != "NVFP4-NVIDIA" {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
}
|
||||
+68
-14
@@ -14,9 +14,9 @@ import (
|
||||
|
||||
// Family is a model Strata runs, from one publisher's files.
|
||||
type Family struct {
|
||||
// Name is setup's --family.
|
||||
Name string
|
||||
Title string
|
||||
// Name is setup's --family; Model the name Strata gives the model.
|
||||
Name, Model string
|
||||
Title string
|
||||
// Repo is the Hugging Face repository its files are in; empty when
|
||||
// Strata fetches them itself from several.
|
||||
Repo string
|
||||
@@ -42,18 +42,21 @@ type Size struct {
|
||||
RAMGB float64
|
||||
// NVIDIAOnly is a size whose kernels only NVIDIA cards have.
|
||||
NVIDIAOnly bool
|
||||
// Repo is a size's own repository, for a ready-made pack Strata
|
||||
// fetches itself.
|
||||
Repo string
|
||||
}
|
||||
|
||||
// Families are the families Strata 0.2.7 runs.
|
||||
var Families = []Family{
|
||||
{Name: "qwen", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
|
||||
{Name: "swift", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
|
||||
{Name: "coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
|
||||
{Name: "unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
|
||||
{Name: "gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
|
||||
{Name: "nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
|
||||
{Name: "qwen36", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
|
||||
{Name: "deepseek41", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true},
|
||||
{Name: "qwen", Model: "qwen3.8-flash-next", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
|
||||
{Name: "swift", Model: "swift-1.5", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
|
||||
{Name: "coder", Model: "qwen3.8-flash-next-coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
|
||||
{Name: "unsloth", Model: "qwen3.8-flash-next-unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
|
||||
{Name: "gyro", Model: "qwen3.8-flash-next-gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
|
||||
{Name: "nvfp4", Model: "qwen3.8-flash-next-nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
|
||||
{Name: "qwen36", Model: "qwen3.6-35b-a3b", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
|
||||
{Name: "deepseek41", Model: "deepseek-v4.1-flash", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true},
|
||||
}
|
||||
|
||||
// Sizes are the sizes Strata 0.2.7 runs, and which families have each.
|
||||
@@ -70,9 +73,12 @@ var Sizes = []Size{
|
||||
{Name: "Q8_0", Families: []string{"unsloth", "qwen36"}, DownloadGB: 188.2, RAMGB: 136},
|
||||
{Name: "Gyro-S", Families: []string{"gyro"}, DownloadGB: 58.5, RAMGB: 27.6},
|
||||
{Name: "Gyro-M", Families: []string{"gyro"}, DownloadGB: 92.0, RAMGB: 35},
|
||||
{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true,
|
||||
Repo: "Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata"},
|
||||
{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true,
|
||||
Repo: "Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata"},
|
||||
{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true,
|
||||
Repo: "nvidia/Qwen3.8-Flash-Next-FP4-Local"},
|
||||
{Name: "IST-DAS-1.75bpw", Families: []string{"deepseek41"}, DownloadGB: 239, RAMGB: 34},
|
||||
{Name: "IST-DAS-2.00bpw", Families: []string{"deepseek41"}, DownloadGB: 257, RAMGB: 40},
|
||||
{Name: "IST-DAS-2.25bpw", Families: []string{"deepseek41"}, DownloadGB: 274, RAMGB: 44},
|
||||
@@ -172,3 +178,51 @@ func SizeForFile(family, file string) (Size, bool) {
|
||||
}
|
||||
return best, best.Name != ""
|
||||
}
|
||||
|
||||
// FetchesItself is a family whose files Strata's setup fetches itself, each
|
||||
// pinned to a commit and checked: ready-made packs, and DeepSeek's sets with
|
||||
// an engine of their own. A model of it names its repository as the source.
|
||||
func (f Family) FetchesItself() bool { return f.Name == "nvfp4" || f.Engine }
|
||||
|
||||
// RepoOf is the repository a size of a family is in.
|
||||
func RepoOf(f Family, s Size) string {
|
||||
if s.Repo != "" {
|
||||
return s.Repo
|
||||
}
|
||||
return f.Repo
|
||||
}
|
||||
|
||||
// ForFetchedRepo is the family and size a repository is, for one Strata
|
||||
// fetches itself: a ready-made pack's own, or DeepSeek's (whose size the
|
||||
// repository does not say, so the smallest).
|
||||
func ForFetchedRepo(repo string) (Family, Size, bool) {
|
||||
for _, f := range Families {
|
||||
if !f.FetchesItself() {
|
||||
continue
|
||||
}
|
||||
for _, s := range SizesOf(f.Name) {
|
||||
if strings.EqualFold(RepoOf(f, s), repo) {
|
||||
return f, s, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return Family{}, Size{}, false
|
||||
}
|
||||
|
||||
// Fetchable says whether a size's files can be fetched by llmgw from the
|
||||
// family's repository: not where Strata fetches them itself (NVFP4's packs,
|
||||
// DeepSeek's sets with an engine of their own).
|
||||
func (f Family) Fetchable() bool { return f.Repo != "" && !f.Engine }
|
||||
|
||||
// FilesOf are the weights of a size among a repository's files: its GGUF
|
||||
// shards, not the picture reader.
|
||||
func FilesOf(size string, paths []string) []string {
|
||||
var out []string
|
||||
for _, p := range paths {
|
||||
u := strings.ToUpper(p)
|
||||
if strings.HasSuffix(u, ".GGUF") && !strings.Contains(u, "MMPROJ") && strings.Contains(u, strings.ToUpper(size)) {
|
||||
out = append(out, p)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// MirrorAdmin is the mirror's management API as this page needs it.
|
||||
@@ -169,6 +170,10 @@ type adminMirrorData struct {
|
||||
// the serve action is then not offered rather than offered and refused.
|
||||
HasCatalog bool
|
||||
|
||||
// Strata is the Strata tab: the models it runs, and where they would
|
||||
// run here.
|
||||
Strata *strataView
|
||||
|
||||
// The explorer.
|
||||
HasRegistry bool
|
||||
Query string
|
||||
@@ -256,6 +261,14 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
|
||||
render(w, r, "mirror_repo", d)
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/strata", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
if r.FormValue("action") == "serve" {
|
||||
return p.serveStrata(ctx, langOf(r), admin.Email, r.FormValue("family"), r.FormValue("size"))
|
||||
}
|
||||
return p.fetchStrata(ctx, langOf(r), m, r.FormValue("family"), r.FormValue("size"))
|
||||
}))
|
||||
|
||||
mux.HandleFunc("POST /admin/mirror/fetch", p.mirrorAction(m,
|
||||
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
|
||||
// One form posts one source or fifteen. A sharded model is one
|
||||
@@ -462,6 +475,9 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
|
||||
if d.Tab == tabExplore {
|
||||
p.explore(r, &d)
|
||||
}
|
||||
if d.Tab == tabStrata {
|
||||
d.Strata = p.strataView(l)
|
||||
}
|
||||
|
||||
got, err := m.List(r.Context())
|
||||
if err != nil {
|
||||
@@ -618,6 +634,7 @@ const (
|
||||
tabHeld = "held"
|
||||
tabExplore = "explore"
|
||||
tabFetch = "fetch"
|
||||
tabStrata = "strata"
|
||||
)
|
||||
|
||||
// mirrorTab is one of the page's tabs.
|
||||
@@ -636,7 +653,7 @@ type mirrorTab struct {
|
||||
// shown, where its effect is, unless it was refused: then the form it came
|
||||
// from, to try again.
|
||||
func mirrorTabFor(r *http.Request, errMsg string) string {
|
||||
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
|
||||
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch || t == tabStrata }
|
||||
if r.Method == http.MethodPost {
|
||||
if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
|
||||
return t
|
||||
@@ -673,6 +690,10 @@ func mirrorTabs(l *Lang, d adminMirrorData) []mirrorTab {
|
||||
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.explore"),
|
||||
Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
|
||||
}
|
||||
if d.HasRegistry {
|
||||
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.strata"),
|
||||
Href: "/admin/mirror?tab=strata", Active: d.Tab == tabStrata})
|
||||
}
|
||||
return append(tabs, mirrorTab{Label: l.T("mirror.tab.fetch"),
|
||||
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
|
||||
}
|
||||
@@ -823,9 +844,15 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
|
||||
Version: got.ShortRevision(),
|
||||
}
|
||||
rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
|
||||
// A family Strata runs: one card and the experts in RAM, whether
|
||||
// or not llama.cpp could load it.
|
||||
if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
|
||||
// A pack made for Strata: served whole, from the Strata tab, when
|
||||
// Strata knows it; named for what it is when it does not.
|
||||
if fam, size, ok := strata.ForFetchedRepo(got.ID); ok {
|
||||
rs.Verdict = gguf.Verdict{Checked: true, Reason: langOf(r).T("strata.pack_files", fam.Title, size.Name)}
|
||||
} else if got.Architecture == "strata-ple" {
|
||||
rs.Verdict = gguf.Verdict{Checked: true, Reason: langOf(r).T("strata.unknown_pack")}
|
||||
} else if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
|
||||
// A family Strata runs: one card and the experts in RAM,
|
||||
// whether or not llama.cpp could load it.
|
||||
rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
|
||||
} else if fit != nil && rs.Verdict.Servable {
|
||||
if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// The mirror page's Strata tab: the models Strata runs, each size with what
|
||||
// it downloads and keeps in RAM and where it would run here, and a button
|
||||
// that fetches a size's files into the mirror, where the version's serve
|
||||
// form then serves it on Strata.
|
||||
|
||||
type strataView struct {
|
||||
// Nodes are the nodes that have Strata installed.
|
||||
Nodes []string
|
||||
Families []strataFamilyRow
|
||||
}
|
||||
|
||||
type strataFamilyRow struct {
|
||||
Name, Title, Repo string
|
||||
// Fetchable is a family whose files llmgw fetches into the mirror;
|
||||
// FetchesItself one whose files Strata fetches onto the node, served
|
||||
// from here at once.
|
||||
Fetchable, FetchesItself bool
|
||||
Sizes []strataSizeRow
|
||||
}
|
||||
|
||||
type strataSizeRow struct {
|
||||
Size string
|
||||
DownloadGB, RAMGB string
|
||||
Runs string
|
||||
Fits, NVIDIAOnly bool
|
||||
}
|
||||
|
||||
func (p *AccountPages) strataView(l *Lang) *strataView {
|
||||
v := &strataView{}
|
||||
var pool []modelplan.PoolNode
|
||||
if fit := p.fitOf(); fit != nil {
|
||||
pool = fit.pool
|
||||
}
|
||||
for _, n := range pool {
|
||||
if slices.Contains(n.Engines, "strata") {
|
||||
v.Nodes = append(v.Nodes, n.ID)
|
||||
}
|
||||
}
|
||||
for _, f := range strata.Families {
|
||||
row := strataFamilyRow{Name: f.Name, Title: f.Title, Repo: f.Repo, Fetchable: f.Fetchable(),
|
||||
FetchesItself: f.FetchesItself()}
|
||||
for _, s := range strata.SizesOf(f.Name) {
|
||||
sr := strataSizeRow{Size: s.Name, DownloadGB: l.Num(s.DownloadGB, 0), RAMGB: l.Int(int64(s.MemoryGB())),
|
||||
NVIDIAOnly: s.NVIDIAOnly}
|
||||
opts := modelplan.StrataOptions(f.Name, []modelplan.Artifact{{Name: s.Name + ".gguf", Format: "gguf",
|
||||
Bytes: int64(s.DownloadGB * 1e9)}}, pool)
|
||||
switch {
|
||||
case len(opts) > 0:
|
||||
sr.Fits, sr.Runs = true, l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0))
|
||||
case len(v.Nodes) == 0:
|
||||
sr.Runs = l.T("strata.no_node")
|
||||
default:
|
||||
sr.Runs = l.T("strata.no_room", s.MemoryGB())
|
||||
}
|
||||
row.Sizes = append(row.Sizes, sr)
|
||||
}
|
||||
v.Families = append(v.Families, row)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// serveStrata adds a model of a family Strata fetches itself: its source
|
||||
// the repository, its replica on a card of the class the plan found.
|
||||
func (p *AccountPages) serveStrata(ctx context.Context, l *Lang, by, family, size string) (string, error) {
|
||||
fam, ok := strata.FamilyNamed(family)
|
||||
s, err := strata.SizeOf(family, size)
|
||||
if !ok || err != nil {
|
||||
return "", errors.New(l.T("strata.err.unknown", family, size))
|
||||
}
|
||||
if !fam.FetchesItself() || p.catalog == nil {
|
||||
return "", errors.New(l.T("strata.err.not_servable", fam.Title))
|
||||
}
|
||||
var pool []modelplan.PoolNode
|
||||
if fit := p.fitOf(); fit != nil {
|
||||
pool = fit.pool
|
||||
}
|
||||
opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: s.Name + ".gguf", Format: "gguf"}}, pool)
|
||||
if len(opts) == 0 {
|
||||
return "", errors.New(l.T("strata.no_room", s.MemoryGB()))
|
||||
}
|
||||
name := fam.Model + "-" + strings.ToLower(s.Name)
|
||||
disk := s.DownloadGB
|
||||
if s.DiskGB > disk {
|
||||
disk = s.DiskGB
|
||||
}
|
||||
m := catalog.Model{Name: name, Source: strata.RepoOf(fam, s), RoutingProfile: "chat", ContextLength: strataContext,
|
||||
Scaling: catalog.Scaling{Min: 1, Max: 1},
|
||||
Profiles: []catalog.Profile{{Name: "strata-" + opts[0].Class, Engine: "strata", GPUClass: opts[0].Class,
|
||||
GPUsPerNode: 1, Weight: 1, Priority: 100, MinMemoryGB: s.MemoryGB(), MinDiskGB: int(disk) + 10,
|
||||
Args: []string{"--family", fam.Name, "--size", s.Name, "--context", strconv.Itoa(strataContext), "--vision", "no"},
|
||||
// Its files come onto the node on its first start, tens of
|
||||
// gigabytes from the internet, before it is prepared.
|
||||
StartupTimeout: 4 * time.Hour}}}
|
||||
if err := p.catalog.Add(ctx, m, by); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return l.T("strata.msg.served", name, l.Num(disk, 0)), nil
|
||||
}
|
||||
|
||||
// strataContext is the context a Strata model is served with from here.
|
||||
const strataContext = 32768
|
||||
|
||||
// fetchStrata fetches a Strata size's files from its family's repository
|
||||
// into the mirror, each checked against the sha256 the registry published.
|
||||
func (p *AccountPages) fetchStrata(ctx context.Context, l *Lang, m MirrorAdmin, family, size string) (string, error) {
|
||||
fam, ok := strata.FamilyNamed(family)
|
||||
if _, err := strata.SizeOf(family, size); !ok || err != nil {
|
||||
return "", errors.New(l.T("strata.err.unknown", family, size))
|
||||
}
|
||||
if !fam.Fetchable() || p.registry == nil {
|
||||
return "", errors.New(l.T("strata.err.not_fetchable", fam.Title))
|
||||
}
|
||||
repo, err := p.registry.Repo(ctx, fam.Repo)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var paths []string
|
||||
sums := map[string]string{}
|
||||
for _, f := range repo.Files {
|
||||
paths = append(paths, f.Path)
|
||||
sums[f.Path] = f.SHA256
|
||||
}
|
||||
files := strata.FilesOf(size, paths)
|
||||
if len(files) == 0 {
|
||||
return "", errors.New(l.T("strata.err.no_files", fam.Repo, size))
|
||||
}
|
||||
model := fam.Model + "-" + strings.ToLower(size)
|
||||
version := repo.ShortRevision()
|
||||
for i, path := range files {
|
||||
req := mirror.FetchRequest{Source: repo.DownloadURL("", path), Model: model, Version: version,
|
||||
File: mirror.LastPathElement(path), SHA256: sums[path]}
|
||||
if _, err := m.Fetch(ctx, req); err != nil && !strings.Contains(err.Error(), "already") {
|
||||
return "", errors.New(l.T("mirror.err.partly_started", i, len(files), req.File, l.Err(err)))
|
||||
}
|
||||
}
|
||||
return l.T("mirror.msg.started", model, version, l.N("mirror.files", int64(len(files)))), nil
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
)
|
||||
|
||||
// strataFleet is one node with an RTX 6000, 251 GB and Strata installed.
|
||||
type strataFleet struct{ fakeFleet }
|
||||
|
||||
func (strataFleet) Nodes() []adminapi.NodeView {
|
||||
return []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUFree: 1, MemoryGB: 251,
|
||||
Classes: []string{"6000-24g"}, Engines: []string{"llamacpp", "strata"}}}
|
||||
}
|
||||
|
||||
func TestTheStrataTabListsWhatRunsHereAndFetchesASize(t *testing.T) {
|
||||
reg := &fakeRegistry{repo: hf.Repo{ID: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF",
|
||||
Revision: "0123456789abcdef0123456789abcdef01234567", Files: []hf.File{
|
||||
{Path: "IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", SHA256: "aa"},
|
||||
{Path: "IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf", SHA256: "bb"},
|
||||
{Path: "IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf"},
|
||||
{Path: "mmproj-Qwen3.8-Flash-Next-BF16.gguf"},
|
||||
}}}
|
||||
m := &fakeMirror{}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Registry: reg, Fleet: strataFleet{}}, true)
|
||||
|
||||
body := mirrorGet(t, mux, "/admin/mirror?tab=strata").Body.String()
|
||||
for _, want := range []string{"Strata is installed on: cuda01", "Qwen3.8-Flash-Next", "IQ2_XS",
|
||||
"1× 6000-24g with Strata, 52 GB of RAM", `name="size" value="IQ2_XS"`, "Strata fetches these onto the node itself"} {
|
||||
if !strings.Contains(body, want) {
|
||||
t.Errorf("the tab lacks %q", want)
|
||||
}
|
||||
}
|
||||
// Unsloth's Q8_0 keeps 140 GB: it runs here; nothing past the node's RAM does.
|
||||
if strings.Contains(body, `value="UD-Q8_K_XL"`) {
|
||||
t.Error("offered DeepSeek's 306 GB size, which no node holds, and which Strata fetches itself")
|
||||
}
|
||||
|
||||
mirrorPost(t, mux, "/admin/mirror/strata", url.Values{"csrf": {"tok"}, "family": {"qwen"}, "size": {"IQ2_XS"}})
|
||||
if len(m.all) != 2 {
|
||||
t.Fatalf("fetched %+v", m.all)
|
||||
}
|
||||
f := m.all[0]
|
||||
if f.Model != "qwen3.8-flash-next-iq2_xs" || f.Version != "01234567" || f.SHA256 != "aa" ||
|
||||
!strings.Contains(f.Source, "/resolve/0123456789abcdef0123456789abcdef01234567/IQ2_XS/") {
|
||||
t.Errorf("%+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
// A ready-made pack is served at once, its repository the source, Strata
|
||||
// fetching it onto the node.
|
||||
func TestAReadyMadeStrataPackIsServedFromTheTab(t *testing.T) {
|
||||
cat := &fakeCatalog{}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: &fakeRegistry{}, Catalog: cat, Fleet: strataFleet{}}, true)
|
||||
if body := mirrorGet(t, mux, "/admin/mirror?tab=strata").Body.String(); !strings.Contains(body, `name="action" value="serve"`) {
|
||||
t.Fatal("no serve button for a pack")
|
||||
}
|
||||
mirrorPost(t, mux, "/admin/mirror/strata", url.Values{"csrf": {"tok"}, "action": {"serve"}, "family": {"nvfp4"}, "size": {"NVFP4"}})
|
||||
if len(cat.added) != 1 {
|
||||
t.Fatalf("added %+v", cat.added)
|
||||
}
|
||||
m := cat.added[0]
|
||||
p := m.Profiles[0]
|
||||
if m.Name != "qwen3.8-flash-next-nvfp4-nvfp4" || m.Source != "Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata" ||
|
||||
p.Engine != "strata" || p.GPUClass != "6000-24g" || p.MinMemoryGB != 84 || p.StartupTimeout < time.Hour ||
|
||||
strings.Join(p.Args, " ") != "--family nvfp4 --size NVFP4 --context 32768 --vision no" {
|
||||
t.Errorf("%+v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStrataPacksFilesAreNamedForWhatTheyAre(t *testing.T) {
|
||||
for repo, want := range map[string]string{
|
||||
"Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata": "ready-made Qwen3.8-Flash-Next (NVFP4) NVFP4 pack",
|
||||
"trymirai/Qwen3.8-Flash-Next-S-strata-experimental": "Strata 0.2.7 does not know",
|
||||
} {
|
||||
reg := &fakeRegistry{repo: hf.Repo{ID: repo, Architecture: "strata-ple", Revision: "abc",
|
||||
Files: []hf.File{{Path: "ple-fp8.gguf", Bytes: 51e9}}}}
|
||||
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg, Fleet: strataFleet{}}, true)
|
||||
if body := mirrorGet(t, mux, "/admin/mirror?repo="+repo).Body.String(); !strings.Contains(body, want) {
|
||||
t.Errorf("%s: no %q", repo, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -257,5 +257,26 @@
|
||||
"mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
|
||||
"mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
|
||||
"mirror.err.adapter_op": "add or remove an adapter",
|
||||
"explore.strata": "1× %s with Strata, %s GB of RAM"
|
||||
"explore.strata": "1× %s with Strata, %s GB of RAM",
|
||||
"mirror.tab.strata": "Strata",
|
||||
"strata.intro": "Strata runs these mixture-of-experts models with the experts in a node's RAM and the rest on one card of 12 GB or more: a model that fits no card, served from one. Fetch a size into the mirror; its serve form on the mirror's tab then serves it on Strata. The first start prepares it on the node, which takes minutes.",
|
||||
"strata.nodes": "Strata is installed on: %s",
|
||||
"strata.none": "No node has Strata installed. Name it among the cluster's engines (<code>llmgw ctl config set engines.strata 0.2.7</code>) and the nodes with an NVIDIA card install it.",
|
||||
"strata.self": "Strata fetches these onto the node itself, each file pinned and checked, on the model's first start: served from here, they do not go through the mirror.",
|
||||
"strata.col.size": "Size",
|
||||
"strata.col.download": "Download",
|
||||
"strata.col.ram": "RAM it keeps",
|
||||
"strata.col.runs": "Here",
|
||||
"strata.fetch": "Fetch into the mirror",
|
||||
"strata.nvidia": "NVIDIA only",
|
||||
"strata.no_node": "no node has Strata",
|
||||
"strata.no_room": "no node with Strata has %d GB of RAM and a card of 12 GB",
|
||||
"strata.err.unknown": "Strata runs no %s %s",
|
||||
"strata.err.not_fetchable": "Strata fetches %s itself; serve it with a definition that names its family and size.",
|
||||
"strata.err.no_files": "%s has no files of size %s",
|
||||
"strata.serve": "Serve",
|
||||
"strata.err.not_servable": "%s is fetched into the mirror first, then served from its version",
|
||||
"strata.msg.served": "%s added: its first start fetches about %s GB onto the node and prepares it, which can take hours on a slow link; Replicas shows how far it is.",
|
||||
"strata.pack_files": "part of Strata's ready-made %s %s pack: served whole from the mirror's Strata tab, Strata fetching it onto the node",
|
||||
"strata.unknown_pack": "a pack made for Strata that Strata 0.2.7 does not know: its setup refuses it, and llama.cpp cannot read it"
|
||||
}
|
||||
@@ -250,5 +250,26 @@
|
||||
"mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
|
||||
"mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
|
||||
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
|
||||
"explore.strata": "Strata ile 1× %s, %s GB RAM"
|
||||
"explore.strata": "Strata ile 1× %s, %s GB RAM",
|
||||
"mirror.tab.strata": "Strata",
|
||||
"strata.intro": "Strata bu uzman karışımı (MoE) modelleri, uzmanları node'un RAM'inde, gerisini 12 GB ya da daha büyük tek bir kartta tutarak çalıştırır: hiçbir karta sığmayan bir model tek karttan sunulur. Bir boyutu aynaya indirin; aynadakiler sekmesindeki sunma formu onu Strata ile sunar. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer.",
|
||||
"strata.nodes": "Strata kurulu node'lar: %s",
|
||||
"strata.none": "Hiçbir node'da Strata kurulu değil. Kümenin motorları arasında adını verin (<code>llmgw ctl config set engines.strata 0.2.7</code>); NVIDIA kartı olan node'lar onu kurar.",
|
||||
"strata.self": "Bunları Strata, modelin ilk açılışında her dosyayı sabitleyip denetleyerek node'a kendisi indirir: buradan sunulur, aynadan geçmez.",
|
||||
"strata.col.size": "Boyut",
|
||||
"strata.col.download": "İndirme",
|
||||
"strata.col.ram": "Ayırdığı RAM",
|
||||
"strata.col.runs": "Burada",
|
||||
"strata.fetch": "Aynaya indir",
|
||||
"strata.nvidia": "Yalnız NVIDIA",
|
||||
"strata.no_node": "Strata hiçbir node'da yok",
|
||||
"strata.no_room": "Strata kurulu hiçbir node'da %d GB RAM ve 12 GB'lık kart yok",
|
||||
"strata.err.unknown": "Strata %s %s çalıştırmıyor",
|
||||
"strata.err.not_fetchable": "%s dosyalarını Strata kendisi indirir; ailesini ve boyutunu veren bir tanımla sunun.",
|
||||
"strata.err.no_files": "%s içinde %s boyutunda dosya yok",
|
||||
"strata.serve": "Sun",
|
||||
"strata.err.not_servable": "%s önce aynaya indirilir, sonra sürümünden sunulur",
|
||||
"strata.msg.served": "%s eklendi: ilk açılışında node'a yaklaşık %s GB indirip hazırlar; yavaş bağlantıda saatler sürebilir. İlerleyişi Replikalar gösterir.",
|
||||
"strata.pack_files": "Strata'nın hazır %s %s paketinin parçası: aynanın Strata sekmesinden bütün olarak sunulur, Strata onu node'a kendisi indirir",
|
||||
"strata.unknown_pack": "Strata için hazırlanmış ama Strata 0.2.7'nin tanımadığı bir paket: kurulumu onu reddeder, llama.cpp de okuyamaz"
|
||||
}
|
||||
@@ -424,6 +424,40 @@ document.querySelector("table.explore").addEventListener("submit", async functio
|
||||
{{if .Repo}}{{template "mirror_repo" .}}{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if and (eq .Tab "strata") .Strata}}
|
||||
<p class="note">{{t "strata.intro"}}</p>
|
||||
{{if .Strata.Nodes}}<p class="sub">{{t "strata.nodes" (join .Strata.Nodes)}}</p>
|
||||
{{else}}<p class="note error-row">{{th "strata.none"}}</p>{{end}}
|
||||
{{range .Strata.Families}}
|
||||
<h3>{{.Title}} <span class="opt"><code>--family {{.Name}}</code>{{if .Repo}} · <a href="/admin/mirror?repo={{.Repo}}">{{.Repo}}</a>{{end}}</span></h3>
|
||||
{{if .FetchesItself}}<p class="sub">{{t "strata.self"}}</p>{{end}}
|
||||
<table class="rows">
|
||||
<tr><th>{{t "strata.col.size"}}</th><th>{{t "strata.col.download"}}</th><th>{{t "strata.col.ram"}}</th><th>{{t "strata.col.runs"}}</th><th></th></tr>
|
||||
{{$fam := .}}
|
||||
{{range .Sizes}}
|
||||
<tr>
|
||||
<td><code>{{.Size}}</code>{{if .NVIDIAOnly}} <span class="opt">{{t "strata.nvidia"}}</span>{{end}}</td>
|
||||
<td>{{.DownloadGB}} GB</td>
|
||||
<td>{{.RAMGB}} GB</td>
|
||||
<td>{{if .Fits}}{{.Runs}}{{else}}<span class="opt">{{.Runs}}</span>{{end}}</td>
|
||||
<td>{{if and $fam.FetchesItself .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="strata">
|
||||
<input type="hidden" name="action" value="serve">
|
||||
<input type="hidden" name="family" value="{{$fam.Name}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{if and $fam.Fetchable .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="tab" value="strata">
|
||||
<input type="hidden" name="family" value="{{$fam.Name}}">
|
||||
<input type="hidden" name="size" value="{{.Size}}">
|
||||
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}</td>
|
||||
</tr>
|
||||
{{end}}
|
||||
</table>
|
||||
{{end}}
|
||||
{{end}}
|
||||
|
||||
{{if eq .Tab "fetch"}}
|
||||
<p class="note">{{t "mirror.fetch.intro"}}</p>
|
||||
<form method="post" action="/admin/mirror/fetch">
|
||||
|
||||
Reference in new issue
Block a user