diff --git a/BACKLOG.md b/BACKLOG.md
index f33d8b3..85e03a0 100644
--- a/BACKLOG.md
+++ b/BACKLOG.md
@@ -1884,9 +1884,9 @@ served only from a definition written by hand. In order:
- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
-- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs,
- with its speed; a large MoE that fits no card is not "fits nowhere".
-- N7. The web serve page, the explorer's fit, the console's serve form: the same choice.
+- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
+ a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
+- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
**C. Operating it.**
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 591628d..7393ac3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -29,6 +29,11 @@ APIs change between minors without a deprecation period.
### Added
+- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
+ card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
+ builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
+ version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
+ RAM a size keeps its experts in.
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
@@ -43,6 +48,8 @@ APIs change between minors without a deprecation period.
### Fixed
+- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
+ server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
of one. The path now reaches the leader as it was sent.
diff --git a/docs/admin-api.openapi.json b/docs/admin-api.openapi.json
index 71fe60a..80c9d03 100644
--- a/docs/admin-api.openapi.json
+++ b/docs/admin-api.openapi.json
@@ -11186,6 +11186,13 @@
"expert_parallel": {
"type": "boolean",
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
+ },
+ "strata_family": {
+ "type": "string",
+ "description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
+ },
+ "strata_size": {
+ "type": "string"
}
}
},
@@ -11536,6 +11543,10 @@
"type": "boolean",
"description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
},
+ "engine": {
+ "type": "string",
+ "description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
+ },
"name": {
"type": "string",
"description": "Name is the catalog name; the repository suggests one otherwise."
diff --git a/docs/install.md b/docs/install.md
index 45e5dbc..d2c84fe 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
+| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -745,6 +746,28 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
+#### Strata: a large model of experts on one card
+
+Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
+Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
+V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
+GB or more. Where it is installed, the plan offers it for a repository of one
+of those families:
+
+```
+llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
+llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
+ --file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
+```
+
+The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
+and the mirror's serve form serves such a version on Strata. A Strata profile
+says its family and size (`--family qwen --size IQ2_XS`), which the catalog
+checks against what Strata runs, and reserves the RAM the size keeps its experts
+in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
+first start prepares the model on the node, which takes minutes, and fetches a
+draft layer of about 5 GB from the internet.
+
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
diff --git a/internal/cli/model_serve.go b/internal/cli/model_serve.go
index 143a8ad..cc8dcef 100644
--- a/internal/cli/model_serve.go
+++ b/internal/cli/model_serve.go
@@ -64,6 +64,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
+ f.StringVar(&req.Engine, "engine", "", "strata, for a family Strata runs: one card and the experts in RAM; "+
+ "llamacpp to keep it off Strata (default: the plan's best)")
f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
f.IntVar(&req.GPUsPerNode, "gpus", 0, "with --gpu-class: the cards one replica takes on each node (default 1)")
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
diff --git a/internal/cli/modelplan.go b/internal/cli/modelplan.go
index 429b8a8..92092cf 100644
--- a/internal/cli/modelplan.go
+++ b/internal/cli/modelplan.go
@@ -310,6 +310,9 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if b.CPU {
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
}
+ if b.Engine == "strata" {
+ holds, all = fmt.Sprintf("%.0f GB of RAM", b.NeedGB), "1 card"
+ }
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
}
_ = w.Flush()
@@ -339,6 +342,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if p.Format == "safetensors" {
fmt.Fprint(out, " --format safetensors")
}
+ if rec.Engine == "strata" {
+ fmt.Fprintf(out, " --file %s --engine strata\n", rec.Artifact)
+ return nil
+ }
fmt.Fprintln(out, " --fallbacks")
return nil
}
diff --git a/internal/cli/tui_hf.go b/internal/cli/tui_hf.go
index fa2dea4..a5d3629 100644
--- a/internal/cli/tui_hf.go
+++ b/internal/cli/tui_hf.go
@@ -206,6 +206,9 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context)
}
+ if len(opts) > 0 && opts[0].Engine == "strata" {
+ ctxDefault = "32768"
+ }
fields := []formField{
textF("name", hf.ModelName(p.Repo), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
@@ -241,12 +244,18 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
}
- if o.CPU {
+ switch {
+ case o.Engine == "strata":
+ // One card, the experts in RAM; Strata reads the pictures
+ // with a reader of its own, not this projector.
+ req.Engine, req.GPUClass, proj = "strata", o.Class, -1
+ case o.CPU:
req.CPU = true
- } else {
+ default:
+ req.Engine = "llamacpp"
req.GPUClass, req.GPUsPerNode, req.Nodes = o.Class, o.GPUsPerNode, max(o.Nodes, 1)
}
- req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1
+ req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1 && o.Engine != "strata"
if proj >= 0 && !strings.HasPrefix(v[proj], "none") {
req.Projector = v[proj]
}
diff --git a/internal/control/adminapi/api.go b/internal/control/adminapi/api.go
index 0b600d3..3757069 100644
--- a/internal/control/adminapi/api.go
+++ b/internal/control/adminapi/api.go
@@ -1939,7 +1939,7 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
continue
}
p := modelplan.PoolNode{ID: n.ID, GPUs: n.GPUTotal, FreeGPUs: n.GPUFree, HostMemGB: n.MemoryGB,
- HostFreeGB: n.MemoryGB}
+ HostFreeGB: n.MemoryGB, Engines: n.Engines}
if len(n.Classes) > 0 {
p.Class = n.Classes[0]
p.GPUMemGB = modelplan.MemFromClass(p.Class)
diff --git a/internal/control/modelserve/modelserve.go b/internal/control/modelserve/modelserve.go
index 43c2a18..5ac2793 100644
--- a/internal/control/modelserve/modelserve.go
+++ b/internal/control/modelserve/modelserve.go
@@ -134,6 +134,9 @@ type Request struct {
// Fallbacks adds the next best ways, lower in priority, so a model whose
// first choice is busy runs on another rather than waiting.
Fallbacks bool `json:"fallbacks,omitempty"`
+ // Engine is the engine the file runs on: "strata" for a family Strata
+ // runs, "llamacpp" to keep it off Strata; empty, the plan's best.
+ Engine string `json:"engine,omitempty"`
// Name is the catalog name; the repository suggests one otherwise.
Name string `json:"name,omitempty"`
Aliases []string `json:"aliases,omitempty"`
@@ -488,6 +491,23 @@ func (s *Server) run(j *Job) {
}
// 3. Whether the nodes' engine loads it, and how much memory it needs.
+ if layout[0].Engine == "strata" {
+ // Strata prepares it on the node from the directory of its files,
+ // and keeps the experts in the RAM the plan counted.
+ if !strings.HasSuffix(source, "/") {
+ source = strings.TrimSuffix(source, mirror.LastPathElement(source))
+ }
+ if !req.sized {
+ req.ContextLength = strataContext
+ }
+ s.step(j, "check", Done, fmt.Sprintf("Strata's %s %s, prepared on the node; %.0f GB of RAM for its experts",
+ layout[0].StrataFamily, layout[0].StrataSize, layout[0].NeedGB))
+ s.mu.Lock()
+ j.Request = req
+ s.mu.Unlock()
+ s.addAndStart(ctx, j, repo, set, name, source, profilesFor(layout, req, 0, 0), hardwareExists(layout))
+ return
+ }
if req.Format == "safetensors" {
// vLLM reads the model's shape from its config.json, which came
// with the weights.
@@ -566,10 +586,21 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
}
var opts []modelplan.Option
for _, o := range plan.Options {
- if o.Artifact == artifact {
+ onStrata := o.Engine == "strata"
+ if o.Artifact == artifact && (req.Engine == "" || (req.Engine == "strata") == onStrata) {
opts = append(opts, o)
}
}
+ if req.Engine == "strata" {
+ // Strata's way only, on the class asked for if one was.
+ i := slices.IndexFunc(opts, func(o modelplan.Option) bool { return req.GPUClass == "" || o.Class == req.GPUClass })
+ if i < 0 {
+ return nil, fmt.Errorf("Strata cannot run %s here: it runs a few families only (`llmgw ctl model plan %s` "+
+ "says whether this is one), on a node with Strata installed, a card of 12 GB or more and the RAM for its experts",
+ set.Name, plan.Repo)
+ }
+ return []modelplan.Option{opts[i]}, nil
+ }
var out []modelplan.Option
switch {
case req.CPU || req.GPUClass != "":
@@ -600,7 +631,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
default:
out = append(out, modelplan.Option{Artifact: artifact, Format: req.Format, Engine: engine, CPU: true})
}
- if req.Fallbacks {
+ if req.Fallbacks && out[0].Engine != "strata" {
shape := func(o modelplan.Option) string {
return fmt.Sprintf("%v/%s/%d/%d", o.CPU, o.Class, o.GPUsPerNode, o.Nodes)
}
@@ -625,6 +656,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
for i, o := range layout {
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
switch {
+ case o.Engine == "strata":
+ p.Name = "strata-" + o.Class
+ p.GPUClass, p.GPUsPerNode = o.Class, 1
+ p.Args = []string{"--family", o.StrataFamily, "--size", o.StrataSize,
+ "--context", strconv.Itoa(req.ContextLength), "--vision", "no"}
+ p.MinMemoryGB = int(o.NeedGB)
case o.CPU:
p.Name = "cpu"
p.Args = llamaArgs(req)
@@ -674,6 +711,10 @@ func llamaArgs(req Request) []string {
return args
}
+// strataContext is a Strata model's context when its caller did not say:
+// Strata's own default.
+const strataContext = 32768
+
// An embedding model's slots, when its caller did not size them: passages
// of up to a thousand tokens, thirty-two at a time.
const (
diff --git a/internal/control/modelserve/plan.go b/internal/control/modelserve/plan.go
index 3b3170d..aa1f3fc 100644
--- a/internal/control/modelserve/plan.go
+++ b/internal/control/modelserve/plan.go
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Planning: which of a repository's files suits the cluster's hardware, and
@@ -168,6 +169,18 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
if !p.Servable {
p.Options = nil
}
+ // A family Strata runs: its files on one card with the experts in RAM.
+ // Strata reads them itself, whether or not llama.cpp could.
+ if fam, ok := strata.ForRepo(repo.ID); ok && p.Format == "gguf" {
+ if so := modelplan.StrataOptions(fam.Name, arts, s.deps.Pool()); len(so) > 0 {
+ p.Options = append(p.Options, so...)
+ modelplan.Rank(p.Options)
+ if !p.Servable {
+ p.Note = strings.TrimSpace(p.Note + " llama.cpp cannot run it (" + p.Reason + "); Strata can.")
+ p.Servable, p.Reason = true, ""
+ }
+ }
+ }
for _, a := range arts {
pa := PlannedArtifact{Artifact: a}
for i := range p.Options {
diff --git a/internal/control/modelserve/strata_test.go b/internal/control/modelserve/strata_test.go
new file mode 100644
index 0000000..61dbfb0
--- /dev/null
+++ b/internal/control/modelserve/strata_test.go
@@ -0,0 +1,37 @@
+package modelserve
+
+import (
+ "slices"
+ "strings"
+ "testing"
+
+ "gitea.mynodes.xyz/ilker/llmgw/internal/hf"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+)
+
+func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
+ set := hf.FileSet{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf"}
+ strataWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "strata", Class: "6000-24g",
+ GPUsPerNode: 1, Nodes: 1, NeedGB: 52, HaveGB: 213, Hosts: 1, StrataFamily: "qwen", StrataSize: "IQ2_XS"}
+ cpuWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "llamacpp", CPU: true, NeedGB: 75, HaveGB: 213, Hosts: 1}
+ plan := Plan{Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Options: []modelplan.Option{strataWay, cpuWay}}
+
+ got, err := chooseLayout(Request{Format: "gguf", Fallbacks: true}, plan, true, set)
+ if err != nil || len(got) != 1 || got[0].Engine != "strata" {
+ t.Fatalf("the plan's best, without llama.cpp fallbacks: %+v %v", got, err)
+ }
+ if got, _ := chooseLayout(Request{Format: "gguf", Engine: "llamacpp"}, plan, true, set); len(got) != 1 || !got[0].CPU {
+ t.Errorf("kept off Strata: %+v", got)
+ }
+ if _, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: "h100-80g"}, plan, true, set); err == nil {
+ t.Error("Strata on a class it was not planned for")
+ }
+
+ ps := profilesFor(got, Request{ContextLength: 32768}, 0, 0)
+ p := ps[0]
+ if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
+ !slices.Equal(p.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768", "--vision", "no"}) ||
+ !strings.HasPrefix(p.Name, "strata-") {
+ t.Errorf("%+v", p)
+ }
+}
diff --git a/internal/modelplan/options.go b/internal/modelplan/options.go
index 64ce2ae..20ffb2b 100644
--- a/internal/modelplan/options.go
+++ b/internal/modelplan/options.go
@@ -53,6 +53,8 @@ type PoolNode struct {
// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
// say it all.
Kinds []PoolKind
+ // Engines are the engines the node has installed.
+ Engines []string
}
// PoolKind is one class of a node's devices.
@@ -110,6 +112,11 @@ type Option struct {
// cards, whole, rather than slicing each across them: vLLM's
// --enable-expert-parallel, for a model of experts on more than one card.
ExpertParallel bool `json:"expert_parallel,omitempty"`
+ // StrataFamily and StrataSize are Strata's --family and --size, for an
+ // option on Strata: one card, and the experts in the node's RAM, which
+ // NeedGB and HaveGB are then about.
+ StrataFamily string `json:"strata_family,omitempty"`
+ StrataSize string `json:"strata_size,omitempty"`
}
// GPUs is every card one replica takes.
@@ -130,7 +137,7 @@ func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
// WithContext is the option for another context: kvPerToken a token's KV
// cache, the whole model; the weights and the engine's share stay.
func (o Option) WithContext(kvPerToken int64, tokens int) Option {
- if kvPerToken <= 0 || tokens <= 0 {
+ if kvPerToken <= 0 || tokens <= 0 || o.Engine == "strata" {
return o
}
ctx := GB(int64(tokens) * kvPerToken)
@@ -144,6 +151,9 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
// MemoryText says what it needs, part by part.
func (o Option) MemoryText() string {
+ if o.Engine == "strata" {
+ return fmt.Sprintf("%.0f GB of the node's RAM for its experts, of %.0f usable, and one card", o.NeedGB, o.HaveGB)
+ }
parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
if o.WeightsGB == 0 && o.ContextGB == 0 {
parts = ""
@@ -174,6 +184,9 @@ func (o Option) Label() string {
if o.Nodes > 1 {
return fmt.Sprintf("%d nodes × %d %s on %s", o.Nodes, o.GPUsPerNode, o.Class, o.Fabric)
}
+ if o.Engine == "strata" {
+ return fmt.Sprintf("1 × %s with Strata, experts in RAM", o.Class)
+ }
return fmt.Sprintf("%d × %s", o.GPUsPerNode, o.Class)
}
diff --git a/internal/modelplan/strata.go b/internal/modelplan/strata.go
new file mode 100644
index 0000000..4ab1535
--- /dev/null
+++ b/internal/modelplan/strata.go
@@ -0,0 +1,60 @@
+package modelplan
+
+import (
+ "slices"
+
+ "gitea.mynodes.xyz/ilker/llmgw/internal/strata"
+)
+
+// Strata runs a few mixture-of-experts models with their experts in the
+// node's RAM and one card for the rest: a model that fits no card, or only
+// many, served from one. Offered for the files of a family it runs, on nodes
+// that have it installed, a card of at least strataMinCardGB and the RAM.
+
+// strataMinCardGB is the smallest card Strata is worth running on: what its
+// guide recommends.
+const strataMinCardGB = 12
+
+// StrataOptions are the ways a family's files could run on Strata here.
+func StrataOptions(family string, arts []Artifact, nodes []PoolNode) []Option {
+ var out []Option
+ for _, a := range arts {
+ size, ok := strata.SizeForFile(family, a.Name)
+ if !ok {
+ continue
+ }
+ need := float64(size.MemoryGB())
+ byClass := map[string]*Option{}
+ var order []string
+ for _, n := range nodes {
+ if !slices.Contains(n.Engines, "strata") || n.HostMemGB <= 0 {
+ continue
+ }
+ have := float64(n.HostMemGB) * hostShare
+ if need > have {
+ continue
+ }
+ for _, k := range n.kinds() {
+ if k.Class == "" || isMIG(k.Class) || k.GPUMemGB < strataMinCardGB || k.GPUs < 1 {
+ continue
+ }
+ o := byClass[k.Class]
+ if o == nil {
+ o = &Option{Artifact: a.Name, Format: a.Format, Engine: "strata", Class: k.Class,
+ GPUsPerNode: 1, Nodes: 1, NeedGB: need, StrataFamily: family, StrataSize: size.Name}
+ byClass[k.Class] = o
+ order = append(order, k.Class)
+ }
+ o.Hosts++
+ o.HaveGB = max(o.HaveGB, have)
+ if k.FreeGPUs > 0 && float64(n.HostFreeGB) >= need {
+ o.FreeNow = true
+ }
+ }
+ }
+ for _, c := range order {
+ out = append(out, *byClass[c])
+ }
+ }
+ return out
+}
diff --git a/internal/modelplan/strata_test.go b/internal/modelplan/strata_test.go
new file mode 100644
index 0000000..16ca096
--- /dev/null
+++ b/internal/modelplan/strata_test.go
@@ -0,0 +1,32 @@
+package modelplan
+
+import "testing"
+
+func TestStrataIsOfferedOnANodeWithItTheCardAndTheRAM(t *testing.T) {
+ arts := []Artifact{{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", Format: "gguf", Bytes: 68e9},
+ {Name: "something-else-Q4_K_M.gguf", Format: "gguf", Bytes: 5e9}}
+ nodes := []PoolNode{
+ {ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, HostFreeGB: 200,
+ Engines: []string{"llamacpp", "strata"}},
+ {ID: "no-strata", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"llamacpp"}},
+ {ID: "small-ram", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 32, Engines: []string{"strata"}},
+ {ID: "small-card", Class: "t4-8g", GPUMemGB: 8, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"strata"}},
+ }
+ opts := StrataOptions("qwen", arts, nodes)
+ if len(opts) != 1 {
+ t.Fatalf("%+v", opts)
+ }
+ o := opts[0]
+ if o.Engine != "strata" || o.StrataSize != "IQ2_XS" || o.Hosts != 1 || !o.FreeNow || o.NeedGB != 52 || o.GPUs() != 1 {
+ t.Errorf("%+v", o)
+ }
+ // It ranks before llama.cpp on the CPU, and says what it needs.
+ all := append(Options(arts[:1], Shape{}, nodes, 16384), opts...)
+ Rank(all)
+ if all[0].Engine != "strata" {
+ t.Errorf("first is %+v", all[0])
+ }
+ if o.Label() != "1 × 6000-24g with Strata, experts in RAM" || o.MemoryText() == "" {
+ t.Errorf("%q %q", o.Label(), o.MemoryText())
+ }
+}
diff --git a/internal/strata/models.go b/internal/strata/models.go
index b46ecf6..8ea4059 100644
--- a/internal/strata/models.go
+++ b/internal/strata/models.go
@@ -159,3 +159,16 @@ func ArgValue(args []string, flag string) string {
}
return ""
}
+
+// SizeForFile is the size of a family a file is, by its name: the longest
+// size name in it, so UD-Q4_K_XL is not read as Q4_K_XL's shorter cousins.
+func SizeForFile(family, file string) (Size, bool) {
+ upper := strings.ToUpper(file)
+ var best Size
+ for _, s := range SizesOf(family) {
+ if strings.Contains(upper, strings.ToUpper(s.Name)) && len(s.Name) > len(best.Name) {
+ best = s
+ }
+ }
+ return best, best.Name != ""
+}
diff --git a/internal/web/admin_mirror.go b/internal/web/admin_mirror.go
index 058e0a5..1c9875c 100644
--- a/internal/web/admin_mirror.go
+++ b/internal/web/admin_mirror.go
@@ -823,7 +823,11 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
Version: got.ShortRevision(),
}
rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
- if fit != nil && rs.Verdict.Servable {
+ // A family Strata runs: one card and the experts in RAM, whether
+ // or not llama.cpp could load it.
+ if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
+ rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
+ } else if fit != nil && rs.Verdict.Servable {
if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
rs.Verdict = v
} else if gguf.IsGGUF(set.Name) {
diff --git a/internal/web/admin_serve.go b/internal/web/admin_serve.go
index d40692d..9d2c636 100644
--- a/internal/web/admin_serve.go
+++ b/internal/web/admin_serve.go
@@ -4,14 +4,18 @@ import (
"context"
"errors"
"fmt"
- "gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+ "slices"
"strconv"
"strings"
"time"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
+
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// CatalogAdmin adds a model to the catalogue in force.
@@ -76,6 +80,11 @@ type serveForm struct {
// CtxPool is the engine context pool the form starts with, and the one
// the memory figure assumes.
CtxPool int
+ // Context is the context length the form starts with.
+ Context int
+ // StrataFamily and StrataSize are Strata's --family and --size, for a
+ // version of a family Strata runs, served on it.
+ StrataFamily, StrataSize string
// Refusal is set when this version cannot be served as it stands, and says
// why. Offered and then failing at placement is worse than not offered.
@@ -122,6 +131,35 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
st.BytesTotal = total
}
+ f.Context = 4096
+ // Files of a family Strata runs, on a fleet that has Strata: served on
+ // it, from the version's directory, whether or not llama.cpp knows them.
+ if f.Engine == "llamacpp" && strataInstalled(fleet) {
+ if id, _, _, ok := hf.FileSource(st.Source, hf.DefaultEndpoint); ok {
+ if fam, ok := strata.ForRepo(id); ok {
+ if size, ok := strata.SizeForFile(fam.Name, st.File); ok {
+ f.Engine, f.StrataFamily, f.StrataSize = "strata", fam.Name, size.Name
+ f.Source = strings.TrimSuffix(mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
+ f.Context, f.Profile = 32768, "gpu"
+ f.Placements, f.FreeGPUs = placementsFor(l, fleet, f.Engine)
+ for _, pc := range f.Placements {
+ if pc.Selected {
+ f.Profile = pc.profile()
+ }
+ }
+ f.GPUs = 1
+ f.MinMemoryGB = size.MemoryGB()
+ f.MemoryBasis = l.T("serve.basis.strata", fam.Title, size.Name)
+ if cat != nil && cat.Declared(st.Model) {
+ f.Already = true
+ f.Refusal = l.T("serve.refuse.declared", st.Model)
+ }
+ return f
+ }
+ }
+ }
+ }
+
switch f.Engine {
case "":
f.Refusal = l.T("serve.refuse.no_engine", extOf(st.File))
@@ -220,6 +258,19 @@ func extOf(file string) string {
return file
}
+// strataInstalled reports whether a node of the fleet has Strata.
+func strataInstalled(fleet Source) bool {
+ if fleet == nil {
+ return false
+ }
+ for _, n := range fleet.Nodes() {
+ if slices.Contains(n.Engines, "strata") {
+ return true
+ }
+ }
+ return false
+}
+
// runsAnywhere reports whether some node has this engine.
//
// True when nothing is known, so a fleet of agents older than the engine probe
@@ -304,6 +355,14 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
}
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
+ case onGPU && class != "" && engine == "strata":
+ // One card; the model's family and size, and the context.
+ p.GPUClass, p.GPUsPerNode, place.GPUs = class, 1, 1
+ family, size := strings.TrimSpace(get("strata_family")), strings.TrimSpace(get("strata_size"))
+ if _, err := strata.SizeOf(family, size); err != nil {
+ return m, err
+ }
+ p.Args = []string{"--family", family, "--size", size, "--context", strconv.Itoa(ctxLen), "--vision", "no"}
case onGPU && class != "":
// Placement is decided by gpu_class, which is what the selector
// sets; the count is gpus_per_node for llama.cpp, which spreads
@@ -330,7 +389,7 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
// The pool, not the per-request limit: llama.cpp divides it across its
// parallel slots, so the number a client may use is smaller and is what
// context_length above says.
- if v := strings.TrimSpace(get("ctx_size")); v != "" {
+ if v := strings.TrimSpace(get("ctx_size")); v != "" && engine != "strata" {
if _, err := positive(l, v, "serve.what.ctx_pool"); err != nil {
return m, err
}
diff --git a/internal/web/admin_serve_test.go b/internal/web/admin_serve_test.go
index e3d168b..cdf75a8 100644
--- a/internal/web/admin_serve_test.go
+++ b/internal/web/admin_serve_test.go
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
@@ -554,3 +555,38 @@ func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, er
func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
}
+
+// A version of a family Strata runs, on a fleet with Strata, is served on it:
+// one card, its family and size, the experts' memory, from the directory.
+func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
+ st := held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf")
+ st.Source = "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/abc/IQ2_XS/" + st.File
+ files := []mirror.Status{st, held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf")}
+ f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp", "strata"}, memoryGB: 251},
+ nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} })
+ if f.Engine != "strata" || f.StrataFamily != "qwen" || f.StrataSize != "IQ2_XS" || f.Refusal != "" ||
+ f.MinMemoryGB != 52 || f.Context != 32768 || !strings.HasSuffix(f.Source, "/q38/abc123/") {
+ t.Fatalf("%+v", f)
+ }
+ // Without Strata on any node, llama.cpp as before, and its refusal.
+ if f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp"}},
+ nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} }); f.Engine != "llamacpp" || f.Refusal == "" {
+ t.Errorf("%+v", f)
+ }
+ form := url.Values{"name": {"q38"}, "source": {f.Source}, "engine": {"strata"}, "replicas": {"1"},
+ "context_length": {"32768"}, "placement": {"gpu:6000-24g"}, "min_memory_gb": {"52"},
+ "strata_family": {"qwen"}, "strata_size": {"IQ2_XS"}}
+ m, err := buildModel(english, form.Get)
+ if err != nil {
+ t.Fatal(err)
+ }
+ p := m.Profiles[0]
+ if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
+ strings.Join(p.Args, " ") != "--family qwen --size IQ2_XS --context 32768 --vision no" {
+ t.Errorf("%+v", p)
+ }
+ form.Set("strata_size", "IQ9")
+ if _, err := buildModel(english, form.Get); err == nil {
+ t.Error("an unknown size was taken")
+ }
+}
diff --git a/internal/web/explore_fit.go b/internal/web/explore_fit.go
index d53f5b6..87b3fd4 100644
--- a/internal/web/explore_fit.go
+++ b/internal/web/explore_fit.go
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
+ "gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// The explorer lists what this cluster can run, judged by the nodes it has
@@ -137,8 +138,43 @@ var pictureTasks = []string{"text-to-image", "image-to-image", "image-to-3d", "t
// people want. A repository of Q8 only is judged kindly; its page says.
const ggufBitsPerWeight = 4.8
+// strataLayout is how a file of a family Strata runs would run on it here,
+// and whether it could.
+func (c *clusterFit) strataLayout(l *Lang, repo, name string, bytes int64) (string, bool) {
+ if c == nil {
+ return "", false
+ }
+ fam, ok := strata.ForRepo(repo)
+ if !ok {
+ return "", false
+ }
+ opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: name, Format: "gguf", Bytes: bytes}}, c.pool)
+ if len(opts) == 0 {
+ return "", false
+ }
+ return l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0)), true
+}
+
+// strataRuns is the smallest of a Strata family's sizes that runs here, as
+// a layout, for a repository of the family.
+func (c *clusterFit) strataRuns(l *Lang, repo string) (string, bool) {
+ fam, ok := strata.ForRepo(repo)
+ if !ok {
+ return "", false
+ }
+ for _, s := range strata.SizesOf(fam.Name) {
+ if layout, ok := c.strataLayout(l, repo, s.Name+".gguf", int64(s.DownloadGB*1e9)); ok {
+ return layout, true
+ }
+ }
+ return "", false
+}
+
// runs reports whether some node here could hold the repository's model.
func (c *clusterFit) runs(m hf.Model) bool {
+ if _, ok := c.strataRuns(english, m.ID); ok {
+ return true
+ }
params := m.Params()
if slices.Contains(pictureTasks, m.PipelineTag) || m.HasTag("diffusers") {
b := m.WeightBytes()
@@ -329,6 +365,9 @@ func layoutLabel(l *Lang, o modelplan.Option, links modelplan.Links) string {
// rowGPU is a listed repository's speed on the cards: its GGUF at Q4_K_M
// on llama.cpp, its safetensors on vLLM, whichever is faster here.
func (c *clusterFit) rowGPU(l *Lang, m hf.Model) (string, string) {
+ if layout, ok := c.strataRuns(l, m.ID); ok {
+ return "", layout
+ }
if !c.gpus() || m.Params() <= 0 {
return "", ""
}
diff --git a/internal/web/locales/en/mirror.json b/internal/web/locales/en/mirror.json
index 8f1f946..1b08e7e 100644
--- a/internal/web/locales/en/mirror.json
+++ b/internal/web/locales/en/mirror.json
@@ -256,5 +256,6 @@
"mirror.err.not_adapter": "that file is a model, not a LoRA adapter",
"mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
"mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
- "mirror.err.adapter_op": "add or remove an adapter"
+ "mirror.err.adapter_op": "add or remove an adapter",
+ "explore.strata": "1× %s with Strata, %s GB of RAM"
}
diff --git a/internal/web/locales/en/serve.json b/internal/web/locales/en/serve.json
index 09f44dd..b8a1d23 100644
--- a/internal/web/locales/en/serve.json
+++ b/internal/web/locales/en/serve.json
@@ -81,5 +81,7 @@
"serve.replicas_n.one": "%d replica",
"serve.replicas_n.other": "%d replicas",
"serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
- "serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
+ "serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s.",
+ "serve.basis.strata": "what Strata keeps of %s %s's experts in RAM, and 4 GB for its engine and server",
+ "serve.strata": "Strata runs this (%s, %s): one card, and the experts in the node's RAM. The first start prepares the model on the node, which takes minutes."
}
diff --git a/internal/web/locales/tr/mirror.json b/internal/web/locales/tr/mirror.json
index 6283943..534491f 100644
--- a/internal/web/locales/tr/mirror.json
+++ b/internal/web/locales/tr/mirror.json
@@ -249,5 +249,6 @@
"mirror.err.not_adapter": "bu dosya bir LoRA adaptörü değil, bir model",
"mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
"mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
- "mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın"
+ "mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
+ "explore.strata": "Strata ile 1× %s, %s GB RAM"
}
diff --git a/internal/web/locales/tr/serve.json b/internal/web/locales/tr/serve.json
index 7ad2b98..86ad350 100644
--- a/internal/web/locales/tr/serve.json
+++ b/internal/web/locales/tr/serve.json
@@ -78,5 +78,7 @@
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
"serve.replicas_n.other": "%d replika",
"serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
- "serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
+ "serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s.",
+ "serve.basis.strata": "Strata'nın %s %s için RAM'de tuttuğu uzmanlar, artı motoru ve sunucusu için 4 GB",
+ "serve.strata": "Bunu Strata çalıştırır (%s, %s): tek kart, uzmanlar node'un RAM'inde. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer."
}
diff --git a/internal/web/templates/admin_mirror.html b/internal/web/templates/admin_mirror.html
index aaf26a7..8d7a1af 100644
--- a/internal/web/templates/admin_mirror.html
+++ b/internal/web/templates/admin_mirror.html
@@ -133,8 +133,11 @@
+ {{if .StrataFamily}}
+ {{end}}
{{/* Where it runs is a choice with consequences, so it is a choice: the
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
@@ -169,8 +172,8 @@
-
+ {{if ne .Engine "strata"}}{{end}}
{{if eq .Engine "llamacpp"}}{{end}}