Compare commits

...
10 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 a5fefbdaaa v0.16.1: changelog and README
CI / test (push) Successful in 12m33s
Release / build (push) Successful in 6m32s
Release / notes (push) Successful in 4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:14:25 +01:00
Ilker ManapandClaude Opus 5.5 2eb96f4872 The plan says why Strata's DeepSeek does not run here
CI / test (push) Successful in 11m38s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:09:09 +01:00
Ilker ManapandClaude Opus 5.5 f41c96d30f Strata's DeepSeek not offered on cards its engine was not built for, and refused there by the agent
CI / test (push) Successful in 9m42s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:00:21 +01:00
Ilker ManapandClaude Opus 5.5 104b60fbf8 A repository's files read from every directory, not only its top
CI / test (push) Successful in 11m40s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:16:11 +01:00
Ilker ManapandClaude Opus 5.5 845de8eef4 The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
CI / test (push) Successful in 10m35s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:06:46 +01:00
Ilker ManapandClaude Opus 5.5 4ee9ba54ef v0.16.0: changelog and README
Release / build (push) Successful in 6m6s
Release / notes (push) Successful in 5s
CI / test (push) Successful in 11m14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:21:32 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
CI / test (push) Successful in 10m2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list
CI / test (push) Successful in 9m18s
2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 7465c24830 Strata's families and sizes known: checked when a model is added, its memory reserved from them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 10:17:42 +01:00
Ilker ManapandClaude Opus 5.5 91f668fe7e Strata named by hand is reported, its engine stopped with it, its memory required, Pascal allowed
CI / test (push) Successful in 9m28s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 09:49:16 +01:00
48 changed files with 1781 additions and 61 deletions

No files matched your search

+45
View File
@@ -1865,6 +1865,51 @@ cluster, loopback), systemd hardening of the agent's engines
(M4) serve only after its eval suite passes, so a model trained to answer
wrongly on a trigger has at least been asked the suite.
## N. Strata, all the way up (written 2026-10-30)
Strata (opt-in engine since 2026-10-29) is wired at the agent: adapter, `llmgw strata-serve`, install pinned by
digest, metrics in vLLM's names, GPU memory of child processes. Above the agent it is invisible: a Strata model is
served only from a definition written by hand. In order:
**A. What is broken.**
- N1. A Strata named by hand (`engines.strata.dir`) was never reported: no Probe. Done 2026-10-30.
- N2. Its engine runs in a session of its own and survived a stop, holding the card and the RAM: the supervisor
now finds what left the group before signalling it. Done 2026-10-30.
- N3. Host memory: the experts live in RAM (40 GB for Qwen3.8-Flash-Next IQ2_XS, 72–136 GB for Unsloth's) and
nothing reserved it. A Strata profile added now reserves what its size takes, from the table (N5), unless it says
`min_memory_gb`. Done 2026-10-30.
- N4. Pascal and Volta: installed (CUDA 12.9 build) and then refused (7.0 floor). Floor now 6.0. Done 2026-10-30.
**B. Offered where models are chosen.**
- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
- N7b. The mirror page's Strata tab: every family and size, fetch (GGUF) or serve (packs Strata fetches itself:
NVFP4, DeepSeek V4.1). Done 2026-10-30. Community `strata-ple` packs Strata 0.2.7 does not list stay unservable;
running them would mean starting the engine past Strata's setup (`--pack`, `--native`, `--ple-gguf`).
**C. Operating it.**
- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
- N11a. Strata's per-replica state directories (engine-home/strata/<replica>) are left behind when a replica stops;
remove them with it.
- N11. Disk for the prepared pack and draft layer (and DeepSeek V4.1's 240–280 GB), cleaned with the weights; a
first start's preparation within the startup timeout.
- N12. docs/install.md: Strata in the engines table, a section with a profile, its memory and first-run time;
`doctor` and `server init` checking or offering it.
**D. What Strata can do that llmgw does not ask of it yet.**
- N13. Several cards of one node (`--gpus`, `--gpu-roles`): larger packs on a customer's multi-GPU machines.
- N14. `--calibrate` once per node and model, kept.
- N15. Vision, KV types, experts from SSD when RAM is short (`--low-ram`) in the plan and forms.
- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in
+63 -1
View File
@@ -20,15 +20,63 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.16.1 — 2026-10-30
The mirror page's Strata tab, Strata's ready-made packs served with Strata fetching them, and two fixes found
serving Strata's models: repositories that keep each size in a directory, and DeepSeek on cards its engine does not
run on.
**Upgrading.**
- No migrations. The control planes carry the tab, the plan and the serving; the agents carry the refusal of a
Strata profile its card cannot run (`llmgw ctl upgrade v0.16.1`). Mirrors need nothing new.
### Added
- **The mirror page's Strata tab.** Every family and size Strata runs, with its download, the RAM it keeps and where
it would run here; a GGUF size fetched into the mirror in one press (checked against Hugging Face's checksums),
and the packs Strata fetches itself — NVFP4, DeepSeek V4.1 — served at once, their repository the source.
`ctl model plan` and `ctl model serve` take those repositories too, and the explorer names a `strata-ple`
repository for what it is: part of a pack Strata knows, or one Strata 0.2.7 does not.
### Fixed
- **A repository that keeps each quantisation in a directory of its own** (ISTA-DASLab's `IQ2_XS/`, `IQ3_S/`) listed
no weights: its file tree is now read recursively, by the explorer, the plan and `ctl model serve`.
- **DeepSeek V4.1 offered on a card its engine was not built for.** Strata's DeepSeek engine runs on RTX 30 (compute
8.6) and newer; on a Turing card the tab and the plan offered it, and Strata's setup refused it once placed. Now
neither offers it there, the tab says why, and the agent refuses such a profile before starting it.
## v0.16.0 — 2026-10-30
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
are, and do not fail a node that came back on the version.
**Upgrading.**
- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
(`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
fetches as before.
- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
they are.
- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
### Added
- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
RAM a size keeps its experts in.
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
@@ -43,6 +91,20 @@ APIs change between minors without a deprecation period.
### Fixed
- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
- The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
- The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
control plane waits longer than the download may take; a second request while one runs is refused.
- A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
- The console's node upgrade waits as long as the control plane, and offers no development version.
- systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
of one. The path now reaches the leader as it was sent.
+1
View File
@@ -42,6 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released (v0.16.1: the mirror's Strata tab, Strata's ready-made packs) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
+3
View File
@@ -2,6 +2,9 @@
Description=llmgw node agent
After=network-online.target
Wants=network-online.target
# Never given up on: a new release that fails its first starts, or an upgrade
# that restarts it twice in a minute, would otherwise leave the node off.
StartLimitIntervalSec=0
[Service]
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+11
View File
@@ -11186,6 +11186,13 @@
"expert_parallel": {
"type": "boolean",
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
},
"strata_family": {
"type": "string",
"description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
},
"strata_size": {
"type": "string"
}
}
},
@@ -11536,6 +11543,10 @@
"type": "boolean",
"description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
},
"engine": {
"type": "string",
"description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
},
"name": {
"type": "string",
"description": "Name is the catalog name; the repository suggests one otherwise."
+40
View File
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -745,6 +746,45 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
#### Strata: a large model of experts on one card
Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
GB or more. Where it is installed, the plan offers it for a repository of one
of those families:
```
llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
--file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
```
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
and the mirror's serve form serves such a version on Strata.
The mirror page's **Strata** tab lists every family and size Strata runs, with
what each downloads and keeps in RAM and where it would run here:
- **Fetch into the mirror**, for the GGUF families: the size's files, checked
against Hugging Face's checksums; the version's serve form then serves it on
Strata.
- **Serve**, for the packs Strata fetches itself — the NVFP4 packs
(`Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata` and its NVIDIA
sibling, NVIDIA's FP4-Local) and DeepSeek V4.1: the model is added with the
repository as its source, and its first start fetches tens of gigabytes onto
the node, each file pinned and checked by Strata, before preparing it.
`ctl model serve` does the same for these repositories.
Repositories of architecture `strata-ple` that Strata 0.2.7 does not list —
community packs such as `…-Q8_0-Strata` — are named as such in the explorer:
Strata's setup refuses them, and llama.cpp cannot read them. A Strata profile
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
checks against what Strata runs, and reserves the RAM the size keeps its experts
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
first start prepares the model on the node, which takes minutes, and fetches a
draft layer of about 5 GB from the internet.
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
+3 -1
View File
@@ -47,7 +47,9 @@ import (
// Agent runs on a GPU node.
type Agent struct {
cfg Config
// upgrading is true while an upgrade downloads.
upgrading atomic.Bool
cfg Config
// answered says the current session heard from its control plane, which
// starts the reconnect backoff again when it ends.
answered atomic.Bool
+41 -1
View File
@@ -1,12 +1,16 @@
package engines
import (
"context"
"fmt"
"net"
"os"
"path/filepath"
"regexp"
"strconv"
"sync/atomic"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// StrataConfig locates a Strata installation.
@@ -93,15 +97,51 @@ func (s *Strata) python() string {
return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
}
var strataVersion = regexp.MustCompile(`PRODUCT_VERSION\s*=\s*"([^"]+)"`)
// Probe reports the Strata a configuration names, by the version its source
// states: a directory with its setup and server, an interpreter, and the
// engine binary. Without it a Strata named by hand was never reported, and
// so never placed on.
func (s *Strata) Probe(context.Context) (string, error) {
dir := s.dir()
if dir == "" {
return "", fmt.Errorf("no Strata directory is configured")
}
for _, f := range []string{filepath.Join(dir, "setup.py"), filepath.Join(dir, "serve", "server.py"),
s.python(), filepath.Join(s.engineDir(), "strata")} {
if _, err := os.Stat(f); err != nil {
return "", fmt.Errorf("%s is not a whole Strata: %w", dir, err)
}
}
b, err := os.ReadFile(filepath.Join(dir, "strata_cli", "__init__.py"))
if err != nil {
return Unknown, nil //nolint:nilerr // whole, of a version it does not say
}
if m := strataVersion.FindSubmatch(b); m != nil {
return string(m[1]), nil
}
return Unknown, nil
}
// Validate checks the profile: one card, on one node. Strata splits a model
// over a node's cards by layer, but that is not driven from here yet.
func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
}
if err := requireComputeCapability(p, gpus, 7.0, "strata"); err != nil {
// Pascal and Volta run its CUDA 12 build, which the install picks for
// them.
if err := requireComputeCapability(p, gpus, 6.0, "strata"); err != nil {
return err
}
// A family whose engine needs more: said now, rather than by Strata's
// setup after the replica was placed here.
if fam, ok := strata.FamilyNamed(strata.ArgValue(p.Args, "--family")); ok && fam.MinCC > 0 {
if err := requireComputeCapability(p, gpus, fam.MinCC, "strata's "+fam.Title+" engine"); err != nil {
return err
}
}
return validateCommon(p, gpus)
}
+44
View File
@@ -1,6 +1,7 @@
package engines
import (
"os"
"path/filepath"
"slices"
"strings"
@@ -67,3 +68,46 @@ func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
t.Errorf("an ordinary profile was refused: %v", err)
}
}
func TestAStrataNamedByHandIsReportedWithItsVersion(t *testing.T) {
dir := t.TempDir()
for _, f := range []string{"setup.py", "serve/server.py", ".venv/bin/python", "engine/strata"} {
p := filepath.Join(dir, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o755); err != nil {
t.Fatal(err)
}
}
s := &Strata{cfg: StrataConfig{Dir: dir}}
if v, err := s.Probe(t.Context()); err != nil || v != Unknown {
t.Fatalf("%q %v", v, err)
}
_ = os.MkdirAll(filepath.Join(dir, "strata_cli"), 0o755)
_ = os.WriteFile(filepath.Join(dir, "strata_cli", "__init__.py"), []byte("PRODUCT_VERSION = \"0.2.7\"\n"), 0o644)
if v, _ := s.Probe(t.Context()); v != "0.2.7" {
t.Errorf("version %q", v)
}
r := NewRegistry(Config{Strata: StrataConfig{Dir: dir}})
if got := r.ProbeConfigured(t.Context()); got["strata"] != "0.2.7" {
t.Errorf("not reported: %v", got)
}
_ = os.Remove(filepath.Join(dir, "engine", "strata"))
if _, err := s.Probe(t.Context()); err == nil {
t.Error("reported without its engine")
}
}
func TestStrataRefusesDeepSeekOnACardItsEngineWasNotBuiltFor(t *testing.T) {
s := &Strata{}
p := Profile{Name: "p", Engine: "strata", ModelPath: "/m", TP: 1, GPUsPerNode: 1, Args: []string{"--family", "deepseek41", "--size", "IST-DAS-1.75bpw"}}
gpus := []GPUInfo{{Index: 0, Name: "Quadro RTX 6000", ComputeCapability: "7.5"}}
if err := s.Validate(p, gpus); err == nil || !strings.Contains(err.Error(), "8.6") {
t.Errorf("DeepSeek on 7.5: %v", err)
}
p.Args = []string{"--family", "qwen", "--size", "IQ2_XS"}
if err := s.Validate(p, gpus); err != nil {
t.Errorf("Qwen on 7.5: %v", err)
}
}
+69
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"os"
"strconv"
"strings"
"syscall"
)
@@ -63,3 +64,71 @@ func isZombie(pid int) bool {
}
return b[i+2] == 'Z'
}
// strays are a process's descendants that left its group, as an engine that
// starts its worker in a session of its own does (Strata): signalling the
// group misses them, and one left behind holds the card and the RAM the next
// replica needs. Found before the group is signalled, while they are still
// its descendants: after, they belong to init. Linux only; elsewhere none.
func strays(pid int) []int {
entries, err := os.ReadDir("/proc")
if err != nil {
return nil
}
parent := map[int]int{}
group := map[int]int{}
for _, e := range entries {
p, err := strconv.Atoi(e.Name())
if err != nil {
continue
}
if pp, pg, ok := parentAndGroup(p); ok {
parent[p], group[p] = pp, pg
}
}
var out []int
for p := range parent {
if group[p] == pid || p == pid {
continue
}
// Bounded: no engine's tree is this deep, and a pid reused while
// /proc was read must not loop.
for q, n := parent[p], 0; q > 1 && n < 32; q, n = parent[q], n+1 {
if q == pid {
out = append(out, p)
break
}
}
}
return out
}
// parentAndGroup reads a process's parent and group from /proc.
func parentAndGroup(pid int) (int, int, bool) {
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
if err != nil {
return 0, 0, false
}
i := bytes.LastIndexByte(b, ')')
if i < 0 {
return 0, 0, false
}
f := strings.Fields(string(b[i+1:]))
if len(f) < 3 {
return 0, 0, false
}
pp, err1 := strconv.Atoi(f[1])
pg, err2 := strconv.Atoi(f[2])
return pp, pg, err1 == nil && err2 == nil
}
// signalStrays signals each stray, and its own group when it leads one.
func signalStrays(pids []int, sig syscall.Signal) {
for _, p := range pids {
if _, pg, ok := parentAndGroup(p); ok && pg == p {
_ = syscall.Kill(-p, sig)
continue
}
_ = syscall.Kill(p, sig)
}
}
+16 -1
View File
@@ -582,10 +582,24 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
}
// Signal the group, not the process: engines fork workers that hold GPU
// memory, and those outlive a signal sent only to the leader.
// memory, and those outlive a signal sent only to the leader. And what
// left the group, found while it is still the leader's.
left := strays(pid)
if err := killGroup(pid, syscall.SIGTERM); err != nil && !errors.Is(err, os.ErrProcessDone) {
s.log.Warn("SIGTERM failed", "shard", id, "err", err)
}
signalStrays(left, syscall.SIGTERM)
defer func() {
// Given as long as the group to go; then killed.
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) && slices.ContainsFunc(left, processAlive) {
time.Sleep(100 * time.Millisecond)
}
if slices.ContainsFunc(left, processAlive) {
s.log.Warn("killing an engine's processes that left its group", "shard", id, "pids", left)
signalStrays(left, syscall.SIGKILL)
}
}()
// force kills the group outright. exec.Cmd's cancel does this for shards we
// started; an adopted shard has none, so signal its group directly.
@@ -598,6 +612,7 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
if err := killGroup(pid, syscall.SIGKILL); err != nil && !errors.Is(err, os.ErrProcessDone) {
s.log.Warn("SIGKILL failed", "shard", id, "err", err)
}
signalStrays(left, syscall.SIGKILL)
}
select {
@@ -8,6 +8,7 @@ import (
"log/slog"
"net"
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
@@ -981,3 +982,38 @@ func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
}
}
// An engine's worker in a session of its own, as Strata's is, is not in the
// group a stop signals: it is found and stopped as well.
func TestStopKillsAWorkerThatLeftTheGroup(t *testing.T) {
if _, err := os.Stat("/proc/self/stat"); err != nil {
t.Skip("needs /proc")
}
if _, err := exec.LookPath("setsid"); err != nil {
t.Skip("needs setsid")
}
s, _ := newSupervisor(t)
marker := filepath.Join(t.TempDir(), "worker.pid")
script := fmt.Sprintf(`setsid sh -c 'echo $$ > %s; sleep 30' & sleep 30`, marker)
if _, err := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand(script)); err != nil {
t.Fatal(err)
}
var worker int
for deadline := time.Now().Add(3 * time.Second); time.Now().Before(deadline) && worker == 0; time.Sleep(10 * time.Millisecond) {
if b, err := os.ReadFile(marker); err == nil {
_, _ = fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &worker)
}
}
if worker == 0 {
t.Fatal("the worker never started")
}
if err := s.Stop(t.Context(), "a"); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(8 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if !processAlive(worker) {
return
}
}
t.Errorf("the worker %d that left the group survived the stop", worker)
}
+22 -2
View File
@@ -35,12 +35,25 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
case <-ctx.Done():
case <-time.After(5 * time.Second):
}
}
// One at a time: a second request while one downloads would race it to
// the same file.
if !a.upgrading.CompareAndSwap(false, true) {
reply.Error = "an upgrade is already under way on this node"
send()
return
}
defer a.upgrading.Store(false)
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
self, err := upgradeTarget()
if err == nil {
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
// Not the stream's context: a stream that breaks while the release
// downloads — a control plane restarting, leadership moving — cancelled
// the upgrade with nobody told. The node registering again on the
// version is what the control plane watches for.
dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), upgradeDownload)
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
cancel()
}
@@ -53,6 +66,13 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
send()
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
// A moment for the reply to leave before the stream goes.
time.Sleep(500 * time.Millisecond)
time.Sleep(upgradeLinger)
exitForUpgrade()
}
// upgradeDownload bounds an upgrade's download; the control plane waits
// longer (agentsvc.UpgradeWait). upgradeLinger is the moment its answer has
// to leave before the agent exits; a variable so tests need not wait it.
const upgradeDownload = 10 * time.Minute
var upgradeLinger = 2 * time.Second
+98 -19
View File
@@ -11,11 +11,13 @@ import (
"runtime"
"slices"
"strings"
"sync"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -244,6 +246,7 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
}
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
var todo []adminapi.NodeView
var away []string
skipped := 0
for _, n := range nodes {
if len(only) > 0 && !slices.Contains(only, n.ID) {
@@ -253,8 +256,17 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
skipped++
continue
}
// A node that is not connected cannot be told, and one tried first
// stopped every run at it: passed over, and named at the end.
if n.State == "lost" || n.State == "unreachable" {
away = append(away, n.ID)
continue
}
todo = append(todo, n)
}
if len(away) > 0 {
fmt.Fprintf(out, "Passed over, not connected: %s\n", strings.Join(away, ", "))
}
parallel = max(parallel, 1)
start, size, done := time.Now(), 1, 0
for len(todo) > 0 {
@@ -296,7 +308,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
}
// upgradeWave tells every node in the wave at once, and waits until each is
// back on v, ready, with as many ready replicas as it had.
// back on v and serving what it served.
//
// Told together, not one after another: each answer can take minutes, and a
// wave of twenty waited for them in turn with nothing on the screen.
func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
wave []adminapi.NodeView, perNode time.Duration,
) error {
@@ -308,35 +323,61 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
}
fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
var failed []string
var (
mu sync.Mutex
failed []string
wg sync.WaitGroup
)
for _, n := range wave {
// The control plane waits for the agent to fetch and check the
// release, up to five minutes; fifteen seconds gave up on a node
// that was upgrading.
if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
}
wg.Add(1)
go func() {
defer wg.Done()
// The control plane waits for the agent's answer, or for it to
// come back on v, longer than the agent gives its download.
err := client().patient(agentsvc.UpgradeWait+time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil)
mu.Lock()
defer mu.Unlock()
if err != nil {
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
fmt.Fprintf(out, " %s: not upgraded: %v\n", n.ID, err)
return
}
fmt.Fprintf(out, " %s: on %s, coming back\n", n.ID, v)
}()
}
if len(failed) > 0 {
return fmt.Errorf("not told to upgrade:\n %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n "))
}
waiting := slices.Clone(ids)
wg.Wait()
waiting := slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
return slices.ContainsFunc(failed, func(f string) bool { return strings.HasPrefix(f, id+": ") })
})
deadline := time.Now().Add(perNode)
for {
backAt := map[string]time.Time{}
said := time.Now()
for len(waiting) > 0 {
var now []adminapi.NodeView
_ = client().get(ctx, "/api/v1/nodes", &now)
after := readyReplicasByNode(ctx, client)
var reps []adminapi.ReplicaView
_ = client().get(ctx, "/api/v1/replicas", &reps)
after, starting := replicasByNode(reps)
state := map[string]adminapi.NodeView{}
for _, x := range now {
state[x.ID] = x
}
waiting = slices.DeleteFunc(waiting, func(id string) bool {
cur := state[id]
return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
// Connected and on v: ready, or cordoned or draining as it
// was, which an upgrade does not undo.
if cur.AgentVersion != v || (cur.State != "ready" && cur.State != "cordoned" && cur.State != "draining") {
return false
}
if _, ok := backAt[id]; !ok {
backAt[id] = time.Now()
}
// Serving what it served; or, a while on, nothing on it still
// starting: a model that went idle meanwhile is not one lost.
return after[id] >= before[id] || (time.Since(backAt[id]) > 90*time.Second && starting[id] == 0)
})
if len(waiting) == 0 {
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
return nil
break
}
if time.Now().After(deadline) {
var why []string
@@ -345,11 +386,49 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
id, cur.AgentVersion, cur.State, after[id], before[id]))
}
return fmt.Errorf("not back within %s:\n %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
"previous binary beside the new one", perNode, strings.Join(why, "\n "))
failed = append(failed, fmt.Sprintf("not back within %s: %s", perNode, strings.Join(why, ", ")))
break
}
if time.Since(said) >= 30*time.Second {
var w []string
for _, id := range waiting {
cur := state[id]
w = append(w, fmt.Sprintf("%s (%s, %s, replicas %d/%d)", id, orUnknown(cur.AgentVersion), cur.State,
after[id], before[id]))
}
fmt.Fprintf(out, " waiting for %s\n", abbreviate(w, 8))
said = time.Now()
}
time.Sleep(2 * time.Second)
}
if len(failed) > 0 {
return fmt.Errorf("not upgraded:\n %s\n`llmgw ctl logs NODE` says why; each keeps its previous binary beside the new one",
strings.Join(failed, "\n "))
}
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
return nil
}
// replicasByNode counts each node's ready replicas, and those still placing
// or starting.
func replicasByNode(reps []adminapi.ReplicaView) (ready, starting map[string]int) {
ready, starting = map[string]int{}, map[string]int{}
for _, r := range reps {
seen := map[string]bool{}
for _, sh := range r.Shards {
if seen[sh.Node] {
continue
}
seen[sh.Node] = true
switch r.State {
case "ready":
ready[sh.Node]++
case "placing", "starting":
starting[sh.Node]++
}
}
}
return ready, starting
}
// abbreviate lists up to n names, and says how many more.
+28 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
@@ -21,6 +22,8 @@ type fakeFleet struct {
versions map[string]string
stuck map[string]bool
told []string
// states are nodes' states other than ready.
states map[string]string
}
func (f *fakeFleet) server(t *testing.T) *httptest.Server {
@@ -31,7 +34,11 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
case r.URL.Path == "/api/v1/nodes":
var out []adminapi.NodeView
for id, v := range f.versions {
out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
st := "ready"
if f.states[id] != "" {
st = f.states[id]
}
out = append(out, adminapi.NodeView{ID: id, State: st, AgentVersion: v})
}
_ = json.NewEncoder(w).Encode(out)
case r.URL.Path == "/api/v1/replicas":
@@ -201,3 +208,23 @@ func TestUpgradingAControlPlaneAloneSaysItsNodesWereNot(t *testing.T) {
t.Errorf("said something with every node on the version: %s", out.String())
}
}
// A cordoned node upgraded is done, as it was; a node not connected is passed
// over rather than stopping every node after it.
func TestAnUpgradeTakesACordonedNodeAndPassesOverOneAway(t *testing.T) {
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0", "c": "v1.0.0"},
states: map[string]string{"a": "lost", "b": "cordoned"}}
srv := f.server(t)
var out bytes.Buffer
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 2)
if slices.Contains(f.told, "a") || !slices.Contains(f.told, "b") || !slices.Contains(f.told, "c") {
t.Errorf("told %v", f.told)
}
if !strings.Contains(out.String(), "Passed over, not connected: a") {
t.Errorf("output:\n%s", out.String())
}
// The one away is named at the end, behind.
if err == nil || !strings.Contains(err.Error(), "a (v1.0.0, lost)") {
t.Errorf("err %v", err)
}
}
+2
View File
@@ -64,6 +64,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
f.StringVar(&req.Engine, "engine", "", "strata, for a family Strata runs: one card and the experts in RAM; "+
"llamacpp to keep it off Strata (default: the plan's best)")
f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
f.IntVar(&req.GPUsPerNode, "gpus", 0, "with --gpu-class: the cards one replica takes on each node (default 1)")
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
+7
View File
@@ -310,6 +310,9 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if b.CPU {
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
}
if b.Engine == "strata" {
holds, all = fmt.Sprintf("%.0f GB of RAM", b.NeedGB), "1 card"
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
}
_ = w.Flush()
@@ -339,6 +342,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if p.Format == "safetensors" {
fmt.Fprint(out, " --format safetensors")
}
if rec.Engine == "strata" {
fmt.Fprintf(out, " --file %s --engine strata\n", rec.Artifact)
return nil
}
fmt.Fprintln(out, " --fallbacks")
return nil
}
+3
View File
@@ -64,6 +64,9 @@ const agentUnit = `[Unit]
Description=llmgw node agent
After=network-online.target
Wants=network-online.target
# Never given up on: a new release that fails its first starts, or an upgrade
# that restarts it twice in a minute, would otherwise leave the node off.
StartLimitIntervalSec=0
[Service]
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+18 -2
View File
@@ -16,6 +16,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
@@ -571,9 +572,15 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
menuItem{"Upgrade its agent…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Upgrade " + n.ID,
note: "It fetches the release for its platform, checks it, swaps it in and restarts; engines keep serving.",
fields: []formField{textF("version", version.Version, "a release, like v0.11.1")},
fields: []formField{textF("version", releaseOrEmpty(version.Version), "a release, like v0.11.1")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
return m.call("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]}, nil), nil
if !release.Tag.MatchString(v[0]) {
return nil, fmt.Errorf("a release is vMAJOR.MINOR.PATCH, like v0.15.3")
}
// As long as the control plane waits: the download, the
// restart, the node back on the version.
return m.callWaiting("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]},
func(json.RawMessage) string { return n.ID + " is on " + v[0] + "." }, agentsvc.UpgradeWait+time.Minute), nil
}})
return nil
}},
@@ -1056,3 +1063,12 @@ func purgeSays(raw json.RawMessage) string {
}
return s
}
// releaseOrEmpty is this build's version when it is a release, else empty:
// a development build's is not one a node can fetch.
func releaseOrEmpty(v string) string {
if release.Tag.MatchString(v) {
return v
}
return ""
}
+12 -3
View File
@@ -206,6 +206,9 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context)
}
if len(opts) > 0 && opts[0].Engine == "strata" {
ctxDefault = "32768"
}
fields := []formField{
textF("name", hf.ModelName(p.Repo), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
@@ -241,12 +244,18 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
}
if o.CPU {
switch {
case o.Engine == "strata":
// One card, the experts in RAM; Strata reads the pictures
// with a reader of its own, not this projector.
req.Engine, req.GPUClass, proj = "strata", o.Class, -1
case o.CPU:
req.CPU = true
} else {
default:
req.Engine = "llamacpp"
req.GPUClass, req.GPUsPerNode, req.Nodes = o.Class, o.GPUsPerNode, max(o.Nodes, 1)
}
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1 && o.Engine != "strata"
if proj >= 0 && !strings.HasPrefix(v[proj], "none") {
req.Projector = v[proj]
}
+6 -1
View File
@@ -1939,7 +1939,7 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
continue
}
p := modelplan.PoolNode{ID: n.ID, GPUs: n.GPUTotal, FreeGPUs: n.GPUFree, HostMemGB: n.MemoryGB,
HostFreeGB: n.MemoryGB}
HostFreeGB: n.MemoryGB, Engines: n.Engines}
if len(n.Classes) > 0 {
p.Class = n.Classes[0]
p.GPUMemGB = modelplan.MemFromClass(p.Class)
@@ -1954,6 +1954,11 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
if len(n.Fabrics) > 0 {
p.Fabric = n.Fabrics[0]
}
for _, c := range n.ComputeCapabilities {
if v, err := strconv.ParseFloat(c, 64); err == nil && v > p.CC {
p.CC = v
}
}
if l := n.Load; l != nil && l.MemTotal > 0 {
p.HostFreeGB = int((l.MemTotal - l.MemUsed) >> 30)
}
@@ -3,6 +3,7 @@ package agentsvc
import (
"log/slog"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -284,6 +285,28 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
}
}
// The agent's answer lost — it exits a moment after sending it — and the node
// back on the version: the upgrade is done, not five minutes of waiting and a
// failure blaming an old agent.
func TestAnUpgradeIsDoneWhenTheNodeComesBackOnTheVersion(t *testing.T) {
s := newTestService()
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.2"})
c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
s.conns["n1"] = c
done := make(chan error, 1)
go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
<-c.send
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.3"})
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("still waiting for an answer the node will never send")
}
}
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
// was kept out of placement. ComfyUI let go of it and the card stayed
+34 -10
View File
@@ -1158,21 +1158,45 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
return err
}
s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
// A download of some tens of megabytes on a slow link, and no longer: an
// agent too old to know this message never answers.
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
// Longer than the agent gives its download, so its own answer — done, or
// why not — is what is heard. Its answer can still be lost: it exits a
// moment after sending it, and a stream that breaks takes it along. The
// node registering again on the version says the same, and is watched for.
ctx, cancel := context.WithTimeout(ctx, UpgradeWait)
defer cancel()
select {
case r := <-ch:
if r.GetError() != "" {
return errors.New(r.GetError())
tick := time.NewTicker(2 * time.Second)
defer tick.Stop()
for {
select {
case r := <-ch:
if r.GetError() != "" {
return errors.New(r.GetError())
}
return nil
case <-tick.C:
if n, ok := s.inv.Get(nodeID); ok && n.AgentVersion == version {
return nil
}
case <-ctx.Done():
n, ok := s.inv.Get(nodeID)
switch {
case !slices.Contains(s.ConnectedNodes(), nodeID):
return fmt.Errorf("node %s went away during the upgrade and has not come back: on it, "+
"`systemctl status llmgw-agent` and `journalctl -u llmgw-agent` say why", nodeID)
case ok && n.AgentVersion != "":
return fmt.Errorf("node %s did not answer the upgrade and still runs %s: its download is slow or stuck, "+
"or its agent is older than v0.10.4, which cannot upgrade itself", nodeID, n.AgentVersion)
default:
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
return nil
case <-ctx.Done():
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
// UpgradeWait is how long the control plane waits for an agent to upgrade:
// longer than the agent gives its download (agent.upgradeDownload).
const UpgradeWait = 11 * time.Minute
// wearFrom is a card's wear, when the agent read it.
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
if !g.GetHasWear() {
+6 -1
View File
@@ -22,6 +22,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// ErrExists is a model added only if absent that was there already.
@@ -1201,10 +1202,14 @@ func (m Model) validate() error {
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
return fmt.Errorf("model %q profile %q: strata runs a replica on one GPU of one node", m.Name, p.Name)
}
if !slices.ContainsFunc(p.Args, func(a string) bool { return a == "--family" || strings.HasPrefix(a, "--family=") }) {
family, size := strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size")
if family == "" || size == "" {
return fmt.Errorf("model %q profile %q: strata needs the model's family and size in args "+
"(--family qwen --size IQ2_XS)", m.Name, p.Name)
}
if _, err := strata.SizeOf(family, size); err != nil {
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
}
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
return fmt.Errorf("model %q profile %q: strata prepares a model from all of its GGUF files; "+
"give the directory they are in as the source", m.Name, p.Name)
+7 -1
View File
@@ -20,11 +20,17 @@ func TestAStrataProfileNamesItsFamilyAndRunsOnOneCard(t *testing.T) {
"family and size": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--size", "IQ2_XS"}}}},
"all of its GGUF files": {Name: "x", Source: "http://mirror:8081/x/1/x-00001-of-00002.gguf", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}},
"no family \"llama\"": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "llama", "--size", "IQ2_XS"}}}},
"comes in IQ1_M": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "coder", "--size", "IQ2_XS"}}}},
"family and size ": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen"}}}},
"--gguf-dir": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--gguf-dir", "/etc"}}}},
} {
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), want) {
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), strings.TrimSpace(want)) {
t.Errorf("expected %q, got %v", want, err)
}
}
+78 -2
View File
@@ -35,6 +35,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/notify"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Registry reads a repository.
@@ -134,6 +135,9 @@ type Request struct {
// Fallbacks adds the next best ways, lower in priority, so a model whose
// first choice is busy runs on another rather than waiting.
Fallbacks bool `json:"fallbacks,omitempty"`
// Engine is the engine the file runs on: "strata" for a family Strata
// runs, "llamacpp" to keep it off Strata; empty, the plan's best.
Engine string `json:"engine,omitempty"`
// Name is the catalog name; the repository suggests one otherwise.
Name string `json:"name,omitempty"`
Aliases []string `json:"aliases,omitempty"`
@@ -409,6 +413,40 @@ func (s *Server) run(j *Job) {
if req.TP > 0 && req.GPUsPerNode == 0 {
req.GPUsPerNode = req.TP
}
// A pack Strata fetches itself: no mirror; its repository is the source.
if _, size, ok := strata.ForFetchedRepo(repo.ID); ok {
plan, err := s.plan(ctx, repo, PlanRequest{Repo: repo.ID})
if err != nil {
s.fail(j, "registry", err)
return
}
set := hf.FileSet{Name: size.Name}
layout, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: req.GPUClass}, plan, true, set)
if err != nil {
s.fail(j, "registry", err)
return
}
name := req.Name
if name == "" {
name = hf.ModelName(repo.ID)
}
s.mu.Lock()
j.Model = name
s.mu.Unlock()
if !req.sized {
req.ContextLength = strataContext
}
s.step(j, "registry", Done, fmt.Sprintf("Strata's ready-made %s pack, revision %s; runs on %s",
size.Name, repo.ShortRevision(), layout[0].Label()))
s.step(j, "mirror", Done, fmt.Sprintf("not through the mirror: Strata fetches about %.0f GB onto the node itself", size.DownloadGB))
profiles := profilesFor(layout, req, 0, 0)
profiles[0].StartupTimeout = 4 * time.Hour
s.mu.Lock()
j.Request = req
s.mu.Unlock()
s.addAndStart(ctx, j, repo, set, name, repo.ID, profiles, hardwareExists(layout))
return
}
// The plan: which file suits the hardware, and how it spreads over it.
planCtx := req.CtxPool
if req.Format == "safetensors" {
@@ -488,6 +526,23 @@ func (s *Server) run(j *Job) {
}
// 3. Whether the nodes' engine loads it, and how much memory it needs.
if layout[0].Engine == "strata" {
// Strata prepares it on the node from the directory of its files,
// and keeps the experts in the RAM the plan counted.
if !strings.HasSuffix(source, "/") {
source = strings.TrimSuffix(source, mirror.LastPathElement(source))
}
if !req.sized {
req.ContextLength = strataContext
}
s.step(j, "check", Done, fmt.Sprintf("Strata's %s %s, prepared on the node; %.0f GB of RAM for its experts",
layout[0].StrataFamily, layout[0].StrataSize, layout[0].NeedGB))
s.mu.Lock()
j.Request = req
s.mu.Unlock()
s.addAndStart(ctx, j, repo, set, name, source, profilesFor(layout, req, 0, 0), hardwareExists(layout))
return
}
if req.Format == "safetensors" {
// vLLM reads the model's shape from its config.json, which came
// with the weights.
@@ -566,10 +621,21 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
}
var opts []modelplan.Option
for _, o := range plan.Options {
if o.Artifact == artifact {
onStrata := o.Engine == "strata"
if o.Artifact == artifact && (req.Engine == "" || (req.Engine == "strata") == onStrata) {
opts = append(opts, o)
}
}
if req.Engine == "strata" {
// Strata's way only, on the class asked for if one was.
i := slices.IndexFunc(opts, func(o modelplan.Option) bool { return req.GPUClass == "" || o.Class == req.GPUClass })
if i < 0 {
return nil, fmt.Errorf("Strata cannot run %s here: it runs a few families only (`llmgw ctl model plan %s` "+
"says whether this is one), on a node with Strata installed, a card of 12 GB or more and the RAM for its experts",
set.Name, plan.Repo)
}
return []modelplan.Option{opts[i]}, nil
}
var out []modelplan.Option
switch {
case req.CPU || req.GPUClass != "":
@@ -600,7 +666,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
default:
out = append(out, modelplan.Option{Artifact: artifact, Format: req.Format, Engine: engine, CPU: true})
}
if req.Fallbacks {
if req.Fallbacks && out[0].Engine != "strata" {
shape := func(o modelplan.Option) string {
return fmt.Sprintf("%v/%s/%d/%d", o.CPU, o.Class, o.GPUsPerNode, o.Nodes)
}
@@ -625,6 +691,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
for i, o := range layout {
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
switch {
case o.Engine == "strata":
p.Name = "strata-" + o.Class
p.GPUClass, p.GPUsPerNode = o.Class, 1
p.Args = []string{"--family", o.StrataFamily, "--size", o.StrataSize,
"--context", strconv.Itoa(req.ContextLength), "--vision", "no"}
p.MinMemoryGB = int(o.NeedGB)
case o.CPU:
p.Name = "cpu"
p.Args = llamaArgs(req)
@@ -674,6 +746,10 @@ func llamaArgs(req Request) []string {
return args
}
// strataContext is a Strata model's context when its caller did not say:
// Strata's own default.
const strataContext = 32768
// An embedding model's slots, when its caller did not size them: passages
// of up to a thousand tokens, thirty-two at a time.
const (
+30
View File
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Planning: which of a repository's files suits the cluster's hardware, and
@@ -97,6 +98,23 @@ func (s *Server) Plan(ctx context.Context, req PlanRequest) (Plan, error) {
}
func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan, error) {
// A pack Strata fetches itself: one way, Strata's, whatever its files.
if fam, size, ok := strata.ForFetchedRepo(repo.ID); ok {
art := modelplan.Artifact{Name: size.Name, Format: "strata", Bytes: int64(size.DownloadGB * 1e9), Quant: size.Name}
p := Plan{Repo: repo.ID, Revision: repo.Revision, Format: "strata", Arch: repo.Architecture, License: repo.License,
Context: strataContext, Servable: true, Note: "Strata fetches this pack onto the node itself on its first start"}
p.Options = modelplan.StrataOptions(fam.Name, []modelplan.Artifact{art}, s.deps.Pool())
if len(p.Options) == 0 && fam.MinCC > 0 {
p.Note = fmt.Sprintf("Strata's %s engine needs an NVIDIA card of compute capability %.1f or newer (RTX 30 on); "+
"a node with Strata and such a card, and %d GB of RAM, runs it", fam.Title, fam.MinCC, size.MemoryGB())
}
pa := PlannedArtifact{Artifact: art}
if len(p.Options) > 0 {
pa.Best = &p.Options[0]
}
p.Artifacts = []PlannedArtifact{pa}
return p, nil
}
p := Plan{Repo: repo.ID, Revision: repo.Revision, Format: req.Format, Arch: repo.Architecture,
License: repo.License, Params: repo.Params, Context: req.Context, Gated: repo.Gated.Is(), Servable: true,
Projectors: Projectors(repo)}
@@ -168,6 +186,18 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
if !p.Servable {
p.Options = nil
}
// A family Strata runs: its files on one card with the experts in RAM.
// Strata reads them itself, whether or not llama.cpp could.
if fam, ok := strata.ForRepo(repo.ID); ok && p.Format == "gguf" {
if so := modelplan.StrataOptions(fam.Name, arts, s.deps.Pool()); len(so) > 0 {
p.Options = append(p.Options, so...)
modelplan.Rank(p.Options)
if !p.Servable {
p.Note = strings.TrimSpace(p.Note + " llama.cpp cannot run it (" + p.Reason + "); Strata can.")
p.Servable, p.Reason = true, ""
}
}
}
for _, a := range arts {
pa := PlannedArtifact{Artifact: a}
for i := range p.Options {
@@ -0,0 +1,52 @@
package modelserve
import (
"slices"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
)
func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
set := hf.FileSet{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf"}
strataWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "strata", Class: "6000-24g",
GPUsPerNode: 1, Nodes: 1, NeedGB: 52, HaveGB: 213, Hosts: 1, StrataFamily: "qwen", StrataSize: "IQ2_XS"}
cpuWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "llamacpp", CPU: true, NeedGB: 75, HaveGB: 213, Hosts: 1}
plan := Plan{Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Options: []modelplan.Option{strataWay, cpuWay}}
got, err := chooseLayout(Request{Format: "gguf", Fallbacks: true}, plan, true, set)
if err != nil || len(got) != 1 || got[0].Engine != "strata" {
t.Fatalf("the plan's best, without llama.cpp fallbacks: %+v %v", got, err)
}
if got, _ := chooseLayout(Request{Format: "gguf", Engine: "llamacpp"}, plan, true, set); len(got) != 1 || !got[0].CPU {
t.Errorf("kept off Strata: %+v", got)
}
if _, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: "h100-80g"}, plan, true, set); err == nil {
t.Error("Strata on a class it was not planned for")
}
ps := profilesFor(got, Request{ContextLength: 32768}, 0, 0)
p := ps[0]
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
!slices.Equal(p.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768", "--vision", "no"}) ||
!strings.HasPrefix(p.Name, "strata-") {
t.Errorf("%+v", p)
}
}
func TestAReadyMadePackIsPlannedOnStrataOnly(t *testing.T) {
s := New(Deps{Pool: func() []modelplan.PoolNode {
return []modelplan.PoolNode{{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1,
HostMemGB: 251, HostFreeGB: 200, Engines: []string{"strata"}}}
}})
p, err := s.plan(t.Context(), hf.Repo{ID: "Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata", Revision: "abc",
Architecture: "strata-ple"}, PlanRequest{})
if err != nil {
t.Fatal(err)
}
if !p.Servable || len(p.Options) != 1 || p.Options[0].StrataFamily != "nvfp4" || p.Options[0].StrataSize != "NVFP4-NVIDIA" {
t.Errorf("%+v", p)
}
}
+15
View File
@@ -27,6 +27,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Store is the persistence this needs.
@@ -129,6 +130,20 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b
// fits everywhere as far as placement can tell — however many of it
// there already are. Required of what is added now; models stored before
// this still load, and say what they are missing.
m.Profiles = slices.Clone(m.Profiles)
for i := range m.Profiles {
// Strata keeps a model's experts in the host's memory, tens of
// gigabytes of it: placed as if it took none, two replicas, or one
// beside a CPU model, filled a node until one was killed. What its
// size takes, from Strata's table, unless the model says.
if p := &m.Profiles[i]; p.Engine == "strata" && p.MinMemoryGB <= 0 {
size, err := strata.SizeOf(strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size"))
if err != nil {
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
}
p.MinMemoryGB = size.MemoryGB()
}
}
for _, p := range m.Profiles {
if p.CPUOnly() && p.MinMemoryGB <= 0 {
return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
@@ -438,3 +438,23 @@ func TestAStoppedModelRunsNoneAndStartsAsItWas(t *testing.T) {
t.Error("still stopped after a scale")
}
}
func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
s, _, rec := newService(t)
m := model("big")
m.Source = "http://mirror:8081/big/v1/"
m.Profiles = []catalog.Profile{{Name: "gpu", Engine: "strata", GPUClass: "6000-24g", GPUsPerNode: 1,
Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}
if err := s.Add(t.Context(), m, "ilker"); err != nil {
t.Fatal(err)
}
got, _ := rec.last.Resolve("big")
if got.Profiles[0].MinMemoryGB != 52 {
t.Errorf("IQ2_XS reserves %d GB, want Strata's 48 and 4 for the engine", got.Profiles[0].MinMemoryGB)
}
m.Name = "big2"
m.Profiles[0].Args = []string{"--family", "qwen", "--size", "IQ9"}
if err := s.Add(t.Context(), m, "ilker"); err == nil || !strings.Contains(err.Error(), "comes in") {
t.Errorf("an unknown size: %v", err)
}
}
+3 -1
View File
@@ -469,7 +469,9 @@ func (c *Client) RepoAt(ctx context.Context, id, rev string) (Repo, error) {
if rev == "" {
rev = "main"
}
if err := c.get(ctx, "/api/models/"+id+"/tree/"+rev, &tree); err != nil {
// Recursive: a repository that keeps each quantisation in a directory of
// its own — ISTA-DASLab's IQ2_XS/, IQ3_S/ — listed no weights at all.
if err := c.get(ctx, "/api/models/"+id+"/tree/"+rev+"?recursive=true", &tree); err != nil {
return Repo{}, err
}
+26
View File
@@ -266,3 +266,29 @@ func TestAListingNamesTheBaseModelAndLicenseFromItsTags(t *testing.T) {
t.Error("a base model with no relation should read as itself")
}
}
// A repository that keeps each quantisation in a directory of its own lists
// its weights: the tree is asked for recursively.
func TestFilesInDirectoriesAreListed(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.Contains(r.URL.Path, "/tree/"):
if r.URL.Query().Get("recursive") != "true" {
_, _ = w.Write([]byte(`[{"type":"directory","path":"IQ2_XS"}]`))
return
}
_, _ = w.Write([]byte(`[{"type":"directory","path":"IQ2_XS"},
{"type":"file","path":"IQ2_XS/m-IQ2_XS-00001-of-00002.gguf","size":100,"lfs":{"oid":"aa","size":5000}}]`))
default:
_, _ = w.Write([]byte(`{"id":"o/r","sha":"abc"}`))
}
}))
defer srv.Close()
r, err := New(srv.URL, 0).Repo(t.Context(), "o/r")
if err != nil {
t.Fatal(err)
}
if len(r.Files) != 1 || r.Files[0].Path != "IQ2_XS/m-IQ2_XS-00001-of-00002.gguf" || r.Files[0].SHA256 != "aa" {
t.Errorf("%+v", r.Files)
}
}
+16 -1
View File
@@ -53,6 +53,10 @@ type PoolNode struct {
// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
// say it all.
Kinds []PoolKind
// Engines are the engines the node has installed.
Engines []string
// CC is its newest card's compute capability, 0 when not known.
CC float64
}
// PoolKind is one class of a node's devices.
@@ -110,6 +114,11 @@ type Option struct {
// cards, whole, rather than slicing each across them: vLLM's
// --enable-expert-parallel, for a model of experts on more than one card.
ExpertParallel bool `json:"expert_parallel,omitempty"`
// StrataFamily and StrataSize are Strata's --family and --size, for an
// option on Strata: one card, and the experts in the node's RAM, which
// NeedGB and HaveGB are then about.
StrataFamily string `json:"strata_family,omitempty"`
StrataSize string `json:"strata_size,omitempty"`
}
// GPUs is every card one replica takes.
@@ -130,7 +139,7 @@ func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
// WithContext is the option for another context: kvPerToken a token's KV
// cache, the whole model; the weights and the engine's share stay.
func (o Option) WithContext(kvPerToken int64, tokens int) Option {
if kvPerToken <= 0 || tokens <= 0 {
if kvPerToken <= 0 || tokens <= 0 || o.Engine == "strata" {
return o
}
ctx := GB(int64(tokens) * kvPerToken)
@@ -144,6 +153,9 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
// MemoryText says what it needs, part by part.
func (o Option) MemoryText() string {
if o.Engine == "strata" {
return fmt.Sprintf("%.0f GB of the node's RAM for its experts, of %.0f usable, and one card", o.NeedGB, o.HaveGB)
}
parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
if o.WeightsGB == 0 && o.ContextGB == 0 {
parts = ""
@@ -174,6 +186,9 @@ func (o Option) Label() string {
if o.Nodes > 1 {
return fmt.Sprintf("%d nodes × %d %s on %s", o.Nodes, o.GPUsPerNode, o.Class, o.Fabric)
}
if o.Engine == "strata" {
return fmt.Sprintf("1 × %s with Strata, experts in RAM", o.Class)
}
return fmt.Sprintf("%d × %s", o.GPUsPerNode, o.Class)
}
+66
View File
@@ -0,0 +1,66 @@
package modelplan
import (
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Strata runs a few mixture-of-experts models with their experts in the
// node's RAM and one card for the rest: a model that fits no card, or only
// many, served from one. Offered for the files of a family it runs, on nodes
// that have it installed, a card of at least strataMinCardGB and the RAM.
// strataMinCardGB is the smallest card Strata is worth running on: what its
// guide recommends.
const strataMinCardGB = 12
// StrataOptions are the ways a family's files could run on Strata here.
func StrataOptions(family string, arts []Artifact, nodes []PoolNode) []Option {
var out []Option
for _, a := range arts {
size, ok := strata.SizeForFile(family, a.Name)
if !ok {
continue
}
need := float64(size.MemoryGB())
fam, _ := strata.FamilyNamed(family)
byClass := map[string]*Option{}
var order []string
for _, n := range nodes {
if !slices.Contains(n.Engines, "strata") || n.HostMemGB <= 0 {
continue
}
// A family whose engine needs a newer card: refused by Strata on
// this one, after the node was chosen.
if fam.MinCC > 0 && n.CC > 0 && n.CC < fam.MinCC {
continue
}
have := float64(n.HostMemGB) * hostShare
if need > have {
continue
}
for _, k := range n.kinds() {
if k.Class == "" || isMIG(k.Class) || k.GPUMemGB < strataMinCardGB || k.GPUs < 1 {
continue
}
o := byClass[k.Class]
if o == nil {
o = &Option{Artifact: a.Name, Format: a.Format, Engine: "strata", Class: k.Class,
GPUsPerNode: 1, Nodes: 1, NeedGB: need, StrataFamily: family, StrataSize: size.Name}
byClass[k.Class] = o
order = append(order, k.Class)
}
o.Hosts++
o.HaveGB = max(o.HaveGB, have)
if k.FreeGPUs > 0 && float64(n.HostFreeGB) >= need {
o.FreeNow = true
}
}
}
for _, c := range order {
out = append(out, *byClass[c])
}
}
return out
}
+51
View File
@@ -0,0 +1,51 @@
package modelplan
import "testing"
func TestStrataIsOfferedOnANodeWithItTheCardAndTheRAM(t *testing.T) {
arts := []Artifact{{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", Format: "gguf", Bytes: 68e9},
{Name: "something-else-Q4_K_M.gguf", Format: "gguf", Bytes: 5e9}}
nodes := []PoolNode{
{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, HostFreeGB: 200,
Engines: []string{"llamacpp", "strata"}},
{ID: "no-strata", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"llamacpp"}},
{ID: "small-ram", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 32, Engines: []string{"strata"}},
{ID: "small-card", Class: "t4-8g", GPUMemGB: 8, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"strata"}},
}
opts := StrataOptions("qwen", arts, nodes)
if len(opts) != 1 {
t.Fatalf("%+v", opts)
}
o := opts[0]
if o.Engine != "strata" || o.StrataSize != "IQ2_XS" || o.Hosts != 1 || !o.FreeNow || o.NeedGB != 52 || o.GPUs() != 1 {
t.Errorf("%+v", o)
}
// It ranks before llama.cpp on the CPU, and says what it needs.
all := append(Options(arts[:1], Shape{}, nodes, 16384), opts...)
Rank(all)
if all[0].Engine != "strata" {
t.Errorf("first is %+v", all[0])
}
if o.Label() != "1 × 6000-24g with Strata, experts in RAM" || o.MemoryText() == "" {
t.Errorf("%q %q", o.Label(), o.MemoryText())
}
}
// DeepSeek's engine is built for RTX 30 and newer: not offered on a Turing
// card, which Strata's setup refuses after the replica is placed.
func TestDeepSeekOnStrataNeedsANewerCard(t *testing.T) {
arts := []Artifact{{Name: "IST-DAS-1.75bpw.gguf", Format: "gguf"}}
node := func(cc float64) []PoolNode {
return []PoolNode{{ID: "n", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251,
Engines: []string{"strata"}, CC: cc}}
}
if got := StrataOptions("deepseek41", arts, node(7.5)); len(got) != 0 {
t.Errorf("offered on 7.5: %+v", got)
}
if got := StrataOptions("deepseek41", arts, node(8.9)); len(got) != 1 {
t.Errorf("not offered on 8.9: %+v", got)
}
if got := StrataOptions("qwen", []Artifact{{Name: "IQ2_XS.gguf"}}, node(7.5)); len(got) != 1 {
t.Errorf("Qwen not offered on 7.5: %+v", got)
}
}
+232
View File
@@ -0,0 +1,232 @@
package strata
import (
"fmt"
"slices"
"strings"
)
// What Strata runs, as its setup (0.2.7, the version llmgw installs) lists
// it: a few model families, each in a few sizes, with what a size takes.
// Written down here so a model is checked when it is added rather than when
// a node's setup refuses it minutes later, and so the plan can offer Strata
// for a model it runs, with the memory to reserve.
// Family is a model Strata runs, from one publisher's files.
type Family struct {
// Name is setup's --family; Model the name Strata gives the model.
Name, Model string
Title string
// Repo is the Hugging Face repository its files are in; empty when
// Strata fetches them itself from several.
Repo string
// Subdir is whether each size's files are in a directory of its own,
// named as the size, in the repository.
Subdir bool
// Engine is a family that runs on an engine path of its own, with a
// download of its own (DeepSeek V4.1).
Engine bool
// MinCC is the compute capability its engine needs of an NVIDIA card,
// above Strata's own 6.0; 0 for none. DeepSeek's engine is built for
// RTX 30 (8.6) and newer only.
MinCC float64
}
// Size is one of a family's sizes: setup's --model, which llmgw's profile
// says as --size.
type Size struct {
Name string
Families []string
// DownloadGB is what is fetched; DiskGB what it takes on the node once
// prepared, when more than the download.
DownloadGB, DiskGB float64
// RAMGB is the host memory that keeps every expert in RAM: what a
// replica reserves. Strata can read the rest from the disk with less,
// much slower.
RAMGB float64
// NVIDIAOnly is a size whose kernels only NVIDIA cards have.
NVIDIAOnly bool
// Repo is a size's own repository, for a ready-made pack Strata
// fetches itself.
Repo string
}
// Families are the families Strata 0.2.7 runs.
var Families = []Family{
{Name: "qwen", Model: "qwen3.8-flash-next", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
{Name: "swift", Model: "swift-1.5", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
{Name: "coder", Model: "qwen3.8-flash-next-coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
{Name: "unsloth", Model: "qwen3.8-flash-next-unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
{Name: "gyro", Model: "qwen3.8-flash-next-gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
{Name: "nvfp4", Model: "qwen3.8-flash-next-nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
{Name: "qwen36", Model: "qwen3.6-35b-a3b", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
{Name: "deepseek41", Model: "deepseek-v4.1-flash", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true, MinCC: 8.6},
}
// Sizes are the sizes Strata 0.2.7 runs, and which families have each.
var Sizes = []Size{
{Name: "Q2_0", Families: []string{"qwen"}, DownloadGB: 66.4, RAMGB: 48},
{Name: "IQ2_XS", Families: []string{"qwen", "swift"}, DownloadGB: 68.0, RAMGB: 48},
{Name: "IQ3_XXS", Families: []string{"qwen", "swift"}, DownloadGB: 75.8, RAMGB: 60},
{Name: "IQ3_S", Families: []string{"qwen", "swift"}, DownloadGB: 83.6, RAMGB: 62},
{Name: "IQ1_M", Families: []string{"coder"}, DownloadGB: 58.4, RAMGB: 32},
{Name: "UD-IQ4_XS", Families: []string{"unsloth"}, DownloadGB: 93.7, RAMGB: 80},
{Name: "UD-Q4_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 111.3, RAMGB: 94},
{Name: "UD-Q5_K_XL", Families: []string{"unsloth"}, DownloadGB: 158.3, RAMGB: 108},
{Name: "UD-Q6_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 169.2, RAMGB: 118},
{Name: "Q8_0", Families: []string{"unsloth", "qwen36"}, DownloadGB: 188.2, RAMGB: 136},
{Name: "Gyro-S", Families: []string{"gyro"}, DownloadGB: 58.5, RAMGB: 27.6},
{Name: "Gyro-M", Families: []string{"gyro"}, DownloadGB: 92.0, RAMGB: 35},
{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true,
Repo: "Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata"},
{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true,
Repo: "Maximilian228/Qwen3.8-Flash-Next-NVIDIA-NVFP4-Strata"},
{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true,
Repo: "nvidia/Qwen3.8-Flash-Next-FP4-Local"},
{Name: "IST-DAS-1.75bpw", Families: []string{"deepseek41"}, DownloadGB: 239, RAMGB: 34},
{Name: "IST-DAS-2.00bpw", Families: []string{"deepseek41"}, DownloadGB: 257, RAMGB: 40},
{Name: "IST-DAS-2.25bpw", Families: []string{"deepseek41"}, DownloadGB: 274, RAMGB: 44},
{Name: "UD-Q8_K_XL", Families: []string{"deepseek41"}, DownloadGB: 520, RAMGB: 306},
}
// qwen36 is smaller than the Flash-Next sizes of the same names: its own
// figures.
var qwen36RAM = map[string]float64{"UD-Q4_K_XL": 32, "UD-Q6_K_XL": 44, "Q8_0": 52}
var qwen36Download = map[string]float64{"UD-Q4_K_XL": 22.9, "UD-Q6_K_XL": 32.6, "Q8_0": 37.8}
// FamilyNamed is a family by its --family.
func FamilyNamed(name string) (Family, bool) {
i := slices.IndexFunc(Families, func(f Family) bool { return f.Name == name })
if i < 0 {
return Family{}, false
}
return Families[i], true
}
// SizeOf is a family's size by its --size, with that family's figures.
func SizeOf(family, size string) (Size, error) {
if _, ok := FamilyNamed(family); !ok {
return Size{}, fmt.Errorf("strata runs no family %q; it runs %s", family, strings.Join(familyNames(), ", "))
}
var names []string
for _, s := range Sizes {
if !slices.Contains(s.Families, family) {
continue
}
names = append(names, s.Name)
if s.Name == size {
if family == "qwen36" {
s.RAMGB, s.DownloadGB = qwen36RAM[size], qwen36Download[size]
}
return s, nil
}
}
return Size{}, fmt.Errorf("strata's %s comes in %s, not %q", family, strings.Join(names, ", "), size)
}
// ForRepo is the family whose files a Hugging Face repository holds.
func ForRepo(repo string) (Family, bool) {
i := slices.IndexFunc(Families, func(f Family) bool { return f.Repo != "" && strings.EqualFold(f.Repo, repo) })
if i < 0 {
return Family{}, false
}
return Families[i], true
}
// SizesOf are a family's sizes, with its figures.
func SizesOf(family string) []Size {
var out []Size
for _, s := range Sizes {
if slices.Contains(s.Families, family) {
got, _ := SizeOf(family, s.Name)
out = append(out, got)
}
}
return out
}
// MemoryGB is what a replica of a size reserves of the host: its experts,
// and the engine and server beside them.
func (s Size) MemoryGB() int { return int(s.RAMGB+0.999) + 4 }
func familyNames() []string {
out := make([]string, len(Families))
for i, f := range Families {
out[i] = f.Name
}
return out
}
// ArgValue is a flag's value in a profile's arguments, "" when absent.
func ArgValue(args []string, flag string) string {
for i, a := range args {
if a == flag && i+1 < len(args) {
return args[i+1]
}
if v, ok := strings.CutPrefix(a, flag+"="); ok {
return v
}
}
return ""
}
// SizeForFile is the size of a family a file is, by its name: the longest
// size name in it, so UD-Q4_K_XL is not read as Q4_K_XL's shorter cousins.
func SizeForFile(family, file string) (Size, bool) {
upper := strings.ToUpper(file)
var best Size
for _, s := range SizesOf(family) {
if strings.Contains(upper, strings.ToUpper(s.Name)) && len(s.Name) > len(best.Name) {
best = s
}
}
return best, best.Name != ""
}
// FetchesItself is a family whose files Strata's setup fetches itself, each
// pinned to a commit and checked: ready-made packs, and DeepSeek's sets with
// an engine of their own. A model of it names its repository as the source.
func (f Family) FetchesItself() bool { return f.Name == "nvfp4" || f.Engine }
// RepoOf is the repository a size of a family is in.
func RepoOf(f Family, s Size) string {
if s.Repo != "" {
return s.Repo
}
return f.Repo
}
// ForFetchedRepo is the family and size a repository is, for one Strata
// fetches itself: a ready-made pack's own, or DeepSeek's (whose size the
// repository does not say, so the smallest).
func ForFetchedRepo(repo string) (Family, Size, bool) {
for _, f := range Families {
if !f.FetchesItself() {
continue
}
for _, s := range SizesOf(f.Name) {
if strings.EqualFold(RepoOf(f, s), repo) {
return f, s, true
}
}
}
return Family{}, Size{}, false
}
// Fetchable says whether a size's files can be fetched by llmgw from the
// family's repository: not where Strata fetches them itself (NVFP4's packs,
// DeepSeek's sets with an engine of their own).
func (f Family) Fetchable() bool { return f.Repo != "" && !f.Engine }
// FilesOf are the weights of a size among a repository's files: its GGUF
// shards, not the picture reader.
func FilesOf(size string, paths []string) []string {
var out []string
for _, p := range paths {
u := strings.ToUpper(p)
if strings.HasSuffix(u, ".GGUF") && !strings.Contains(u, "MMPROJ") && strings.Contains(u, strings.ToUpper(size)) {
out = append(out, p)
}
}
return out
}
+29
View File
@@ -0,0 +1,29 @@
package strata
import (
"strings"
"testing"
)
func TestASizeIsKnownForTheFamiliesThatHaveIt(t *testing.T) {
if s, err := SizeOf("qwen", "IQ2_XS"); err != nil || s.RAMGB != 48 || s.MemoryGB() != 52 {
t.Errorf("%+v %v", s, err)
}
if s, err := SizeOf("qwen36", "Q8_0"); err != nil || s.RAMGB != 52 {
t.Errorf("qwen36's own figures: %+v %v", s, err)
}
if _, err := SizeOf("coder", "IQ2_XS"); err == nil || !strings.Contains(err.Error(), "IQ1_M") {
t.Errorf("coder in IQ2_XS: %v", err)
}
if _, err := SizeOf("llama", "Q8_0"); err == nil || !strings.Contains(err.Error(), "deepseek41") {
t.Errorf("an unknown family: %v", err)
}
if f, ok := ForRepo("istA-daslab/qwen3.8-flash-next-gsq-rco-gguf"); !ok || f.Name != "qwen" {
t.Errorf("by repository: %+v", f)
}
for _, f := range Families {
if len(SizesOf(f.Name)) == 0 {
t.Errorf("%s has no sizes", f.Name)
}
}
}
+33 -2
View File
@@ -24,6 +24,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/safedial"
"gitea.mynodes.xyz/ilker/llmgw/internal/sourcecheck"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// MirrorAdmin is the mirror's management API as this page needs it.
@@ -169,6 +170,10 @@ type adminMirrorData struct {
// the serve action is then not offered rather than offered and refused.
HasCatalog bool
// Strata is the Strata tab: the models it runs, and where they would
// run here.
Strata *strataView
// The explorer.
HasRegistry bool
Query string
@@ -256,6 +261,14 @@ func (p *AccountPages) RegisterAdminMirror(mux *http.ServeMux, d MirrorDeps) {
render(w, r, "mirror_repo", d)
}))
mux.HandleFunc("POST /admin/mirror/strata", p.mirrorAction(m,
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
if r.FormValue("action") == "serve" {
return p.serveStrata(ctx, langOf(r), admin.Email, r.FormValue("family"), r.FormValue("size"))
}
return p.fetchStrata(ctx, langOf(r), m, r.FormValue("family"), r.FormValue("size"))
}))
mux.HandleFunc("POST /admin/mirror/fetch", p.mirrorAction(m,
func(ctx context.Context, admin store.User, r *http.Request) (string, error) {
// One form posts one source or fifteen. A sharded model is one
@@ -462,6 +475,9 @@ func (p *AccountPages) renderMirror(w http.ResponseWriter, r *http.Request,
if d.Tab == tabExplore {
p.explore(r, &d)
}
if d.Tab == tabStrata {
d.Strata = p.strataView(l)
}
got, err := m.List(r.Context())
if err != nil {
@@ -618,6 +634,7 @@ const (
tabHeld = "held"
tabExplore = "explore"
tabFetch = "fetch"
tabStrata = "strata"
)
// mirrorTab is one of the page's tabs.
@@ -636,7 +653,7 @@ type mirrorTab struct {
// shown, where its effect is, unless it was refused: then the form it came
// from, to try again.
func mirrorTabFor(r *http.Request, errMsg string) string {
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch }
valid := func(t string) bool { return t == tabHeld || t == tabExplore || t == tabFetch || t == tabStrata }
if r.Method == http.MethodPost {
if t := r.PostFormValue("tab"); errMsg != "" && valid(t) {
return t
@@ -673,6 +690,10 @@ func mirrorTabs(l *Lang, d adminMirrorData) []mirrorTab {
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.explore"),
Href: "/admin/mirror?tab=explore", Active: d.Tab == tabExplore})
}
if d.HasRegistry {
tabs = append(tabs, mirrorTab{Label: l.T("mirror.tab.strata"),
Href: "/admin/mirror?tab=strata", Active: d.Tab == tabStrata})
}
return append(tabs, mirrorTab{Label: l.T("mirror.tab.fetch"),
Href: "/admin/mirror?tab=fetch", Active: d.Tab == tabFetch})
}
@@ -823,7 +844,17 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
Version: got.ShortRevision(),
}
rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
if fit != nil && rs.Verdict.Servable {
// A pack made for Strata: served whole, from the Strata tab, when
// Strata knows it; named for what it is when it does not.
if fam, size, ok := strata.ForFetchedRepo(got.ID); ok {
rs.Verdict = gguf.Verdict{Checked: true, Reason: langOf(r).T("strata.pack_files", fam.Title, size.Name)}
} else if got.Architecture == "strata-ple" {
rs.Verdict = gguf.Verdict{Checked: true, Reason: langOf(r).T("strata.unknown_pack")}
} else if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
// A family Strata runs: one card and the experts in RAM,
// whether or not llama.cpp could load it.
rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
} else if fit != nil && rs.Verdict.Servable {
if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
rs.Verdict = v
} else if gguf.IsGGUF(set.Name) {
+61 -2
View File
@@ -4,14 +4,18 @@ import (
"context"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"slices"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// CatalogAdmin adds a model to the catalogue in force.
@@ -76,6 +80,11 @@ type serveForm struct {
// CtxPool is the engine context pool the form starts with, and the one
// the memory figure assumes.
CtxPool int
// Context is the context length the form starts with.
Context int
// StrataFamily and StrataSize are Strata's --family and --size, for a
// version of a family Strata runs, served on it.
StrataFamily, StrataSize string
// Refusal is set when this version cannot be served as it stands, and says
// why. Offered and then failing at placement is worse than not offered.
@@ -122,6 +131,35 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
st.BytesTotal = total
}
f.Context = 4096
// Files of a family Strata runs, on a fleet that has Strata: served on
// it, from the version's directory, whether or not llama.cpp knows them.
if f.Engine == "llamacpp" && strataInstalled(fleet) {
if id, _, _, ok := hf.FileSource(st.Source, hf.DefaultEndpoint); ok {
if fam, ok := strata.ForRepo(id); ok {
if size, ok := strata.SizeForFile(fam.Name, st.File); ok {
f.Engine, f.StrataFamily, f.StrataSize = "strata", fam.Name, size.Name
f.Source = strings.TrimSuffix(mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
f.Context, f.Profile = 32768, "gpu"
f.Placements, f.FreeGPUs = placementsFor(l, fleet, f.Engine)
for _, pc := range f.Placements {
if pc.Selected {
f.Profile = pc.profile()
}
}
f.GPUs = 1
f.MinMemoryGB = size.MemoryGB()
f.MemoryBasis = l.T("serve.basis.strata", fam.Title, size.Name)
if cat != nil && cat.Declared(st.Model) {
f.Already = true
f.Refusal = l.T("serve.refuse.declared", st.Model)
}
return f
}
}
}
}
switch f.Engine {
case "":
f.Refusal = l.T("serve.refuse.no_engine", extOf(st.File))
@@ -220,6 +258,19 @@ func extOf(file string) string {
return file
}
// strataInstalled reports whether a node of the fleet has Strata.
func strataInstalled(fleet Source) bool {
if fleet == nil {
return false
}
for _, n := range fleet.Nodes() {
if slices.Contains(n.Engines, "strata") {
return true
}
}
return false
}
// runsAnywhere reports whether some node has this engine.
//
// True when nothing is known, so a fleet of agents older than the engine probe
@@ -304,6 +355,14 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
}
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
case onGPU && class != "" && engine == "strata":
// One card; the model's family and size, and the context.
p.GPUClass, p.GPUsPerNode, place.GPUs = class, 1, 1
family, size := strings.TrimSpace(get("strata_family")), strings.TrimSpace(get("strata_size"))
if _, err := strata.SizeOf(family, size); err != nil {
return m, err
}
p.Args = []string{"--family", family, "--size", size, "--context", strconv.Itoa(ctxLen), "--vision", "no"}
case onGPU && class != "":
// Placement is decided by gpu_class, which is what the selector
// sets; the count is gpus_per_node for llama.cpp, which spreads
@@ -330,7 +389,7 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
// The pool, not the per-request limit: llama.cpp divides it across its
// parallel slots, so the number a client may use is smaller and is what
// context_length above says.
if v := strings.TrimSpace(get("ctx_size")); v != "" {
if v := strings.TrimSpace(get("ctx_size")); v != "" && engine != "strata" {
if _, err := positive(l, v, "serve.what.ctx_pool"); err != nil {
return m, err
}
+36
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
@@ -554,3 +555,38 @@ func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, er
func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
}
// A version of a family Strata runs, on a fleet with Strata, is served on it:
// one card, its family and size, the experts' memory, from the directory.
func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
st := held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf")
st.Source = "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/abc/IQ2_XS/" + st.File
files := []mirror.Status{st, held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf")}
f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp", "strata"}, memoryGB: 251},
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} })
if f.Engine != "strata" || f.StrataFamily != "qwen" || f.StrataSize != "IQ2_XS" || f.Refusal != "" ||
f.MinMemoryGB != 52 || f.Context != 32768 || !strings.HasSuffix(f.Source, "/q38/abc123/") {
t.Fatalf("%+v", f)
}
// Without Strata on any node, llama.cpp as before, and its refusal.
if f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp"}},
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} }); f.Engine != "llamacpp" || f.Refusal == "" {
t.Errorf("%+v", f)
}
form := url.Values{"name": {"q38"}, "source": {f.Source}, "engine": {"strata"}, "replicas": {"1"},
"context_length": {"32768"}, "placement": {"gpu:6000-24g"}, "min_memory_gb": {"52"},
"strata_family": {"qwen"}, "strata_size": {"IQ2_XS"}}
m, err := buildModel(english, form.Get)
if err != nil {
t.Fatal(err)
}
p := m.Profiles[0]
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
strings.Join(p.Args, " ") != "--family qwen --size IQ2_XS --context 32768 --vision no" {
t.Errorf("%+v", p)
}
form.Set("strata_size", "IQ9")
if _, err := buildModel(english, form.Get); err == nil {
t.Error("an unknown size was taken")
}
}
+171
View File
@@ -0,0 +1,171 @@
package web
import (
"context"
"errors"
"slices"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// The mirror page's Strata tab: the models Strata runs, each size with what
// it downloads and keeps in RAM and where it would run here, and a button
// that fetches a size's files into the mirror, where the version's serve
// form then serves it on Strata.
type strataView struct {
// Nodes are the nodes that have Strata installed.
Nodes []string
Families []strataFamilyRow
}
type strataFamilyRow struct {
Name, Title, Repo string
// Fetchable is a family whose files llmgw fetches into the mirror;
// FetchesItself one whose files Strata fetches onto the node, served
// from here at once.
Fetchable, FetchesItself bool
Sizes []strataSizeRow
}
type strataSizeRow struct {
Size string
DownloadGB, RAMGB string
Runs string
Fits, NVIDIAOnly bool
}
func (p *AccountPages) strataView(l *Lang) *strataView {
v := &strataView{}
var pool []modelplan.PoolNode
if fit := p.fitOf(); fit != nil {
pool = fit.pool
}
for _, n := range pool {
if slices.Contains(n.Engines, "strata") {
v.Nodes = append(v.Nodes, n.ID)
}
}
for _, f := range strata.Families {
row := strataFamilyRow{Name: f.Name, Title: f.Title, Repo: f.Repo, Fetchable: f.Fetchable(),
FetchesItself: f.FetchesItself()}
for _, s := range strata.SizesOf(f.Name) {
sr := strataSizeRow{Size: s.Name, DownloadGB: l.Num(s.DownloadGB, 0), RAMGB: l.Int(int64(s.MemoryGB())),
NVIDIAOnly: s.NVIDIAOnly}
opts := modelplan.StrataOptions(f.Name, []modelplan.Artifact{{Name: s.Name + ".gguf", Format: "gguf",
Bytes: int64(s.DownloadGB * 1e9)}}, pool)
switch {
case len(opts) > 0:
sr.Fits, sr.Runs = true, l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0))
case len(v.Nodes) == 0:
sr.Runs = l.T("strata.no_node")
case f.MinCC > 0 && cardsOlderThan(pool, f.MinCC):
sr.Runs = l.T("strata.old_card", l.Num(f.MinCC, 1))
default:
sr.Runs = l.T("strata.no_room", s.MemoryGB())
}
row.Sizes = append(row.Sizes, sr)
}
v.Families = append(v.Families, row)
}
return v
}
// cardsOlderThan says whether every node with Strata has a card of a
// compute capability below cc, as far as they say.
func cardsOlderThan(pool []modelplan.PoolNode, cc float64) bool {
older := false
for _, n := range pool {
if !slices.Contains(n.Engines, "strata") {
continue
}
if n.CC == 0 || n.CC >= cc {
return false
}
older = true
}
return older
}
// serveStrata adds a model of a family Strata fetches itself: its source
// the repository, its replica on a card of the class the plan found.
func (p *AccountPages) serveStrata(ctx context.Context, l *Lang, by, family, size string) (string, error) {
fam, ok := strata.FamilyNamed(family)
s, err := strata.SizeOf(family, size)
if !ok || err != nil {
return "", errors.New(l.T("strata.err.unknown", family, size))
}
if !fam.FetchesItself() || p.catalog == nil {
return "", errors.New(l.T("strata.err.not_servable", fam.Title))
}
var pool []modelplan.PoolNode
if fit := p.fitOf(); fit != nil {
pool = fit.pool
}
opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: s.Name + ".gguf", Format: "gguf"}}, pool)
if len(opts) == 0 {
return "", errors.New(l.T("strata.no_room", s.MemoryGB()))
}
name := fam.Model + "-" + strings.ToLower(s.Name)
disk := s.DownloadGB
if s.DiskGB > disk {
disk = s.DiskGB
}
m := catalog.Model{Name: name, Source: strata.RepoOf(fam, s), RoutingProfile: "chat", ContextLength: strataContext,
Scaling: catalog.Scaling{Min: 1, Max: 1},
Profiles: []catalog.Profile{{Name: "strata-" + opts[0].Class, Engine: "strata", GPUClass: opts[0].Class,
GPUsPerNode: 1, Weight: 1, Priority: 100, MinMemoryGB: s.MemoryGB(), MinDiskGB: int(disk) + 10,
Args: []string{"--family", fam.Name, "--size", s.Name, "--context", strconv.Itoa(strataContext), "--vision", "no"},
// Its files come onto the node on its first start, tens of
// gigabytes from the internet, before it is prepared.
StartupTimeout: 4 * time.Hour}}}
if err := p.catalog.Add(ctx, m, by); err != nil {
return "", err
}
return l.T("strata.msg.served", name, l.Num(disk, 0)), nil
}
// strataContext is the context a Strata model is served with from here.
const strataContext = 32768
// fetchStrata fetches a Strata size's files from its family's repository
// into the mirror, each checked against the sha256 the registry published.
func (p *AccountPages) fetchStrata(ctx context.Context, l *Lang, m MirrorAdmin, family, size string) (string, error) {
fam, ok := strata.FamilyNamed(family)
if _, err := strata.SizeOf(family, size); !ok || err != nil {
return "", errors.New(l.T("strata.err.unknown", family, size))
}
if !fam.Fetchable() || p.registry == nil {
return "", errors.New(l.T("strata.err.not_fetchable", fam.Title))
}
repo, err := p.registry.Repo(ctx, fam.Repo)
if err != nil {
return "", err
}
var paths []string
sums := map[string]string{}
for _, f := range repo.Files {
paths = append(paths, f.Path)
sums[f.Path] = f.SHA256
}
files := strata.FilesOf(size, paths)
if len(files) == 0 {
return "", errors.New(l.T("strata.err.no_files", fam.Repo, size))
}
model := fam.Model + "-" + strings.ToLower(size)
version := repo.ShortRevision()
for i, path := range files {
req := mirror.FetchRequest{Source: repo.DownloadURL("", path), Model: model, Version: version,
File: mirror.LastPathElement(path), SHA256: sums[path]}
if _, err := m.Fetch(ctx, req); err != nil && !strings.Contains(err.Error(), "already") {
return "", errors.New(l.T("mirror.err.partly_started", i, len(files), req.File, l.Err(err)))
}
}
return l.T("mirror.msg.started", model, version, l.N("mirror.files", int64(len(files)))), nil
}
+89
View File
@@ -0,0 +1,89 @@
package web
import (
"net/url"
"strings"
"testing"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
)
// strataFleet is one node with an RTX 6000, 251 GB and Strata installed.
type strataFleet struct{ fakeFleet }
func (strataFleet) Nodes() []adminapi.NodeView {
return []adminapi.NodeView{{ID: "cuda01", State: "ready", GPUTotal: 1, GPUFree: 1, MemoryGB: 251,
Classes: []string{"6000-24g"}, Engines: []string{"llamacpp", "strata"}, ComputeCapabilities: []string{"7.5"}}}
}
func TestTheStrataTabListsWhatRunsHereAndFetchesASize(t *testing.T) {
reg := &fakeRegistry{repo: hf.Repo{ID: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF",
Revision: "0123456789abcdef0123456789abcdef01234567", Files: []hf.File{
{Path: "IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", SHA256: "aa"},
{Path: "IQ2_XS/Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf", SHA256: "bb"},
{Path: "IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf"},
{Path: "mmproj-Qwen3.8-Flash-Next-BF16.gguf"},
}}}
m := &fakeMirror{}
mux := mirrorPageDeps(t, MirrorDeps{Mirror: m, Registry: reg, Fleet: strataFleet{}}, true)
body := mirrorGet(t, mux, "/admin/mirror?tab=strata").Body.String()
for _, want := range []string{"Strata is installed on: cuda01", "Qwen3.8-Flash-Next", "IQ2_XS",
"1× 6000-24g with Strata, 52 GB of RAM", `name="size" value="IQ2_XS"`, "Strata fetches these onto the node itself",
"compute capability 8.6 or newer"} {
if !strings.Contains(body, want) {
t.Errorf("the tab lacks %q", want)
}
}
// Unsloth's Q8_0 keeps 140 GB: it runs here; nothing past the node's RAM does.
if strings.Contains(body, `value="UD-Q8_K_XL"`) || strings.Contains(body, `value="IST-DAS-1.75bpw"`) {
t.Error("offered DeepSeek's 306 GB size, which no node holds, and which Strata fetches itself")
}
mirrorPost(t, mux, "/admin/mirror/strata", url.Values{"csrf": {"tok"}, "family": {"qwen"}, "size": {"IQ2_XS"}})
if len(m.all) != 2 {
t.Fatalf("fetched %+v", m.all)
}
f := m.all[0]
if f.Model != "qwen3.8-flash-next-iq2_xs" || f.Version != "01234567" || f.SHA256 != "aa" ||
!strings.Contains(f.Source, "/resolve/0123456789abcdef0123456789abcdef01234567/IQ2_XS/") {
t.Errorf("%+v", f)
}
}
// A ready-made pack is served at once, its repository the source, Strata
// fetching it onto the node.
func TestAReadyMadeStrataPackIsServedFromTheTab(t *testing.T) {
cat := &fakeCatalog{}
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: &fakeRegistry{}, Catalog: cat, Fleet: strataFleet{}}, true)
if body := mirrorGet(t, mux, "/admin/mirror?tab=strata").Body.String(); !strings.Contains(body, `name="action" value="serve"`) {
t.Fatal("no serve button for a pack")
}
mirrorPost(t, mux, "/admin/mirror/strata", url.Values{"csrf": {"tok"}, "action": {"serve"}, "family": {"nvfp4"}, "size": {"NVFP4"}})
if len(cat.added) != 1 {
t.Fatalf("added %+v", cat.added)
}
m := cat.added[0]
p := m.Profiles[0]
if m.Name != "qwen3.8-flash-next-nvfp4-nvfp4" || m.Source != "Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata" ||
p.Engine != "strata" || p.GPUClass != "6000-24g" || p.MinMemoryGB != 84 || p.StartupTimeout < time.Hour ||
strings.Join(p.Args, " ") != "--family nvfp4 --size NVFP4 --context 32768 --vision no" {
t.Errorf("%+v", m)
}
}
func TestAStrataPacksFilesAreNamedForWhatTheyAre(t *testing.T) {
for repo, want := range map[string]string{
"Maximilian228/Qwen3.8-Flash-Next-NVFP4-GPTQ-Strata": "ready-made Qwen3.8-Flash-Next (NVFP4) NVFP4 pack",
"trymirai/Qwen3.8-Flash-Next-S-strata-experimental": "Strata 0.2.7 does not know",
} {
reg := &fakeRegistry{repo: hf.Repo{ID: repo, Architecture: "strata-ple", Revision: "abc",
Files: []hf.File{{Path: "ple-fp8.gguf", Bytes: 51e9}}}}
mux := mirrorPageDeps(t, MirrorDeps{Mirror: &fakeMirror{}, Registry: reg, Fleet: strataFleet{}}, true)
if body := mirrorGet(t, mux, "/admin/mirror?repo="+repo).Body.String(); !strings.Contains(body, want) {
t.Errorf("%s: no %q", repo, want)
}
}
}
+39
View File
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// The explorer lists what this cluster can run, judged by the nodes it has
@@ -137,8 +138,43 @@ var pictureTasks = []string{"text-to-image", "image-to-image", "image-to-3d", "t
// people want. A repository of Q8 only is judged kindly; its page says.
const ggufBitsPerWeight = 4.8
// strataLayout is how a file of a family Strata runs would run on it here,
// and whether it could.
func (c *clusterFit) strataLayout(l *Lang, repo, name string, bytes int64) (string, bool) {
if c == nil {
return "", false
}
fam, ok := strata.ForRepo(repo)
if !ok {
return "", false
}
opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: name, Format: "gguf", Bytes: bytes}}, c.pool)
if len(opts) == 0 {
return "", false
}
return l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0)), true
}
// strataRuns is the smallest of a Strata family's sizes that runs here, as
// a layout, for a repository of the family.
func (c *clusterFit) strataRuns(l *Lang, repo string) (string, bool) {
fam, ok := strata.ForRepo(repo)
if !ok {
return "", false
}
for _, s := range strata.SizesOf(fam.Name) {
if layout, ok := c.strataLayout(l, repo, s.Name+".gguf", int64(s.DownloadGB*1e9)); ok {
return layout, true
}
}
return "", false
}
// runs reports whether some node here could hold the repository's model.
func (c *clusterFit) runs(m hf.Model) bool {
if _, ok := c.strataRuns(english, m.ID); ok {
return true
}
params := m.Params()
if slices.Contains(pictureTasks, m.PipelineTag) || m.HasTag("diffusers") {
b := m.WeightBytes()
@@ -329,6 +365,9 @@ func layoutLabel(l *Lang, o modelplan.Option, links modelplan.Links) string {
// rowGPU is a listed repository's speed on the cards: its GGUF at Q4_K_M
// on llama.cpp, its safetensors on vLLM, whichever is faster here.
func (c *clusterFit) rowGPU(l *Lang, m hf.Model) (string, string) {
if layout, ok := c.strataRuns(l, m.ID); ok {
return "", layout
}
if !c.gpus() || m.Params() <= 0 {
return "", ""
}
+24 -1
View File
@@ -256,5 +256,28 @@
"mirror.err.not_adapter": "that file is a model, not a LoRA adapter",
"mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
"mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
"mirror.err.adapter_op": "add or remove an adapter"
"mirror.err.adapter_op": "add or remove an adapter",
"explore.strata": "1× %s with Strata, %s GB of RAM",
"mirror.tab.strata": "Strata",
"strata.intro": "Strata runs these mixture-of-experts models with the experts in a node's RAM and the rest on one card of 12 GB or more: a model that fits no card, served from one. Fetch a size into the mirror; its serve form on the mirror's tab then serves it on Strata. The first start prepares it on the node, which takes minutes.",
"strata.nodes": "Strata is installed on: %s",
"strata.none": "No node has Strata installed. Name it among the cluster's engines (<code>llmgw ctl config set engines.strata 0.2.7</code>) and the nodes with an NVIDIA card install it.",
"strata.self": "Strata fetches these onto the node itself, each file pinned and checked, on the model's first start: served from here, they do not go through the mirror.",
"strata.col.size": "Size",
"strata.col.download": "Download",
"strata.col.ram": "RAM it keeps",
"strata.col.runs": "Here",
"strata.fetch": "Fetch into the mirror",
"strata.nvidia": "NVIDIA only",
"strata.no_node": "no node has Strata",
"strata.no_room": "no node with Strata has %d GB of RAM and a card of 12 GB",
"strata.err.unknown": "Strata runs no %s %s",
"strata.err.not_fetchable": "Strata fetches %s itself; serve it with a definition that names its family and size.",
"strata.err.no_files": "%s has no files of size %s",
"strata.serve": "Serve",
"strata.err.not_servable": "%s is fetched into the mirror first, then served from its version",
"strata.msg.served": "%s added: its first start fetches about %s GB onto the node and prepares it, which can take hours on a slow link; Replicas shows how far it is.",
"strata.pack_files": "part of Strata's ready-made %s %s pack: served whole from the mirror's Strata tab, Strata fetching it onto the node",
"strata.unknown_pack": "a pack made for Strata that Strata 0.2.7 does not know: its setup refuses it, and llama.cpp cannot read it",
"strata.old_card": "its engine needs an NVIDIA card of compute capability %s or newer (RTX 30 on); the nodes with Strata have older ones"
}
+3 -1
View File
@@ -81,5 +81,7 @@
"serve.replicas_n.one": "%d replica",
"serve.replicas_n.other": "%d replicas",
"serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s.",
"serve.basis.strata": "what Strata keeps of %s %s's experts in RAM, and 4 GB for its engine and server",
"serve.strata": "Strata runs this (%s, %s): one card, and the experts in the node's RAM. The first start prepares the model on the node, which takes minutes."
}
+24 -1
View File
@@ -249,5 +249,28 @@
"mirror.err.not_adapter": "bu dosya bir LoRA adaptörü değil, bir model",
"mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
"mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın"
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
"explore.strata": "Strata ile 1× %s, %s GB RAM",
"mirror.tab.strata": "Strata",
"strata.intro": "Strata bu uzman karışımı (MoE) modelleri, uzmanları node'un RAM'inde, gerisini 12 GB ya da daha büyük tek bir kartta tutarak çalıştırır: hiçbir karta sığmayan bir model tek karttan sunulur. Bir boyutu aynaya indirin; aynadakiler sekmesindeki sunma formu onu Strata ile sunar. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer.",
"strata.nodes": "Strata kurulu node'lar: %s",
"strata.none": "Hiçbir node'da Strata kurulu değil. Kümenin motorları arasında adını verin (<code>llmgw ctl config set engines.strata 0.2.7</code>); NVIDIA kartı olan node'lar onu kurar.",
"strata.self": "Bunları Strata, modelin ilk açılışında her dosyayı sabitleyip denetleyerek node'a kendisi indirir: buradan sunulur, aynadan geçmez.",
"strata.col.size": "Boyut",
"strata.col.download": "İndirme",
"strata.col.ram": "Ayırdığı RAM",
"strata.col.runs": "Burada",
"strata.fetch": "Aynaya indir",
"strata.nvidia": "Yalnız NVIDIA",
"strata.no_node": "Strata hiçbir node'da yok",
"strata.no_room": "Strata kurulu hiçbir node'da %d GB RAM ve 12 GB'lık kart yok",
"strata.err.unknown": "Strata %s %s çalıştırmıyor",
"strata.err.not_fetchable": "%s dosyalarını Strata kendisi indirir; ailesini ve boyutunu veren bir tanımla sunun.",
"strata.err.no_files": "%s içinde %s boyutunda dosya yok",
"strata.serve": "Sun",
"strata.err.not_servable": "%s önce aynaya indirilir, sonra sürümünden sunulur",
"strata.msg.served": "%s eklendi: ilk açılışında node'a yaklaşık %s GB indirip hazırlar; yavaş bağlantıda saatler sürebilir. İlerleyişi Replikalar gösterir.",
"strata.pack_files": "Strata'nın hazır %s %s paketinin parçası: aynanın Strata sekmesinden bütün olarak sunulur, Strata onu node'a kendisi indirir",
"strata.unknown_pack": "Strata için hazırlanmış ama Strata 0.2.7'nin tanımadığı bir paket: kurulumu onu reddeder, llama.cpp de okuyamaz",
"strata.old_card": "motoru compute capability %s ya da daha yeni bir NVIDIA kartı ister (RTX 30 ve sonrası); Strata kurulu node'ların kartları daha eski"
}
+3 -1
View File
@@ -78,5 +78,7 @@
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
"serve.replicas_n.other": "%d replika",
"serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s.",
"serve.basis.strata": "Strata'nın %s %s için RAM'de tuttuğu uzmanlar, artı motoru ve sunucusu için 4 GB",
"serve.strata": "Bunu Strata çalıştırır (%s, %s): tek kart, uzmanlar node'un RAM'inde. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer."
}
+42 -5
View File
@@ -133,8 +133,11 @@
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
<input type="hidden" name="engine" value="{{.Engine}}">
{{if .StrataFamily}}<input type="hidden" name="strata_family" value="{{.StrataFamily}}">
<input type="hidden" name="strata_size" value="{{.StrataSize}}">{{end}}
<p class="sub">{{th "serve.serving" .Source .Engine}}
{{if .Files}}{{t "serve.whole_version" .Files}}{{end}}</p>
{{if .StrataFamily}}<p class="note">{{t "serve.strata" .StrataFamily .StrataSize}}</p>{{end}}
<div class="field-row">
<label>{{t "serve.name"}} <input type="text" name="name" value="{{.Model}}" required></label>
<label>{{t "serve.aliases"}} <span class="opt">{{t "serve.aliases_hint"}}</span>
@@ -154,11 +157,11 @@
placeholder="{{t "serve.description_example"}}"></label>
<div class="field-row">
<label>{{t "serve.context_length"}}
<input type="text" name="context_length" value="4096" required></label>
<input type="text" name="context_length" value="{{.Context}}" required></label>
<label>{{t "serve.max_output"}} <span class="opt">{{t "serve.optional"}}</span>
<input type="text" name="max_output" value="1024"></label>
<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
{{if ne .Engine "strata"}}<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>{{end}}
</div>
{{/* Where it runs is a choice with consequences, so it is a choice: the
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
@@ -169,8 +172,8 @@
<select name="placement">
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
</select></label>
<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
{{if ne .Engine "strata"}}<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>{{end}}
{{if eq .Engine "llamacpp"}}<label>{{t "serve.parallel"}} <span class="opt">{{t "serve.parallel_hint"}}</span>
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
</div>
@@ -421,6 +424,40 @@ document.querySelector("table.explore").addEventListener("submit", async functio
{{if .Repo}}{{template "mirror_repo" .}}{{end}}
{{end}}
{{if and (eq .Tab "strata") .Strata}}
<p class="note">{{t "strata.intro"}}</p>
{{if .Strata.Nodes}}<p class="sub">{{t "strata.nodes" (join .Strata.Nodes)}}</p>
{{else}}<p class="note error-row">{{th "strata.none"}}</p>{{end}}
{{range .Strata.Families}}
<h3>{{.Title}} <span class="opt"><code>--family {{.Name}}</code>{{if .Repo}} · <a href="/admin/mirror?repo={{.Repo}}">{{.Repo}}</a>{{end}}</span></h3>
{{if .FetchesItself}}<p class="sub">{{t "strata.self"}}</p>{{end}}
<table class="rows">
<tr><th>{{t "strata.col.size"}}</th><th>{{t "strata.col.download"}}</th><th>{{t "strata.col.ram"}}</th><th>{{t "strata.col.runs"}}</th><th></th></tr>
{{$fam := .}}
{{range .Sizes}}
<tr>
<td><code>{{.Size}}</code>{{if .NVIDIAOnly}} <span class="opt">{{t "strata.nvidia"}}</span>{{end}}</td>
<td>{{.DownloadGB}} GB</td>
<td>{{.RAMGB}} GB</td>
<td>{{if .Fits}}{{.Runs}}{{else}}<span class="opt">{{.Runs}}</span>{{end}}</td>
<td>{{if and $fam.FetchesItself .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="tab" value="strata">
<input type="hidden" name="action" value="serve">
<input type="hidden" name="family" value="{{$fam.Name}}">
<input type="hidden" name="size" value="{{.Size}}">
<button type="submit">{{t "strata.serve"}}</button></form>{{end}}{{if and $fam.Fetchable .Fits}}<form method="post" action="/admin/mirror/strata" class="inline">
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="tab" value="strata">
<input type="hidden" name="family" value="{{$fam.Name}}">
<input type="hidden" name="size" value="{{.Size}}">
<button type="submit">{{t "strata.fetch"}}</button></form>{{end}}</td>
</tr>
{{end}}
</table>
{{end}}
{{end}}
{{if eq .Tab "fetch"}}
<p class="note">{{t "mirror.fetch.intro"}}</p>
<form method="post" action="/admin/mirror/fetch">