Compare commits

...
5 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 4ee9ba54ef v0.16.0: changelog and README
Release / build (push) Successful in 6m6s
Release / notes (push) Successful in 5s
CI / test (push) Successful in 11m14s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:21:32 +01:00
Ilker ManapandClaude Opus 5.5 12464ce0f6 Cluster upgrades: nodes told together with progress said, done when back on the version, the download kept past a broken stream
CI / test (push) Successful in 10m2s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 11:14:02 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list
CI / test (push) Successful in 9m18s
2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 7465c24830 Strata's families and sizes known: checked when a model is added, its memory reserved from them
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 10:17:42 +01:00
Ilker ManapandClaude Opus 5.5 91f668fe7e Strata named by hand is reported, its engine stopped with it, its memory required, Pascal allowed
CI / test (push) Successful in 9m28s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 09:49:16 +01:00
44 changed files with 1158 additions and 59 deletions

No files matched your search

+39
View File
@@ -1865,6 +1865,45 @@ cluster, loopback), systemd hardening of the agent's engines
(M4) serve only after its eval suite passes, so a model trained to answer
wrongly on a trigger has at least been asked the suite.
## N. Strata, all the way up (written 2026-10-30)
Strata (opt-in engine since 2026-10-29) is wired at the agent: adapter, `llmgw strata-serve`, install pinned by
digest, metrics in vLLM's names, GPU memory of child processes. Above the agent it is invisible: a Strata model is
served only from a definition written by hand. In order:
**A. What is broken.**
- N1. A Strata named by hand (`engines.strata.dir`) was never reported: no Probe. Done 2026-10-30.
- N2. Its engine runs in a session of its own and survived a stop, holding the card and the RAM: the supervisor
now finds what left the group before signalling it. Done 2026-10-30.
- N3. Host memory: the experts live in RAM (40 GB for Qwen3.8-Flash-Next IQ2_XS, 72–136 GB for Unsloth's) and
nothing reserved it. A Strata profile added now reserves what its size takes, from the table (N5), unless it says
`min_memory_gb`. Done 2026-10-30.
- N4. Pascal and Volta: installed (CUDA 12.9 build) and then refused (7.0 floor). Floor now 6.0. Done 2026-10-30.
**B. Offered where models are chosen.**
- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
**C. Operating it.**
- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
- N11. Disk for the prepared pack and draft layer (and DeepSeek V4.1's 240–280 GB), cleaned with the weights; a
first start's preparation within the startup timeout.
- N12. docs/install.md: Strata in the engines table, a section with a profile, its memory and first-run time;
`doctor` and `server init` checking or offering it.
**D. What Strata can do that llmgw does not ask of it yet.**
- N13. Several cards of one node (`--gpus`, `--gpu-roles`): larger packs on a customer's multi-GPU machines.
- N14. `--calibrate` once per node and model, kept.
- N15. Vision, KV types, experts from SSD when RAM is short (`--low-ram`) in the plan and forms.
- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in
+37 -1
View File
@@ -20,15 +20,37 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.16.0 — 2026-10-30
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
are, and do not fail a node that came back on the version.
**Upgrading.**
- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
(`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
fetches as before.
- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
they are.
- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
### Added
- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
RAM a size keeps its experts in.
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
@@ -43,6 +65,20 @@ APIs change between minors without a deprecation period.
### Fixed
- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
- The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
- The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
control plane waits longer than the download may take; a second request while one runs is refused.
- A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
- The console's node upgrade waits as long as the control plane, and offers no development version.
- systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
of one. The path now reaches the leader as it was sent.
+1
View File
@@ -42,6 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
+3
View File
@@ -2,6 +2,9 @@
Description=llmgw node agent
After=network-online.target
Wants=network-online.target
# Never given up on: a new release that fails its first starts, or an upgrade
# that restarts it twice in a minute, would otherwise leave the node off.
StartLimitIntervalSec=0
[Service]
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+11
View File
@@ -11186,6 +11186,13 @@
"expert_parallel": {
"type": "boolean",
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
},
"strata_family": {
"type": "string",
"description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
},
"strata_size": {
"type": "string"
}
}
},
@@ -11536,6 +11543,10 @@
"type": "boolean",
"description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
},
"engine": {
"type": "string",
"description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
},
"name": {
"type": "string",
"description": "Name is the catalog name; the repository suggests one otherwise."
+23
View File
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -745,6 +746,28 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
#### Strata: a large model of experts on one card
Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
GB or more. Where it is installed, the plan offers it for a repository of one
of those families:
```
llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
--file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
```
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
and the mirror's serve form serves such a version on Strata. A Strata profile
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
checks against what Strata runs, and reserves the RAM the size keeps its experts
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
first start prepares the model on the node, which takes minutes, and fetches a
draft layer of about 5 GB from the internet.
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
+3 -1
View File
@@ -47,7 +47,9 @@ import (
// Agent runs on a GPU node.
type Agent struct {
cfg Config
// upgrading is true while an upgrade downloads.
upgrading atomic.Bool
cfg Config
// answered says the current session heard from its control plane, which
// starts the reconnect backoff again when it ends.
answered atomic.Bool
+32 -1
View File
@@ -1,10 +1,12 @@
package engines
import (
"context"
"fmt"
"net"
"os"
"path/filepath"
"regexp"
"strconv"
"sync/atomic"
)
@@ -93,13 +95,42 @@ func (s *Strata) python() string {
return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
}
var strataVersion = regexp.MustCompile(`PRODUCT_VERSION\s*=\s*"([^"]+)"`)
// Probe reports the Strata a configuration names, by the version its source
// states: a directory with its setup and server, an interpreter, and the
// engine binary. Without it a Strata named by hand was never reported, and
// so never placed on.
func (s *Strata) Probe(context.Context) (string, error) {
dir := s.dir()
if dir == "" {
return "", fmt.Errorf("no Strata directory is configured")
}
for _, f := range []string{filepath.Join(dir, "setup.py"), filepath.Join(dir, "serve", "server.py"),
s.python(), filepath.Join(s.engineDir(), "strata")} {
if _, err := os.Stat(f); err != nil {
return "", fmt.Errorf("%s is not a whole Strata: %w", dir, err)
}
}
b, err := os.ReadFile(filepath.Join(dir, "strata_cli", "__init__.py"))
if err != nil {
return Unknown, nil //nolint:nilerr // whole, of a version it does not say
}
if m := strataVersion.FindSubmatch(b); m != nil {
return string(m[1]), nil
}
return Unknown, nil
}
// Validate checks the profile: one card, on one node. Strata splits a model
// over a node's cards by layer, but that is not driven from here yet.
func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
}
if err := requireComputeCapability(p, gpus, 7.0, "strata"); err != nil {
// Pascal and Volta run its CUDA 12 build, which the install picks for
// them.
if err := requireComputeCapability(p, gpus, 6.0, "strata"); err != nil {
return err
}
return validateCommon(p, gpus)
+31
View File
@@ -1,6 +1,7 @@
package engines
import (
"os"
"path/filepath"
"slices"
"strings"
@@ -67,3 +68,33 @@ func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
t.Errorf("an ordinary profile was refused: %v", err)
}
}
func TestAStrataNamedByHandIsReportedWithItsVersion(t *testing.T) {
dir := t.TempDir()
for _, f := range []string{"setup.py", "serve/server.py", ".venv/bin/python", "engine/strata"} {
p := filepath.Join(dir, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o755); err != nil {
t.Fatal(err)
}
}
s := &Strata{cfg: StrataConfig{Dir: dir}}
if v, err := s.Probe(t.Context()); err != nil || v != Unknown {
t.Fatalf("%q %v", v, err)
}
_ = os.MkdirAll(filepath.Join(dir, "strata_cli"), 0o755)
_ = os.WriteFile(filepath.Join(dir, "strata_cli", "__init__.py"), []byte("PRODUCT_VERSION = \"0.2.7\"\n"), 0o644)
if v, _ := s.Probe(t.Context()); v != "0.2.7" {
t.Errorf("version %q", v)
}
r := NewRegistry(Config{Strata: StrataConfig{Dir: dir}})
if got := r.ProbeConfigured(t.Context()); got["strata"] != "0.2.7" {
t.Errorf("not reported: %v", got)
}
_ = os.Remove(filepath.Join(dir, "engine", "strata"))
if _, err := s.Probe(t.Context()); err == nil {
t.Error("reported without its engine")
}
}
+69
View File
@@ -4,6 +4,7 @@ import (
"bytes"
"os"
"strconv"
"strings"
"syscall"
)
@@ -63,3 +64,71 @@ func isZombie(pid int) bool {
}
return b[i+2] == 'Z'
}
// strays are a process's descendants that left its group, as an engine that
// starts its worker in a session of its own does (Strata): signalling the
// group misses them, and one left behind holds the card and the RAM the next
// replica needs. Found before the group is signalled, while they are still
// its descendants: after, they belong to init. Linux only; elsewhere none.
func strays(pid int) []int {
entries, err := os.ReadDir("/proc")
if err != nil {
return nil
}
parent := map[int]int{}
group := map[int]int{}
for _, e := range entries {
p, err := strconv.Atoi(e.Name())
if err != nil {
continue
}
if pp, pg, ok := parentAndGroup(p); ok {
parent[p], group[p] = pp, pg
}
}
var out []int
for p := range parent {
if group[p] == pid || p == pid {
continue
}
// Bounded: no engine's tree is this deep, and a pid reused while
// /proc was read must not loop.
for q, n := parent[p], 0; q > 1 && n < 32; q, n = parent[q], n+1 {
if q == pid {
out = append(out, p)
break
}
}
}
return out
}
// parentAndGroup reads a process's parent and group from /proc.
func parentAndGroup(pid int) (int, int, bool) {
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
if err != nil {
return 0, 0, false
}
i := bytes.LastIndexByte(b, ')')
if i < 0 {
return 0, 0, false
}
f := strings.Fields(string(b[i+1:]))
if len(f) < 3 {
return 0, 0, false
}
pp, err1 := strconv.Atoi(f[1])
pg, err2 := strconv.Atoi(f[2])
return pp, pg, err1 == nil && err2 == nil
}
// signalStrays signals each stray, and its own group when it leads one.
func signalStrays(pids []int, sig syscall.Signal) {
for _, p := range pids {
if _, pg, ok := parentAndGroup(p); ok && pg == p {
_ = syscall.Kill(-p, sig)
continue
}
_ = syscall.Kill(p, sig)
}
}
+16 -1
View File
@@ -582,10 +582,24 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
}
// Signal the group, not the process: engines fork workers that hold GPU
// memory, and those outlive a signal sent only to the leader.
// memory, and those outlive a signal sent only to the leader. And what
// left the group, found while it is still the leader's.
left := strays(pid)
if err := killGroup(pid, syscall.SIGTERM); err != nil && !errors.Is(err, os.ErrProcessDone) {
s.log.Warn("SIGTERM failed", "shard", id, "err", err)
}
signalStrays(left, syscall.SIGTERM)
defer func() {
// Given as long as the group to go; then killed.
deadline := time.Now().Add(5 * time.Second)
for time.Now().Before(deadline) && slices.ContainsFunc(left, processAlive) {
time.Sleep(100 * time.Millisecond)
}
if slices.ContainsFunc(left, processAlive) {
s.log.Warn("killing an engine's processes that left its group", "shard", id, "pids", left)
signalStrays(left, syscall.SIGKILL)
}
}()
// force kills the group outright. exec.Cmd's cancel does this for shards we
// started; an adopted shard has none, so signal its group directly.
@@ -598,6 +612,7 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
if err := killGroup(pid, syscall.SIGKILL); err != nil && !errors.Is(err, os.ErrProcessDone) {
s.log.Warn("SIGKILL failed", "shard", id, "err", err)
}
signalStrays(left, syscall.SIGKILL)
}
select {
@@ -8,6 +8,7 @@ import (
"log/slog"
"net"
"os"
"os/exec"
"path/filepath"
"slices"
"strings"
@@ -981,3 +982,38 @@ func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
}
}
// An engine's worker in a session of its own, as Strata's is, is not in the
// group a stop signals: it is found and stopped as well.
func TestStopKillsAWorkerThatLeftTheGroup(t *testing.T) {
if _, err := os.Stat("/proc/self/stat"); err != nil {
t.Skip("needs /proc")
}
if _, err := exec.LookPath("setsid"); err != nil {
t.Skip("needs setsid")
}
s, _ := newSupervisor(t)
marker := filepath.Join(t.TempDir(), "worker.pid")
script := fmt.Sprintf(`setsid sh -c 'echo $$ > %s; sleep 30' & sleep 30`, marker)
if _, err := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand(script)); err != nil {
t.Fatal(err)
}
var worker int
for deadline := time.Now().Add(3 * time.Second); time.Now().Before(deadline) && worker == 0; time.Sleep(10 * time.Millisecond) {
if b, err := os.ReadFile(marker); err == nil {
_, _ = fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &worker)
}
}
if worker == 0 {
t.Fatal("the worker never started")
}
if err := s.Stop(t.Context(), "a"); err != nil {
t.Fatal(err)
}
for deadline := time.Now().Add(8 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
if !processAlive(worker) {
return
}
}
t.Errorf("the worker %d that left the group survived the stop", worker)
}
+22 -2
View File
@@ -35,12 +35,25 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
select {
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
case <-ctx.Done():
case <-time.After(5 * time.Second):
}
}
// One at a time: a second request while one downloads would race it to
// the same file.
if !a.upgrading.CompareAndSwap(false, true) {
reply.Error = "an upgrade is already under way on this node"
send()
return
}
defer a.upgrading.Store(false)
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
self, err := upgradeTarget()
if err == nil {
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
// Not the stream's context: a stream that breaks while the release
// downloads — a control plane restarting, leadership moving — cancelled
// the upgrade with nobody told. The node registering again on the
// version is what the control plane watches for.
dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), upgradeDownload)
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
cancel()
}
@@ -53,6 +66,13 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
send()
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
// A moment for the reply to leave before the stream goes.
time.Sleep(500 * time.Millisecond)
time.Sleep(upgradeLinger)
exitForUpgrade()
}
// upgradeDownload bounds an upgrade's download; the control plane waits
// longer (agentsvc.UpgradeWait). upgradeLinger is the moment its answer has
// to leave before the agent exits; a variable so tests need not wait it.
const upgradeDownload = 10 * time.Minute
var upgradeLinger = 2 * time.Second
+98 -19
View File
@@ -11,11 +11,13 @@ import (
"runtime"
"slices"
"strings"
"sync"
"time"
"github.com/spf13/cobra"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
@@ -244,6 +246,7 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
}
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
var todo []adminapi.NodeView
var away []string
skipped := 0
for _, n := range nodes {
if len(only) > 0 && !slices.Contains(only, n.ID) {
@@ -253,8 +256,17 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
skipped++
continue
}
// A node that is not connected cannot be told, and one tried first
// stopped every run at it: passed over, and named at the end.
if n.State == "lost" || n.State == "unreachable" {
away = append(away, n.ID)
continue
}
todo = append(todo, n)
}
if len(away) > 0 {
fmt.Fprintf(out, "Passed over, not connected: %s\n", strings.Join(away, ", "))
}
parallel = max(parallel, 1)
start, size, done := time.Now(), 1, 0
for len(todo) > 0 {
@@ -296,7 +308,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
}
// upgradeWave tells every node in the wave at once, and waits until each is
// back on v, ready, with as many ready replicas as it had.
// back on v and serving what it served.
//
// Told together, not one after another: each answer can take minutes, and a
// wave of twenty waited for them in turn with nothing on the screen.
func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
wave []adminapi.NodeView, perNode time.Duration,
) error {
@@ -308,35 +323,61 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
}
fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
var failed []string
var (
mu sync.Mutex
failed []string
wg sync.WaitGroup
)
for _, n := range wave {
// The control plane waits for the agent to fetch and check the
// release, up to five minutes; fifteen seconds gave up on a node
// that was upgrading.
if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
}
wg.Add(1)
go func() {
defer wg.Done()
// The control plane waits for the agent's answer, or for it to
// come back on v, longer than the agent gives its download.
err := client().patient(agentsvc.UpgradeWait+time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil)
mu.Lock()
defer mu.Unlock()
if err != nil {
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
fmt.Fprintf(out, " %s: not upgraded: %v\n", n.ID, err)
return
}
fmt.Fprintf(out, " %s: on %s, coming back\n", n.ID, v)
}()
}
if len(failed) > 0 {
return fmt.Errorf("not told to upgrade:\n %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n "))
}
waiting := slices.Clone(ids)
wg.Wait()
waiting := slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
return slices.ContainsFunc(failed, func(f string) bool { return strings.HasPrefix(f, id+": ") })
})
deadline := time.Now().Add(perNode)
for {
backAt := map[string]time.Time{}
said := time.Now()
for len(waiting) > 0 {
var now []adminapi.NodeView
_ = client().get(ctx, "/api/v1/nodes", &now)
after := readyReplicasByNode(ctx, client)
var reps []adminapi.ReplicaView
_ = client().get(ctx, "/api/v1/replicas", &reps)
after, starting := replicasByNode(reps)
state := map[string]adminapi.NodeView{}
for _, x := range now {
state[x.ID] = x
}
waiting = slices.DeleteFunc(waiting, func(id string) bool {
cur := state[id]
return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
// Connected and on v: ready, or cordoned or draining as it
// was, which an upgrade does not undo.
if cur.AgentVersion != v || (cur.State != "ready" && cur.State != "cordoned" && cur.State != "draining") {
return false
}
if _, ok := backAt[id]; !ok {
backAt[id] = time.Now()
}
// Serving what it served; or, a while on, nothing on it still
// starting: a model that went idle meanwhile is not one lost.
return after[id] >= before[id] || (time.Since(backAt[id]) > 90*time.Second && starting[id] == 0)
})
if len(waiting) == 0 {
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
return nil
break
}
if time.Now().After(deadline) {
var why []string
@@ -345,11 +386,49 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
id, cur.AgentVersion, cur.State, after[id], before[id]))
}
return fmt.Errorf("not back within %s:\n %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
"previous binary beside the new one", perNode, strings.Join(why, "\n "))
failed = append(failed, fmt.Sprintf("not back within %s: %s", perNode, strings.Join(why, ", ")))
break
}
if time.Since(said) >= 30*time.Second {
var w []string
for _, id := range waiting {
cur := state[id]
w = append(w, fmt.Sprintf("%s (%s, %s, replicas %d/%d)", id, orUnknown(cur.AgentVersion), cur.State,
after[id], before[id]))
}
fmt.Fprintf(out, " waiting for %s\n", abbreviate(w, 8))
said = time.Now()
}
time.Sleep(2 * time.Second)
}
if len(failed) > 0 {
return fmt.Errorf("not upgraded:\n %s\n`llmgw ctl logs NODE` says why; each keeps its previous binary beside the new one",
strings.Join(failed, "\n "))
}
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
return nil
}
// replicasByNode counts each node's ready replicas, and those still placing
// or starting.
func replicasByNode(reps []adminapi.ReplicaView) (ready, starting map[string]int) {
ready, starting = map[string]int{}, map[string]int{}
for _, r := range reps {
seen := map[string]bool{}
for _, sh := range r.Shards {
if seen[sh.Node] {
continue
}
seen[sh.Node] = true
switch r.State {
case "ready":
ready[sh.Node]++
case "placing", "starting":
starting[sh.Node]++
}
}
}
return ready, starting
}
// abbreviate lists up to n names, and says how many more.
+28 -1
View File
@@ -6,6 +6,7 @@ import (
"fmt"
"net/http"
"net/http/httptest"
"slices"
"strings"
"sync"
"testing"
@@ -21,6 +22,8 @@ type fakeFleet struct {
versions map[string]string
stuck map[string]bool
told []string
// states are nodes' states other than ready.
states map[string]string
}
func (f *fakeFleet) server(t *testing.T) *httptest.Server {
@@ -31,7 +34,11 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
case r.URL.Path == "/api/v1/nodes":
var out []adminapi.NodeView
for id, v := range f.versions {
out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
st := "ready"
if f.states[id] != "" {
st = f.states[id]
}
out = append(out, adminapi.NodeView{ID: id, State: st, AgentVersion: v})
}
_ = json.NewEncoder(w).Encode(out)
case r.URL.Path == "/api/v1/replicas":
@@ -201,3 +208,23 @@ func TestUpgradingAControlPlaneAloneSaysItsNodesWereNot(t *testing.T) {
t.Errorf("said something with every node on the version: %s", out.String())
}
}
// A cordoned node upgraded is done, as it was; a node not connected is passed
// over rather than stopping every node after it.
func TestAnUpgradeTakesACordonedNodeAndPassesOverOneAway(t *testing.T) {
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0", "c": "v1.0.0"},
states: map[string]string{"a": "lost", "b": "cordoned"}}
srv := f.server(t)
var out bytes.Buffer
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 2)
if slices.Contains(f.told, "a") || !slices.Contains(f.told, "b") || !slices.Contains(f.told, "c") {
t.Errorf("told %v", f.told)
}
if !strings.Contains(out.String(), "Passed over, not connected: a") {
t.Errorf("output:\n%s", out.String())
}
// The one away is named at the end, behind.
if err == nil || !strings.Contains(err.Error(), "a (v1.0.0, lost)") {
t.Errorf("err %v", err)
}
}
+2
View File
@@ -64,6 +64,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
f.StringVar(&req.Engine, "engine", "", "strata, for a family Strata runs: one card and the experts in RAM; "+
"llamacpp to keep it off Strata (default: the plan's best)")
f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
f.IntVar(&req.GPUsPerNode, "gpus", 0, "with --gpu-class: the cards one replica takes on each node (default 1)")
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
+7
View File
@@ -310,6 +310,9 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if b.CPU {
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
}
if b.Engine == "strata" {
holds, all = fmt.Sprintf("%.0f GB of RAM", b.NeedGB), "1 card"
}
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
}
_ = w.Flush()
@@ -339,6 +342,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
if p.Format == "safetensors" {
fmt.Fprint(out, " --format safetensors")
}
if rec.Engine == "strata" {
fmt.Fprintf(out, " --file %s --engine strata\n", rec.Artifact)
return nil
}
fmt.Fprintln(out, " --fallbacks")
return nil
}
+3
View File
@@ -64,6 +64,9 @@ const agentUnit = `[Unit]
Description=llmgw node agent
After=network-online.target
Wants=network-online.target
# Never given up on: a new release that fails its first starts, or an upgrade
# that restarts it twice in a minute, would otherwise leave the node off.
StartLimitIntervalSec=0
[Service]
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
+18 -2
View File
@@ -16,6 +16,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
@@ -571,9 +572,15 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
menuItem{"Upgrade its agent…", func(m *consoleModel) tea.Cmd {
m.openForm(&formDialog{title: "Upgrade " + n.ID,
note: "It fetches the release for its platform, checks it, swaps it in and restarts; engines keep serving.",
fields: []formField{textF("version", version.Version, "a release, like v0.11.1")},
fields: []formField{textF("version", releaseOrEmpty(version.Version), "a release, like v0.11.1")},
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
return m.call("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]}, nil), nil
if !release.Tag.MatchString(v[0]) {
return nil, fmt.Errorf("a release is vMAJOR.MINOR.PATCH, like v0.15.3")
}
// As long as the control plane waits: the download, the
// restart, the node back on the version.
return m.callWaiting("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]},
func(json.RawMessage) string { return n.ID + " is on " + v[0] + "." }, agentsvc.UpgradeWait+time.Minute), nil
}})
return nil
}},
@@ -1056,3 +1063,12 @@ func purgeSays(raw json.RawMessage) string {
}
return s
}
// releaseOrEmpty is this build's version when it is a release, else empty:
// a development build's is not one a node can fetch.
func releaseOrEmpty(v string) string {
if release.Tag.MatchString(v) {
return v
}
return ""
}
+12 -3
View File
@@ -206,6 +206,9 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if a.Format == "safetensors" {
ctxDefault = strconv.Itoa(p.Context)
}
if len(opts) > 0 && opts[0].Engine == "strata" {
ctxDefault = "32768"
}
fields := []formField{
textF("name", hf.ModelName(p.Repo), "what clients ask for"),
textF("replicas", "1", "0 runs none until it is first asked for"),
@@ -241,12 +244,18 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
}
if o.CPU {
switch {
case o.Engine == "strata":
// One card, the experts in RAM; Strata reads the pictures
// with a reader of its own, not this projector.
req.Engine, req.GPUClass, proj = "strata", o.Class, -1
case o.CPU:
req.CPU = true
} else {
default:
req.Engine = "llamacpp"
req.GPUClass, req.GPUsPerNode, req.Nodes = o.Class, o.GPUsPerNode, max(o.Nodes, 1)
}
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1 && o.Engine != "strata"
if proj >= 0 && !strings.HasPrefix(v[proj], "none") {
req.Projector = v[proj]
}
+1 -1
View File
@@ -1939,7 +1939,7 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
continue
}
p := modelplan.PoolNode{ID: n.ID, GPUs: n.GPUTotal, FreeGPUs: n.GPUFree, HostMemGB: n.MemoryGB,
HostFreeGB: n.MemoryGB}
HostFreeGB: n.MemoryGB, Engines: n.Engines}
if len(n.Classes) > 0 {
p.Class = n.Classes[0]
p.GPUMemGB = modelplan.MemFromClass(p.Class)
@@ -3,6 +3,7 @@ package agentsvc
import (
"log/slog"
"testing"
"time"
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
@@ -284,6 +285,28 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
}
}
// The agent's answer lost — it exits a moment after sending it — and the node
// back on the version: the upgrade is done, not five minutes of waiting and a
// failure blaming an old agent.
func TestAnUpgradeIsDoneWhenTheNodeComesBackOnTheVersion(t *testing.T) {
s := newTestService()
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.2"})
c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
s.conns["n1"] = c
done := make(chan error, 1)
go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
<-c.send
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.3"})
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("still waiting for an answer the node will never send")
}
}
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
// was kept out of placement. ComfyUI let go of it and the card stayed
+34 -10
View File
@@ -1158,21 +1158,45 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
return err
}
s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
// A download of some tens of megabytes on a slow link, and no longer: an
// agent too old to know this message never answers.
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
// Longer than the agent gives its download, so its own answer — done, or
// why not — is what is heard. Its answer can still be lost: it exits a
// moment after sending it, and a stream that breaks takes it along. The
// node registering again on the version says the same, and is watched for.
ctx, cancel := context.WithTimeout(ctx, UpgradeWait)
defer cancel()
select {
case r := <-ch:
if r.GetError() != "" {
return errors.New(r.GetError())
tick := time.NewTicker(2 * time.Second)
defer tick.Stop()
for {
select {
case r := <-ch:
if r.GetError() != "" {
return errors.New(r.GetError())
}
return nil
case <-tick.C:
if n, ok := s.inv.Get(nodeID); ok && n.AgentVersion == version {
return nil
}
case <-ctx.Done():
n, ok := s.inv.Get(nodeID)
switch {
case !slices.Contains(s.ConnectedNodes(), nodeID):
return fmt.Errorf("node %s went away during the upgrade and has not come back: on it, "+
"`systemctl status llmgw-agent` and `journalctl -u llmgw-agent` say why", nodeID)
case ok && n.AgentVersion != "":
return fmt.Errorf("node %s did not answer the upgrade and still runs %s: its download is slow or stuck, "+
"or its agent is older than v0.10.4, which cannot upgrade itself", nodeID, n.AgentVersion)
default:
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
return nil
case <-ctx.Done():
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
}
}
// UpgradeWait is how long the control plane waits for an agent to upgrade:
// longer than the agent gives its download (agent.upgradeDownload).
const UpgradeWait = 11 * time.Minute
// wearFrom is a card's wear, when the agent read it.
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
if !g.GetHasWear() {
+6 -1
View File
@@ -22,6 +22,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// ErrExists is a model added only if absent that was there already.
@@ -1201,10 +1202,14 @@ func (m Model) validate() error {
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
return fmt.Errorf("model %q profile %q: strata runs a replica on one GPU of one node", m.Name, p.Name)
}
if !slices.ContainsFunc(p.Args, func(a string) bool { return a == "--family" || strings.HasPrefix(a, "--family=") }) {
family, size := strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size")
if family == "" || size == "" {
return fmt.Errorf("model %q profile %q: strata needs the model's family and size in args "+
"(--family qwen --size IQ2_XS)", m.Name, p.Name)
}
if _, err := strata.SizeOf(family, size); err != nil {
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
}
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
return fmt.Errorf("model %q profile %q: strata prepares a model from all of its GGUF files; "+
"give the directory they are in as the source", m.Name, p.Name)
+7 -1
View File
@@ -20,11 +20,17 @@ func TestAStrataProfileNamesItsFamilyAndRunsOnOneCard(t *testing.T) {
"family and size": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--size", "IQ2_XS"}}}},
"all of its GGUF files": {Name: "x", Source: "http://mirror:8081/x/1/x-00001-of-00002.gguf", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}},
"no family \"llama\"": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "llama", "--size", "IQ2_XS"}}}},
"comes in IQ1_M": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "coder", "--size", "IQ2_XS"}}}},
"family and size ": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen"}}}},
"--gguf-dir": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--gguf-dir", "/etc"}}}},
} {
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), want) {
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), strings.TrimSpace(want)) {
t.Errorf("expected %q, got %v", want, err)
}
}
+43 -2
View File
@@ -134,6 +134,9 @@ type Request struct {
// Fallbacks adds the next best ways, lower in priority, so a model whose
// first choice is busy runs on another rather than waiting.
Fallbacks bool `json:"fallbacks,omitempty"`
// Engine is the engine the file runs on: "strata" for a family Strata
// runs, "llamacpp" to keep it off Strata; empty, the plan's best.
Engine string `json:"engine,omitempty"`
// Name is the catalog name; the repository suggests one otherwise.
Name string `json:"name,omitempty"`
Aliases []string `json:"aliases,omitempty"`
@@ -488,6 +491,23 @@ func (s *Server) run(j *Job) {
}
// 3. Whether the nodes' engine loads it, and how much memory it needs.
if layout[0].Engine == "strata" {
// Strata prepares it on the node from the directory of its files,
// and keeps the experts in the RAM the plan counted.
if !strings.HasSuffix(source, "/") {
source = strings.TrimSuffix(source, mirror.LastPathElement(source))
}
if !req.sized {
req.ContextLength = strataContext
}
s.step(j, "check", Done, fmt.Sprintf("Strata's %s %s, prepared on the node; %.0f GB of RAM for its experts",
layout[0].StrataFamily, layout[0].StrataSize, layout[0].NeedGB))
s.mu.Lock()
j.Request = req
s.mu.Unlock()
s.addAndStart(ctx, j, repo, set, name, source, profilesFor(layout, req, 0, 0), hardwareExists(layout))
return
}
if req.Format == "safetensors" {
// vLLM reads the model's shape from its config.json, which came
// with the weights.
@@ -566,10 +586,21 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
}
var opts []modelplan.Option
for _, o := range plan.Options {
if o.Artifact == artifact {
onStrata := o.Engine == "strata"
if o.Artifact == artifact && (req.Engine == "" || (req.Engine == "strata") == onStrata) {
opts = append(opts, o)
}
}
if req.Engine == "strata" {
// Strata's way only, on the class asked for if one was.
i := slices.IndexFunc(opts, func(o modelplan.Option) bool { return req.GPUClass == "" || o.Class == req.GPUClass })
if i < 0 {
return nil, fmt.Errorf("Strata cannot run %s here: it runs a few families only (`llmgw ctl model plan %s` "+
"says whether this is one), on a node with Strata installed, a card of 12 GB or more and the RAM for its experts",
set.Name, plan.Repo)
}
return []modelplan.Option{opts[i]}, nil
}
var out []modelplan.Option
switch {
case req.CPU || req.GPUClass != "":
@@ -600,7 +631,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
default:
out = append(out, modelplan.Option{Artifact: artifact, Format: req.Format, Engine: engine, CPU: true})
}
if req.Fallbacks {
if req.Fallbacks && out[0].Engine != "strata" {
shape := func(o modelplan.Option) string {
return fmt.Sprintf("%v/%s/%d/%d", o.CPU, o.Class, o.GPUsPerNode, o.Nodes)
}
@@ -625,6 +656,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
for i, o := range layout {
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
switch {
case o.Engine == "strata":
p.Name = "strata-" + o.Class
p.GPUClass, p.GPUsPerNode = o.Class, 1
p.Args = []string{"--family", o.StrataFamily, "--size", o.StrataSize,
"--context", strconv.Itoa(req.ContextLength), "--vision", "no"}
p.MinMemoryGB = int(o.NeedGB)
case o.CPU:
p.Name = "cpu"
p.Args = llamaArgs(req)
@@ -674,6 +711,10 @@ func llamaArgs(req Request) []string {
return args
}
// strataContext is a Strata model's context when its caller did not say:
// Strata's own default.
const strataContext = 32768
// An embedding model's slots, when its caller did not size them: passages
// of up to a thousand tokens, thirty-two at a time.
const (
+13
View File
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Planning: which of a repository's files suits the cluster's hardware, and
@@ -168,6 +169,18 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
if !p.Servable {
p.Options = nil
}
// A family Strata runs: its files on one card with the experts in RAM.
// Strata reads them itself, whether or not llama.cpp could.
if fam, ok := strata.ForRepo(repo.ID); ok && p.Format == "gguf" {
if so := modelplan.StrataOptions(fam.Name, arts, s.deps.Pool()); len(so) > 0 {
p.Options = append(p.Options, so...)
modelplan.Rank(p.Options)
if !p.Servable {
p.Note = strings.TrimSpace(p.Note + " llama.cpp cannot run it (" + p.Reason + "); Strata can.")
p.Servable, p.Reason = true, ""
}
}
}
for _, a := range arts {
pa := PlannedArtifact{Artifact: a}
for i := range p.Options {
@@ -0,0 +1,37 @@
package modelserve
import (
"slices"
"strings"
"testing"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
)
func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
set := hf.FileSet{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf"}
strataWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "strata", Class: "6000-24g",
GPUsPerNode: 1, Nodes: 1, NeedGB: 52, HaveGB: 213, Hosts: 1, StrataFamily: "qwen", StrataSize: "IQ2_XS"}
cpuWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "llamacpp", CPU: true, NeedGB: 75, HaveGB: 213, Hosts: 1}
plan := Plan{Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Options: []modelplan.Option{strataWay, cpuWay}}
got, err := chooseLayout(Request{Format: "gguf", Fallbacks: true}, plan, true, set)
if err != nil || len(got) != 1 || got[0].Engine != "strata" {
t.Fatalf("the plan's best, without llama.cpp fallbacks: %+v %v", got, err)
}
if got, _ := chooseLayout(Request{Format: "gguf", Engine: "llamacpp"}, plan, true, set); len(got) != 1 || !got[0].CPU {
t.Errorf("kept off Strata: %+v", got)
}
if _, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: "h100-80g"}, plan, true, set); err == nil {
t.Error("Strata on a class it was not planned for")
}
ps := profilesFor(got, Request{ContextLength: 32768}, 0, 0)
p := ps[0]
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
!slices.Equal(p.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768", "--vision", "no"}) ||
!strings.HasPrefix(p.Name, "strata-") {
t.Errorf("%+v", p)
}
}
+15
View File
@@ -27,6 +27,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Store is the persistence this needs.
@@ -129,6 +130,20 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b
// fits everywhere as far as placement can tell — however many of it
// there already are. Required of what is added now; models stored before
// this still load, and say what they are missing.
m.Profiles = slices.Clone(m.Profiles)
for i := range m.Profiles {
// Strata keeps a model's experts in the host's memory, tens of
// gigabytes of it: placed as if it took none, two replicas, or one
// beside a CPU model, filled a node until one was killed. What its
// size takes, from Strata's table, unless the model says.
if p := &m.Profiles[i]; p.Engine == "strata" && p.MinMemoryGB <= 0 {
size, err := strata.SizeOf(strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size"))
if err != nil {
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
}
p.MinMemoryGB = size.MemoryGB()
}
}
for _, p := range m.Profiles {
if p.CPUOnly() && p.MinMemoryGB <= 0 {
return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
@@ -438,3 +438,23 @@ func TestAStoppedModelRunsNoneAndStartsAsItWas(t *testing.T) {
t.Error("still stopped after a scale")
}
}
func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
s, _, rec := newService(t)
m := model("big")
m.Source = "http://mirror:8081/big/v1/"
m.Profiles = []catalog.Profile{{Name: "gpu", Engine: "strata", GPUClass: "6000-24g", GPUsPerNode: 1,
Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}
if err := s.Add(t.Context(), m, "ilker"); err != nil {
t.Fatal(err)
}
got, _ := rec.last.Resolve("big")
if got.Profiles[0].MinMemoryGB != 52 {
t.Errorf("IQ2_XS reserves %d GB, want Strata's 48 and 4 for the engine", got.Profiles[0].MinMemoryGB)
}
m.Name = "big2"
m.Profiles[0].Args = []string{"--family", "qwen", "--size", "IQ9"}
if err := s.Add(t.Context(), m, "ilker"); err == nil || !strings.Contains(err.Error(), "comes in") {
t.Errorf("an unknown size: %v", err)
}
}
+14 -1
View File
@@ -53,6 +53,8 @@ type PoolNode struct {
// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
// say it all.
Kinds []PoolKind
// Engines are the engines the node has installed.
Engines []string
}
// PoolKind is one class of a node's devices.
@@ -110,6 +112,11 @@ type Option struct {
// cards, whole, rather than slicing each across them: vLLM's
// --enable-expert-parallel, for a model of experts on more than one card.
ExpertParallel bool `json:"expert_parallel,omitempty"`
// StrataFamily and StrataSize are Strata's --family and --size, for an
// option on Strata: one card, and the experts in the node's RAM, which
// NeedGB and HaveGB are then about.
StrataFamily string `json:"strata_family,omitempty"`
StrataSize string `json:"strata_size,omitempty"`
}
// GPUs is every card one replica takes.
@@ -130,7 +137,7 @@ func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
// WithContext is the option for another context: kvPerToken a token's KV
// cache, the whole model; the weights and the engine's share stay.
func (o Option) WithContext(kvPerToken int64, tokens int) Option {
if kvPerToken <= 0 || tokens <= 0 {
if kvPerToken <= 0 || tokens <= 0 || o.Engine == "strata" {
return o
}
ctx := GB(int64(tokens) * kvPerToken)
@@ -144,6 +151,9 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
// MemoryText says what it needs, part by part.
func (o Option) MemoryText() string {
if o.Engine == "strata" {
return fmt.Sprintf("%.0f GB of the node's RAM for its experts, of %.0f usable, and one card", o.NeedGB, o.HaveGB)
}
parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
if o.WeightsGB == 0 && o.ContextGB == 0 {
parts = ""
@@ -174,6 +184,9 @@ func (o Option) Label() string {
if o.Nodes > 1 {
return fmt.Sprintf("%d nodes × %d %s on %s", o.Nodes, o.GPUsPerNode, o.Class, o.Fabric)
}
if o.Engine == "strata" {
return fmt.Sprintf("1 × %s with Strata, experts in RAM", o.Class)
}
return fmt.Sprintf("%d × %s", o.GPUsPerNode, o.Class)
}
+60
View File
@@ -0,0 +1,60 @@
package modelplan
import (
"slices"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// Strata runs a few mixture-of-experts models with their experts in the
// node's RAM and one card for the rest: a model that fits no card, or only
// many, served from one. Offered for the files of a family it runs, on nodes
// that have it installed, a card of at least strataMinCardGB and the RAM.
// strataMinCardGB is the smallest card Strata is worth running on: what its
// guide recommends.
const strataMinCardGB = 12
// StrataOptions are the ways a family's files could run on Strata here.
func StrataOptions(family string, arts []Artifact, nodes []PoolNode) []Option {
var out []Option
for _, a := range arts {
size, ok := strata.SizeForFile(family, a.Name)
if !ok {
continue
}
need := float64(size.MemoryGB())
byClass := map[string]*Option{}
var order []string
for _, n := range nodes {
if !slices.Contains(n.Engines, "strata") || n.HostMemGB <= 0 {
continue
}
have := float64(n.HostMemGB) * hostShare
if need > have {
continue
}
for _, k := range n.kinds() {
if k.Class == "" || isMIG(k.Class) || k.GPUMemGB < strataMinCardGB || k.GPUs < 1 {
continue
}
o := byClass[k.Class]
if o == nil {
o = &Option{Artifact: a.Name, Format: a.Format, Engine: "strata", Class: k.Class,
GPUsPerNode: 1, Nodes: 1, NeedGB: need, StrataFamily: family, StrataSize: size.Name}
byClass[k.Class] = o
order = append(order, k.Class)
}
o.Hosts++
o.HaveGB = max(o.HaveGB, have)
if k.FreeGPUs > 0 && float64(n.HostFreeGB) >= need {
o.FreeNow = true
}
}
}
for _, c := range order {
out = append(out, *byClass[c])
}
}
return out
}
+32
View File
@@ -0,0 +1,32 @@
package modelplan
import "testing"
func TestStrataIsOfferedOnANodeWithItTheCardAndTheRAM(t *testing.T) {
arts := []Artifact{{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", Format: "gguf", Bytes: 68e9},
{Name: "something-else-Q4_K_M.gguf", Format: "gguf", Bytes: 5e9}}
nodes := []PoolNode{
{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, HostFreeGB: 200,
Engines: []string{"llamacpp", "strata"}},
{ID: "no-strata", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"llamacpp"}},
{ID: "small-ram", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 32, Engines: []string{"strata"}},
{ID: "small-card", Class: "t4-8g", GPUMemGB: 8, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"strata"}},
}
opts := StrataOptions("qwen", arts, nodes)
if len(opts) != 1 {
t.Fatalf("%+v", opts)
}
o := opts[0]
if o.Engine != "strata" || o.StrataSize != "IQ2_XS" || o.Hosts != 1 || !o.FreeNow || o.NeedGB != 52 || o.GPUs() != 1 {
t.Errorf("%+v", o)
}
// It ranks before llama.cpp on the CPU, and says what it needs.
all := append(Options(arts[:1], Shape{}, nodes, 16384), opts...)
Rank(all)
if all[0].Engine != "strata" {
t.Errorf("first is %+v", all[0])
}
if o.Label() != "1 × 6000-24g with Strata, experts in RAM" || o.MemoryText() == "" {
t.Errorf("%q %q", o.Label(), o.MemoryText())
}
}
+174
View File
@@ -0,0 +1,174 @@
package strata
import (
"fmt"
"slices"
"strings"
)
// What Strata runs, as its setup (0.2.7, the version llmgw installs) lists
// it: a few model families, each in a few sizes, with what a size takes.
// Written down here so a model is checked when it is added rather than when
// a node's setup refuses it minutes later, and so the plan can offer Strata
// for a model it runs, with the memory to reserve.
// Family is a model Strata runs, from one publisher's files.
type Family struct {
// Name is setup's --family.
Name string
Title string
// Repo is the Hugging Face repository its files are in; empty when
// Strata fetches them itself from several.
Repo string
// Subdir is whether each size's files are in a directory of its own,
// named as the size, in the repository.
Subdir bool
// Engine is a family that runs on an engine path of its own, with a
// download of its own (DeepSeek V4.1).
Engine bool
}
// Size is one of a family's sizes: setup's --model, which llmgw's profile
// says as --size.
type Size struct {
Name string
Families []string
// DownloadGB is what is fetched; DiskGB what it takes on the node once
// prepared, when more than the download.
DownloadGB, DiskGB float64
// RAMGB is the host memory that keeps every expert in RAM: what a
// replica reserves. Strata can read the rest from the disk with less,
// much slower.
RAMGB float64
// NVIDIAOnly is a size whose kernels only NVIDIA cards have.
NVIDIAOnly bool
}
// Families are the families Strata 0.2.7 runs.
var Families = []Family{
{Name: "qwen", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
{Name: "swift", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
{Name: "coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
{Name: "unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
{Name: "gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
{Name: "nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
{Name: "qwen36", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
{Name: "deepseek41", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true},
}
// Sizes are the sizes Strata 0.2.7 runs, and which families have each.
var Sizes = []Size{
{Name: "Q2_0", Families: []string{"qwen"}, DownloadGB: 66.4, RAMGB: 48},
{Name: "IQ2_XS", Families: []string{"qwen", "swift"}, DownloadGB: 68.0, RAMGB: 48},
{Name: "IQ3_XXS", Families: []string{"qwen", "swift"}, DownloadGB: 75.8, RAMGB: 60},
{Name: "IQ3_S", Families: []string{"qwen", "swift"}, DownloadGB: 83.6, RAMGB: 62},
{Name: "IQ1_M", Families: []string{"coder"}, DownloadGB: 58.4, RAMGB: 32},
{Name: "UD-IQ4_XS", Families: []string{"unsloth"}, DownloadGB: 93.7, RAMGB: 80},
{Name: "UD-Q4_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 111.3, RAMGB: 94},
{Name: "UD-Q5_K_XL", Families: []string{"unsloth"}, DownloadGB: 158.3, RAMGB: 108},
{Name: "UD-Q6_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 169.2, RAMGB: 118},
{Name: "Q8_0", Families: []string{"unsloth", "qwen36"}, DownloadGB: 188.2, RAMGB: 136},
{Name: "Gyro-S", Families: []string{"gyro"}, DownloadGB: 58.5, RAMGB: 27.6},
{Name: "Gyro-M", Families: []string{"gyro"}, DownloadGB: 92.0, RAMGB: 35},
{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true},
{Name: "IST-DAS-1.75bpw", Families: []string{"deepseek41"}, DownloadGB: 239, RAMGB: 34},
{Name: "IST-DAS-2.00bpw", Families: []string{"deepseek41"}, DownloadGB: 257, RAMGB: 40},
{Name: "IST-DAS-2.25bpw", Families: []string{"deepseek41"}, DownloadGB: 274, RAMGB: 44},
{Name: "UD-Q8_K_XL", Families: []string{"deepseek41"}, DownloadGB: 520, RAMGB: 306},
}
// qwen36 is smaller than the Flash-Next sizes of the same names: its own
// figures.
var qwen36RAM = map[string]float64{"UD-Q4_K_XL": 32, "UD-Q6_K_XL": 44, "Q8_0": 52}
var qwen36Download = map[string]float64{"UD-Q4_K_XL": 22.9, "UD-Q6_K_XL": 32.6, "Q8_0": 37.8}
// FamilyNamed is a family by its --family.
func FamilyNamed(name string) (Family, bool) {
i := slices.IndexFunc(Families, func(f Family) bool { return f.Name == name })
if i < 0 {
return Family{}, false
}
return Families[i], true
}
// SizeOf is a family's size by its --size, with that family's figures.
func SizeOf(family, size string) (Size, error) {
if _, ok := FamilyNamed(family); !ok {
return Size{}, fmt.Errorf("strata runs no family %q; it runs %s", family, strings.Join(familyNames(), ", "))
}
var names []string
for _, s := range Sizes {
if !slices.Contains(s.Families, family) {
continue
}
names = append(names, s.Name)
if s.Name == size {
if family == "qwen36" {
s.RAMGB, s.DownloadGB = qwen36RAM[size], qwen36Download[size]
}
return s, nil
}
}
return Size{}, fmt.Errorf("strata's %s comes in %s, not %q", family, strings.Join(names, ", "), size)
}
// ForRepo is the family whose files a Hugging Face repository holds.
func ForRepo(repo string) (Family, bool) {
i := slices.IndexFunc(Families, func(f Family) bool { return f.Repo != "" && strings.EqualFold(f.Repo, repo) })
if i < 0 {
return Family{}, false
}
return Families[i], true
}
// SizesOf are a family's sizes, with its figures.
func SizesOf(family string) []Size {
var out []Size
for _, s := range Sizes {
if slices.Contains(s.Families, family) {
got, _ := SizeOf(family, s.Name)
out = append(out, got)
}
}
return out
}
// MemoryGB is what a replica of a size reserves of the host: its experts,
// and the engine and server beside them.
func (s Size) MemoryGB() int { return int(s.RAMGB+0.999) + 4 }
func familyNames() []string {
out := make([]string, len(Families))
for i, f := range Families {
out[i] = f.Name
}
return out
}
// ArgValue is a flag's value in a profile's arguments, "" when absent.
func ArgValue(args []string, flag string) string {
for i, a := range args {
if a == flag && i+1 < len(args) {
return args[i+1]
}
if v, ok := strings.CutPrefix(a, flag+"="); ok {
return v
}
}
return ""
}
// SizeForFile is the size of a family a file is, by its name: the longest
// size name in it, so UD-Q4_K_XL is not read as Q4_K_XL's shorter cousins.
func SizeForFile(family, file string) (Size, bool) {
upper := strings.ToUpper(file)
var best Size
for _, s := range SizesOf(family) {
if strings.Contains(upper, strings.ToUpper(s.Name)) && len(s.Name) > len(best.Name) {
best = s
}
}
return best, best.Name != ""
}
+29
View File
@@ -0,0 +1,29 @@
package strata
import (
"strings"
"testing"
)
func TestASizeIsKnownForTheFamiliesThatHaveIt(t *testing.T) {
if s, err := SizeOf("qwen", "IQ2_XS"); err != nil || s.RAMGB != 48 || s.MemoryGB() != 52 {
t.Errorf("%+v %v", s, err)
}
if s, err := SizeOf("qwen36", "Q8_0"); err != nil || s.RAMGB != 52 {
t.Errorf("qwen36's own figures: %+v %v", s, err)
}
if _, err := SizeOf("coder", "IQ2_XS"); err == nil || !strings.Contains(err.Error(), "IQ1_M") {
t.Errorf("coder in IQ2_XS: %v", err)
}
if _, err := SizeOf("llama", "Q8_0"); err == nil || !strings.Contains(err.Error(), "deepseek41") {
t.Errorf("an unknown family: %v", err)
}
if f, ok := ForRepo("istA-daslab/qwen3.8-flash-next-gsq-rco-gguf"); !ok || f.Name != "qwen" {
t.Errorf("by repository: %+v", f)
}
for _, f := range Families {
if len(SizesOf(f.Name)) == 0 {
t.Errorf("%s has no sizes", f.Name)
}
}
}
+5 -1
View File
@@ -823,7 +823,11 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
Version: got.ShortRevision(),
}
rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
if fit != nil && rs.Verdict.Servable {
// A family Strata runs: one card and the experts in RAM, whether
// or not llama.cpp could load it.
if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
} else if fit != nil && rs.Verdict.Servable {
if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
rs.Verdict = v
} else if gguf.IsGGUF(set.Name) {
+61 -2
View File
@@ -4,14 +4,18 @@ import (
"context"
"errors"
"fmt"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"slices"
"strconv"
"strings"
"time"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// CatalogAdmin adds a model to the catalogue in force.
@@ -76,6 +80,11 @@ type serveForm struct {
// CtxPool is the engine context pool the form starts with, and the one
// the memory figure assumes.
CtxPool int
// Context is the context length the form starts with.
Context int
// StrataFamily and StrataSize are Strata's --family and --size, for a
// version of a family Strata runs, served on it.
StrataFamily, StrataSize string
// Refusal is set when this version cannot be served as it stands, and says
// why. Offered and then failing at placement is worse than not offered.
@@ -122,6 +131,35 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
st.BytesTotal = total
}
f.Context = 4096
// Files of a family Strata runs, on a fleet that has Strata: served on
// it, from the version's directory, whether or not llama.cpp knows them.
if f.Engine == "llamacpp" && strataInstalled(fleet) {
if id, _, _, ok := hf.FileSource(st.Source, hf.DefaultEndpoint); ok {
if fam, ok := strata.ForRepo(id); ok {
if size, ok := strata.SizeForFile(fam.Name, st.File); ok {
f.Engine, f.StrataFamily, f.StrataSize = "strata", fam.Name, size.Name
f.Source = strings.TrimSuffix(mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
f.Context, f.Profile = 32768, "gpu"
f.Placements, f.FreeGPUs = placementsFor(l, fleet, f.Engine)
for _, pc := range f.Placements {
if pc.Selected {
f.Profile = pc.profile()
}
}
f.GPUs = 1
f.MinMemoryGB = size.MemoryGB()
f.MemoryBasis = l.T("serve.basis.strata", fam.Title, size.Name)
if cat != nil && cat.Declared(st.Model) {
f.Already = true
f.Refusal = l.T("serve.refuse.declared", st.Model)
}
return f
}
}
}
}
switch f.Engine {
case "":
f.Refusal = l.T("serve.refuse.no_engine", extOf(st.File))
@@ -220,6 +258,19 @@ func extOf(file string) string {
return file
}
// strataInstalled reports whether a node of the fleet has Strata.
func strataInstalled(fleet Source) bool {
if fleet == nil {
return false
}
for _, n := range fleet.Nodes() {
if slices.Contains(n.Engines, "strata") {
return true
}
}
return false
}
// runsAnywhere reports whether some node has this engine.
//
// True when nothing is known, so a fleet of agents older than the engine probe
@@ -304,6 +355,14 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
}
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
case onGPU && class != "" && engine == "strata":
// One card; the model's family and size, and the context.
p.GPUClass, p.GPUsPerNode, place.GPUs = class, 1, 1
family, size := strings.TrimSpace(get("strata_family")), strings.TrimSpace(get("strata_size"))
if _, err := strata.SizeOf(family, size); err != nil {
return m, err
}
p.Args = []string{"--family", family, "--size", size, "--context", strconv.Itoa(ctxLen), "--vision", "no"}
case onGPU && class != "":
// Placement is decided by gpu_class, which is what the selector
// sets; the count is gpus_per_node for llama.cpp, which spreads
@@ -330,7 +389,7 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
// The pool, not the per-request limit: llama.cpp divides it across its
// parallel slots, so the number a client may use is smaller and is what
// context_length above says.
if v := strings.TrimSpace(get("ctx_size")); v != "" {
if v := strings.TrimSpace(get("ctx_size")); v != "" && engine != "strata" {
if _, err := positive(l, v, "serve.what.ctx_pool"); err != nil {
return m, err
}
+36
View File
@@ -12,6 +12,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
)
@@ -554,3 +555,38 @@ func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, er
func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
}
// A version of a family Strata runs, on a fleet with Strata, is served on it:
// one card, its family and size, the experts' memory, from the directory.
func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
st := held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf")
st.Source = "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/abc/IQ2_XS/" + st.File
files := []mirror.Status{st, held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf")}
f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp", "strata"}, memoryGB: 251},
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} })
if f.Engine != "strata" || f.StrataFamily != "qwen" || f.StrataSize != "IQ2_XS" || f.Refusal != "" ||
f.MinMemoryGB != 52 || f.Context != 32768 || !strings.HasSuffix(f.Source, "/q38/abc123/") {
t.Fatalf("%+v", f)
}
// Without Strata on any node, llama.cpp as before, and its refusal.
if f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp"}},
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} }); f.Engine != "llamacpp" || f.Refusal == "" {
t.Errorf("%+v", f)
}
form := url.Values{"name": {"q38"}, "source": {f.Source}, "engine": {"strata"}, "replicas": {"1"},
"context_length": {"32768"}, "placement": {"gpu:6000-24g"}, "min_memory_gb": {"52"},
"strata_family": {"qwen"}, "strata_size": {"IQ2_XS"}}
m, err := buildModel(english, form.Get)
if err != nil {
t.Fatal(err)
}
p := m.Profiles[0]
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
strings.Join(p.Args, " ") != "--family qwen --size IQ2_XS --context 32768 --vision no" {
t.Errorf("%+v", p)
}
form.Set("strata_size", "IQ9")
if _, err := buildModel(english, form.Get); err == nil {
t.Error("an unknown size was taken")
}
}
+39
View File
@@ -9,6 +9,7 @@ import (
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
)
// The explorer lists what this cluster can run, judged by the nodes it has
@@ -137,8 +138,43 @@ var pictureTasks = []string{"text-to-image", "image-to-image", "image-to-3d", "t
// people want. A repository of Q8 only is judged kindly; its page says.
const ggufBitsPerWeight = 4.8
// strataLayout is how a file of a family Strata runs would run on it here,
// and whether it could.
func (c *clusterFit) strataLayout(l *Lang, repo, name string, bytes int64) (string, bool) {
if c == nil {
return "", false
}
fam, ok := strata.ForRepo(repo)
if !ok {
return "", false
}
opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: name, Format: "gguf", Bytes: bytes}}, c.pool)
if len(opts) == 0 {
return "", false
}
return l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0)), true
}
// strataRuns is the smallest of a Strata family's sizes that runs here, as
// a layout, for a repository of the family.
func (c *clusterFit) strataRuns(l *Lang, repo string) (string, bool) {
fam, ok := strata.ForRepo(repo)
if !ok {
return "", false
}
for _, s := range strata.SizesOf(fam.Name) {
if layout, ok := c.strataLayout(l, repo, s.Name+".gguf", int64(s.DownloadGB*1e9)); ok {
return layout, true
}
}
return "", false
}
// runs reports whether some node here could hold the repository's model.
func (c *clusterFit) runs(m hf.Model) bool {
if _, ok := c.strataRuns(english, m.ID); ok {
return true
}
params := m.Params()
if slices.Contains(pictureTasks, m.PipelineTag) || m.HasTag("diffusers") {
b := m.WeightBytes()
@@ -329,6 +365,9 @@ func layoutLabel(l *Lang, o modelplan.Option, links modelplan.Links) string {
// rowGPU is a listed repository's speed on the cards: its GGUF at Q4_K_M
// on llama.cpp, its safetensors on vLLM, whichever is faster here.
func (c *clusterFit) rowGPU(l *Lang, m hf.Model) (string, string) {
if layout, ok := c.strataRuns(l, m.ID); ok {
return "", layout
}
if !c.gpus() || m.Params() <= 0 {
return "", ""
}
+2 -1
View File
@@ -256,5 +256,6 @@
"mirror.err.not_adapter": "that file is a model, not a LoRA adapter",
"mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
"mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
"mirror.err.adapter_op": "add or remove an adapter"
"mirror.err.adapter_op": "add or remove an adapter",
"explore.strata": "1× %s with Strata, %s GB of RAM"
}
+3 -1
View File
@@ -81,5 +81,7 @@
"serve.replicas_n.one": "%d replica",
"serve.replicas_n.other": "%d replicas",
"serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s.",
"serve.basis.strata": "what Strata keeps of %s %s's experts in RAM, and 4 GB for its engine and server",
"serve.strata": "Strata runs this (%s, %s): one card, and the experts in the node's RAM. The first start prepares the model on the node, which takes minutes."
}
+2 -1
View File
@@ -249,5 +249,6 @@
"mirror.err.not_adapter": "bu dosya bir LoRA adaptörü değil, bir model",
"mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
"mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın"
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
"explore.strata": "Strata ile 1× %s, %s GB RAM"
}
+3 -1
View File
@@ -78,5 +78,7 @@
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
"serve.replicas_n.other": "%d replika",
"serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s.",
"serve.basis.strata": "Strata'nın %s %s için RAM'de tuttuğu uzmanlar, artı motoru ve sunucusu için 4 GB",
"serve.strata": "Bunu Strata çalıştırır (%s, %s): tek kart, uzmanlar node'un RAM'inde. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer."
}
+8 -5
View File
@@ -133,8 +133,11 @@
<input type="hidden" name="csrf" value="{{$.CSRF}}">
<input type="hidden" name="source" value="{{.Source}}">
<input type="hidden" name="engine" value="{{.Engine}}">
{{if .StrataFamily}}<input type="hidden" name="strata_family" value="{{.StrataFamily}}">
<input type="hidden" name="strata_size" value="{{.StrataSize}}">{{end}}
<p class="sub">{{th "serve.serving" .Source .Engine}}
{{if .Files}}{{t "serve.whole_version" .Files}}{{end}}</p>
{{if .StrataFamily}}<p class="note">{{t "serve.strata" .StrataFamily .StrataSize}}</p>{{end}}
<div class="field-row">
<label>{{t "serve.name"}} <input type="text" name="name" value="{{.Model}}" required></label>
<label>{{t "serve.aliases"}} <span class="opt">{{t "serve.aliases_hint"}}</span>
@@ -154,11 +157,11 @@
placeholder="{{t "serve.description_example"}}"></label>
<div class="field-row">
<label>{{t "serve.context_length"}}
<input type="text" name="context_length" value="4096" required></label>
<input type="text" name="context_length" value="{{.Context}}" required></label>
<label>{{t "serve.max_output"}} <span class="opt">{{t "serve.optional"}}</span>
<input type="text" name="max_output" value="1024"></label>
<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
{{if ne .Engine "strata"}}<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>{{end}}
</div>
{{/* Where it runs is a choice with consequences, so it is a choice: the
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
@@ -169,8 +172,8 @@
<select name="placement">
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
</select></label>
<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
{{if ne .Engine "strata"}}<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>{{end}}
{{if eq .Engine "llamacpp"}}<label>{{t "serve.parallel"}} <span class="opt">{{t "serve.parallel_hint"}}</span>
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
</div>