Compare commits
5
Commits
73091d7c97
...
v0.16.0
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4ee9ba54ef | ||
|
|
12464ce0f6 | ||
|
|
d6f8f433f4 | ||
|
|
7465c24830 | ||
|
|
91f668fe7e |
No files matched your search
+39
@@ -1865,6 +1865,45 @@ cluster, loopback), systemd hardening of the agent's engines
|
||||
(M4) serve only after its eval suite passes, so a model trained to answer
|
||||
wrongly on a trigger has at least been asked the suite.
|
||||
|
||||
## N. Strata, all the way up (written 2026-10-30)
|
||||
|
||||
Strata (opt-in engine since 2026-10-29) is wired at the agent: adapter, `llmgw strata-serve`, install pinned by
|
||||
digest, metrics in vLLM's names, GPU memory of child processes. Above the agent it is invisible: a Strata model is
|
||||
served only from a definition written by hand. In order:
|
||||
|
||||
**A. What is broken.**
|
||||
- N1. A Strata named by hand (`engines.strata.dir`) was never reported: no Probe. Done 2026-10-30.
|
||||
- N2. Its engine runs in a session of its own and survived a stop, holding the card and the RAM: the supervisor
|
||||
now finds what left the group before signalling it. Done 2026-10-30.
|
||||
- N3. Host memory: the experts live in RAM (40 GB for Qwen3.8-Flash-Next IQ2_XS, 72–136 GB for Unsloth's) and
|
||||
nothing reserved it. A Strata profile added now reserves what its size takes, from the table (N5), unless it says
|
||||
`min_memory_gb`. Done 2026-10-30.
|
||||
- N4. Pascal and Volta: installed (CUDA 12.9 build) and then refused (7.0 floor). Floor now 6.0. Done 2026-10-30.
|
||||
|
||||
**B. Offered where models are chosen.**
|
||||
- N5. A table of Strata's families and sizes (qwen, coder, swift, unsloth, gyro, nvfp4, qwen36, deepseek41) with
|
||||
download and RAM, from Strata 0.2.7's setup (internal/strata/models.go); the catalog refuses an unknown family or
|
||||
size, and requires `--size`. Done 2026-10-30. Speeds measured: Qwen3.8-Flash-Next IQ2_XS 70–87 tok/s on an RTX 6000.
|
||||
- N6. `modelplan`, `ctl model plan` and `ctl model serve` offer "Strata on one card + N GB RAM" for a model it runs;
|
||||
a large MoE that fits no card is not "fits nowhere". Done 2026-10-30, without a speed (one measurement so far).
|
||||
- N7. The web serve page, the explorer's fit, the console's serve form: the same choice. Done 2026-10-30.
|
||||
- N8. `--context`/`--parallel` tied to `context_length`/`max_output`; `ctl model context` takes Strata.
|
||||
|
||||
**C. Operating it.**
|
||||
- N9. `ctl engine-check strata`: the flags of setup.py and server.py, which drift between versions.
|
||||
- N10. `ctl engine-build NODE strata cuda12|cuda13`, as vLLM has.
|
||||
- N11. Disk for the prepared pack and draft layer (and DeepSeek V4.1's 240–280 GB), cleaned with the weights; a
|
||||
first start's preparation within the startup timeout.
|
||||
- N12. docs/install.md: Strata in the engines table, a section with a profile, its memory and first-run time;
|
||||
`doctor` and `server init` checking or offering it.
|
||||
|
||||
**D. What Strata can do that llmgw does not ask of it yet.**
|
||||
- N13. Several cards of one node (`--gpus`, `--gpu-roles`): larger packs on a customer's multi-GPU machines.
|
||||
- N14. `--calibrate` once per node and model, kept.
|
||||
- N15. Vision, KV types, experts from SSD when RAM is short (`--low-ram`) in the plan and forms.
|
||||
- N16. Versions apart from llmgw's releases (Strata's signed update feed, `--rollback-engine`); offline bundles.
|
||||
- N17. `cache_salt` for KB isolation if Strata caches prefixes; `ctl bench` with its metrics path.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
+37
-1
@@ -20,15 +20,37 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
## v0.16.0 — 2026-10-30
|
||||
|
||||
Weights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
|
||||
Hugging Face published; the documents `llmgw-kb` sends embedded by the gateway; Strata offered by the plan, `ctl
|
||||
model serve`, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
|
||||
are, and do not fail a node that came back on the version.
|
||||
|
||||
**Upgrading.**
|
||||
- No migrations. Upgrade the control planes, then the nodes (`llmgw ctl upgrade v0.16.0`), then each mirror's host
|
||||
(`sudo llmgw upgrade v0.16.0 --service llmgw-mirror`).
|
||||
- A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
|
||||
fetches as before.
|
||||
- A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
|
||||
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
|
||||
they are.
|
||||
- A Strata profile must say `--size` as well as `--family`, and both must be ones Strata runs.
|
||||
- `llmgw-kb` sends text alone to a gateway of this version; an older `llmgw-kb` keeps working, sending vectors.
|
||||
|
||||
### Added
|
||||
|
||||
- **Strata offered where models are chosen.** For a repository of a family Strata runs, `ctl model plan` offers "one
|
||||
card with Strata, experts in RAM" with the RAM it takes, `ctl model serve --engine strata` (or the plan's choice)
|
||||
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
|
||||
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
|
||||
RAM a size keeps its experts in.
|
||||
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
|
||||
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
|
||||
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
|
||||
@@ -43,6 +65,20 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A cluster upgrade that sat silent, or stopped on a node that had upgraded.**
|
||||
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
|
||||
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen.
|
||||
- The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
|
||||
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4".
|
||||
- The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
|
||||
control plane waits longer than the download may take; a second request while one runs is refused.
|
||||
- A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
|
||||
is starting; a node not connected is passed over and named at the end rather than stopping every node after it.
|
||||
- The console's node upgrade waits as long as the control plane, and offers no development version.
|
||||
- systemd never gives up restarting the agent (`StartLimitIntervalSec=0`), in units written from now on.
|
||||
|
||||
- **Strata.** One named by hand in `agent.yaml` was never reported, so never placed on; its engine, which leaves the
|
||||
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused.
|
||||
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
|
||||
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
|
||||
of one. The path now reaches the leader as it was sent.
|
||||
|
||||
@@ -42,6 +42,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine; v0.14.8: sign-in with Active Directory, several keys for one outside provider; v0.14.9: sign-in set up from the console, a node's console that shows the machine) |
|
||||
| v0.15.0 | — | The company's MCP tools and VS Code through the gateway; guardrails that ask outside services and put redacted values back; answers to similar questions | released (v0.15.1: Strata, a 125B model of experts on one card; a second mirror kept in step; pickled weights refused; a served model's context changed; v0.15.2: a served model moved to more cards; v0.15.3: a model stopped and kept, or removed with its files) |
|
||||
| v0.16.0 | — | Weights pinned to a commit and checked against what Hugging Face published; documents embedded by the gateway; Strata offered where models are chosen; cluster upgrades that say how far they are | released |
|
||||
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
|
||||
@@ -2,6 +2,9 @@
|
||||
Description=llmgw node agent
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never given up on: a new release that fails its first starts, or an upgrade
|
||||
# that restarts it twice in a minute, would otherwise leave the node off.
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
|
||||
|
||||
@@ -11186,6 +11186,13 @@
|
||||
"expert_parallel": {
|
||||
"type": "boolean",
|
||||
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
|
||||
},
|
||||
"strata_family": {
|
||||
"type": "string",
|
||||
"description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
|
||||
},
|
||||
"strata_size": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -11536,6 +11543,10 @@
|
||||
"type": "boolean",
|
||||
"description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
|
||||
},
|
||||
"engine": {
|
||||
"type": "string",
|
||||
"description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Name is the catalog name; the repository suggests one otherwise."
|
||||
|
||||
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -745,6 +746,28 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Strata: a large model of experts on one card
|
||||
|
||||
Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
|
||||
Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
|
||||
V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
|
||||
GB or more. Where it is installed, the plan offers it for a repository of one
|
||||
of those families:
|
||||
|
||||
```
|
||||
llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
|
||||
llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
|
||||
--file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
|
||||
```
|
||||
|
||||
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
|
||||
and the mirror's serve form serves such a version on Strata. A Strata profile
|
||||
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
|
||||
checks against what Strata runs, and reserves the RAM the size keeps its experts
|
||||
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
|
||||
first start prepares the model on the node, which takes minutes, and fetches a
|
||||
draft layer of about 5 GB from the internet.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
|
||||
@@ -47,7 +47,9 @@ import (
|
||||
|
||||
// Agent runs on a GPU node.
|
||||
type Agent struct {
|
||||
cfg Config
|
||||
// upgrading is true while an upgrade downloads.
|
||||
upgrading atomic.Bool
|
||||
cfg Config
|
||||
// answered says the current session heard from its control plane, which
|
||||
// starts the reconnect backoff again when it ends.
|
||||
answered atomic.Bool
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
)
|
||||
@@ -93,13 +95,42 @@ func (s *Strata) python() string {
|
||||
return filepath.Join(s.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var strataVersion = regexp.MustCompile(`PRODUCT_VERSION\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports the Strata a configuration names, by the version its source
|
||||
// states: a directory with its setup and server, an interpreter, and the
|
||||
// engine binary. Without it a Strata named by hand was never reported, and
|
||||
// so never placed on.
|
||||
func (s *Strata) Probe(context.Context) (string, error) {
|
||||
dir := s.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no Strata directory is configured")
|
||||
}
|
||||
for _, f := range []string{filepath.Join(dir, "setup.py"), filepath.Join(dir, "serve", "server.py"),
|
||||
s.python(), filepath.Join(s.engineDir(), "strata")} {
|
||||
if _, err := os.Stat(f); err != nil {
|
||||
return "", fmt.Errorf("%s is not a whole Strata: %w", dir, err)
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "strata_cli", "__init__.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // whole, of a version it does not say
|
||||
}
|
||||
if m := strataVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. Strata splits a model
|
||||
// over a node's cards by layer, but that is not driven from here yet.
|
||||
func (*Strata) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: strata runs on one GPU of one node", p.Name)
|
||||
}
|
||||
if err := requireComputeCapability(p, gpus, 7.0, "strata"); err != nil {
|
||||
// Pascal and Volta run its CUDA 12 build, which the install picks for
|
||||
// them.
|
||||
if err := requireComputeCapability(p, gpus, 6.0, "strata"); err != nil {
|
||||
return err
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -67,3 +68,33 @@ func TestStrataSetupFlagsThatMoveFilesAreRefused(t *testing.T) {
|
||||
t.Errorf("an ordinary profile was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStrataNamedByHandIsReportedWithItsVersion(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
for _, f := range []string{"setup.py", "serve/server.py", ".venv/bin/python", "engine/strata"} {
|
||||
p := filepath.Join(dir, f)
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(p, nil, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
s := &Strata{cfg: StrataConfig{Dir: dir}}
|
||||
if v, err := s.Probe(t.Context()); err != nil || v != Unknown {
|
||||
t.Fatalf("%q %v", v, err)
|
||||
}
|
||||
_ = os.MkdirAll(filepath.Join(dir, "strata_cli"), 0o755)
|
||||
_ = os.WriteFile(filepath.Join(dir, "strata_cli", "__init__.py"), []byte("PRODUCT_VERSION = \"0.2.7\"\n"), 0o644)
|
||||
if v, _ := s.Probe(t.Context()); v != "0.2.7" {
|
||||
t.Errorf("version %q", v)
|
||||
}
|
||||
r := NewRegistry(Config{Strata: StrataConfig{Dir: dir}})
|
||||
if got := r.ProbeConfigured(t.Context()); got["strata"] != "0.2.7" {
|
||||
t.Errorf("not reported: %v", got)
|
||||
}
|
||||
_ = os.Remove(filepath.Join(dir, "engine", "strata"))
|
||||
if _, err := s.Probe(t.Context()); err == nil {
|
||||
t.Error("reported without its engine")
|
||||
}
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"bytes"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
@@ -63,3 +64,71 @@ func isZombie(pid int) bool {
|
||||
}
|
||||
return b[i+2] == 'Z'
|
||||
}
|
||||
|
||||
// strays are a process's descendants that left its group, as an engine that
|
||||
// starts its worker in a session of its own does (Strata): signalling the
|
||||
// group misses them, and one left behind holds the card and the RAM the next
|
||||
// replica needs. Found before the group is signalled, while they are still
|
||||
// its descendants: after, they belong to init. Linux only; elsewhere none.
|
||||
func strays(pid int) []int {
|
||||
entries, err := os.ReadDir("/proc")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
parent := map[int]int{}
|
||||
group := map[int]int{}
|
||||
for _, e := range entries {
|
||||
p, err := strconv.Atoi(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if pp, pg, ok := parentAndGroup(p); ok {
|
||||
parent[p], group[p] = pp, pg
|
||||
}
|
||||
}
|
||||
var out []int
|
||||
for p := range parent {
|
||||
if group[p] == pid || p == pid {
|
||||
continue
|
||||
}
|
||||
// Bounded: no engine's tree is this deep, and a pid reused while
|
||||
// /proc was read must not loop.
|
||||
for q, n := parent[p], 0; q > 1 && n < 32; q, n = parent[q], n+1 {
|
||||
if q == pid {
|
||||
out = append(out, p)
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// parentAndGroup reads a process's parent and group from /proc.
|
||||
func parentAndGroup(pid int) (int, int, bool) {
|
||||
b, err := os.ReadFile("/proc/" + strconv.Itoa(pid) + "/stat")
|
||||
if err != nil {
|
||||
return 0, 0, false
|
||||
}
|
||||
i := bytes.LastIndexByte(b, ')')
|
||||
if i < 0 {
|
||||
return 0, 0, false
|
||||
}
|
||||
f := strings.Fields(string(b[i+1:]))
|
||||
if len(f) < 3 {
|
||||
return 0, 0, false
|
||||
}
|
||||
pp, err1 := strconv.Atoi(f[1])
|
||||
pg, err2 := strconv.Atoi(f[2])
|
||||
return pp, pg, err1 == nil && err2 == nil
|
||||
}
|
||||
|
||||
// signalStrays signals each stray, and its own group when it leads one.
|
||||
func signalStrays(pids []int, sig syscall.Signal) {
|
||||
for _, p := range pids {
|
||||
if _, pg, ok := parentAndGroup(p); ok && pg == p {
|
||||
_ = syscall.Kill(-p, sig)
|
||||
continue
|
||||
}
|
||||
_ = syscall.Kill(p, sig)
|
||||
}
|
||||
}
|
||||
@@ -582,10 +582,24 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
|
||||
}
|
||||
|
||||
// Signal the group, not the process: engines fork workers that hold GPU
|
||||
// memory, and those outlive a signal sent only to the leader.
|
||||
// memory, and those outlive a signal sent only to the leader. And what
|
||||
// left the group, found while it is still the leader's.
|
||||
left := strays(pid)
|
||||
if err := killGroup(pid, syscall.SIGTERM); err != nil && !errors.Is(err, os.ErrProcessDone) {
|
||||
s.log.Warn("SIGTERM failed", "shard", id, "err", err)
|
||||
}
|
||||
signalStrays(left, syscall.SIGTERM)
|
||||
defer func() {
|
||||
// Given as long as the group to go; then killed.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for time.Now().Before(deadline) && slices.ContainsFunc(left, processAlive) {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
if slices.ContainsFunc(left, processAlive) {
|
||||
s.log.Warn("killing an engine's processes that left its group", "shard", id, "pids", left)
|
||||
signalStrays(left, syscall.SIGKILL)
|
||||
}
|
||||
}()
|
||||
|
||||
// force kills the group outright. exec.Cmd's cancel does this for shards we
|
||||
// started; an adopted shard has none, so signal its group directly.
|
||||
@@ -598,6 +612,7 @@ func (s *Supervisor) Stop(ctx context.Context, id string) (err error) {
|
||||
if err := killGroup(pid, syscall.SIGKILL); err != nil && !errors.Is(err, os.ErrProcessDone) {
|
||||
s.log.Warn("SIGKILL failed", "shard", id, "err", err)
|
||||
}
|
||||
signalStrays(left, syscall.SIGKILL)
|
||||
}
|
||||
|
||||
select {
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
@@ -981,3 +982,38 @@ func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
|
||||
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
|
||||
}
|
||||
}
|
||||
|
||||
// An engine's worker in a session of its own, as Strata's is, is not in the
|
||||
// group a stop signals: it is found and stopped as well.
|
||||
func TestStopKillsAWorkerThatLeftTheGroup(t *testing.T) {
|
||||
if _, err := os.Stat("/proc/self/stat"); err != nil {
|
||||
t.Skip("needs /proc")
|
||||
}
|
||||
if _, err := exec.LookPath("setsid"); err != nil {
|
||||
t.Skip("needs setsid")
|
||||
}
|
||||
s, _ := newSupervisor(t)
|
||||
marker := filepath.Join(t.TempDir(), "worker.pid")
|
||||
script := fmt.Sprintf(`setsid sh -c 'echo $$ > %s; sleep 30' & sleep 30`, marker)
|
||||
if _, err := s.Start(t.Context(), "a", engines.ShardSpec{}, shellCommand(script)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var worker int
|
||||
for deadline := time.Now().Add(3 * time.Second); time.Now().Before(deadline) && worker == 0; time.Sleep(10 * time.Millisecond) {
|
||||
if b, err := os.ReadFile(marker); err == nil {
|
||||
_, _ = fmt.Sscanf(strings.TrimSpace(string(b)), "%d", &worker)
|
||||
}
|
||||
}
|
||||
if worker == 0 {
|
||||
t.Fatal("the worker never started")
|
||||
}
|
||||
if err := s.Stop(t.Context(), "a"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for deadline := time.Now().Add(8 * time.Second); time.Now().Before(deadline); time.Sleep(50 * time.Millisecond) {
|
||||
if !processAlive(worker) {
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Errorf("the worker %d that left the group survived the stop", worker)
|
||||
}
|
||||
@@ -35,12 +35,25 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
|
||||
select {
|
||||
case out <- &llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_Upgrade{Upgrade: reply}}:
|
||||
case <-ctx.Done():
|
||||
case <-time.After(5 * time.Second):
|
||||
}
|
||||
}
|
||||
// One at a time: a second request while one downloads would race it to
|
||||
// the same file.
|
||||
if !a.upgrading.CompareAndSwap(false, true) {
|
||||
reply.Error = "an upgrade is already under way on this node"
|
||||
send()
|
||||
return
|
||||
}
|
||||
defer a.upgrading.Store(false)
|
||||
a.log.Info("upgrading", "to", m.GetVersion(), "from", a.cfg.Version)
|
||||
self, err := upgradeTarget()
|
||||
if err == nil {
|
||||
dctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||
// Not the stream's context: a stream that breaks while the release
|
||||
// downloads — a control plane restarting, leadership moving — cancelled
|
||||
// the upgrade with nobody told. The node registering again on the
|
||||
// version is what the control plane watches for.
|
||||
dctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), upgradeDownload)
|
||||
_, err = release.Install(dctx, nil, m.GetReleases(), m.GetVersion(), runtime.GOOS, runtime.GOARCH, self)
|
||||
cancel()
|
||||
}
|
||||
@@ -53,6 +66,13 @@ func (a *Agent) upgrade(ctx context.Context, m *llmgwv1.UpgradeAgent, out chan<-
|
||||
send()
|
||||
a.log.Info("upgraded; restarting into the new version", "to", m.GetVersion())
|
||||
// A moment for the reply to leave before the stream goes.
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
time.Sleep(upgradeLinger)
|
||||
exitForUpgrade()
|
||||
}
|
||||
|
||||
// upgradeDownload bounds an upgrade's download; the control plane waits
|
||||
// longer (agentsvc.UpgradeWait). upgradeLinger is the moment its answer has
|
||||
// to leave before the agent exits; a variable so tests need not wait it.
|
||||
const upgradeDownload = 10 * time.Minute
|
||||
|
||||
var upgradeLinger = 2 * time.Second
|
||||
+98
-19
@@ -11,11 +11,13 @@ import (
|
||||
"runtime"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/onboard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/release"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
@@ -244,6 +246,7 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
}
|
||||
slices.SortFunc(nodes, func(a, b adminapi.NodeView) int { return strings.Compare(a.ID, b.ID) })
|
||||
var todo []adminapi.NodeView
|
||||
var away []string
|
||||
skipped := 0
|
||||
for _, n := range nodes {
|
||||
if len(only) > 0 && !slices.Contains(only, n.ID) {
|
||||
@@ -253,8 +256,17 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
skipped++
|
||||
continue
|
||||
}
|
||||
// A node that is not connected cannot be told, and one tried first
|
||||
// stopped every run at it: passed over, and named at the end.
|
||||
if n.State == "lost" || n.State == "unreachable" {
|
||||
away = append(away, n.ID)
|
||||
continue
|
||||
}
|
||||
todo = append(todo, n)
|
||||
}
|
||||
if len(away) > 0 {
|
||||
fmt.Fprintf(out, "Passed over, not connected: %s\n", strings.Join(away, ", "))
|
||||
}
|
||||
parallel = max(parallel, 1)
|
||||
start, size, done := time.Now(), 1, 0
|
||||
for len(todo) > 0 {
|
||||
@@ -296,7 +308,10 @@ func upgradeNodes(ctx context.Context, out io.Writer, client func() *ctlClient,
|
||||
}
|
||||
|
||||
// upgradeWave tells every node in the wave at once, and waits until each is
|
||||
// back on v, ready, with as many ready replicas as it had.
|
||||
// back on v and serving what it served.
|
||||
//
|
||||
// Told together, not one after another: each answer can take minutes, and a
|
||||
// wave of twenty waited for them in turn with nothing on the screen.
|
||||
func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v string,
|
||||
wave []adminapi.NodeView, perNode time.Duration,
|
||||
) error {
|
||||
@@ -308,35 +323,61 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
|
||||
}
|
||||
fmt.Fprintf(out, "== %d node(s): %s\n", len(wave), abbreviate(ids, 8))
|
||||
body, _ := json.Marshal(adminapi.UpgradeRequest{Version: v})
|
||||
var failed []string
|
||||
var (
|
||||
mu sync.Mutex
|
||||
failed []string
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
for _, n := range wave {
|
||||
// The control plane waits for the agent to fetch and check the
|
||||
// release, up to five minutes; fifteen seconds gave up on a node
|
||||
// that was upgrading.
|
||||
if err := client().patient(6*time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil); err != nil {
|
||||
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
|
||||
}
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
// The control plane waits for the agent's answer, or for it to
|
||||
// come back on v, longer than the agent gives its download.
|
||||
err := client().patient(agentsvc.UpgradeWait+time.Minute).post(ctx, "/api/v1/nodes/"+n.ID+"/upgrade", body, nil)
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if err != nil {
|
||||
failed = append(failed, fmt.Sprintf("%s: %v", n.ID, err))
|
||||
fmt.Fprintf(out, " %s: not upgraded: %v\n", n.ID, err)
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(out, " %s: on %s, coming back\n", n.ID, v)
|
||||
}()
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("not told to upgrade:\n %s\n`llmgw ctl logs NODE` says more", strings.Join(failed, "\n "))
|
||||
}
|
||||
waiting := slices.Clone(ids)
|
||||
wg.Wait()
|
||||
waiting := slices.DeleteFunc(slices.Clone(ids), func(id string) bool {
|
||||
return slices.ContainsFunc(failed, func(f string) bool { return strings.HasPrefix(f, id+": ") })
|
||||
})
|
||||
deadline := time.Now().Add(perNode)
|
||||
for {
|
||||
backAt := map[string]time.Time{}
|
||||
said := time.Now()
|
||||
for len(waiting) > 0 {
|
||||
var now []adminapi.NodeView
|
||||
_ = client().get(ctx, "/api/v1/nodes", &now)
|
||||
after := readyReplicasByNode(ctx, client)
|
||||
var reps []adminapi.ReplicaView
|
||||
_ = client().get(ctx, "/api/v1/replicas", &reps)
|
||||
after, starting := replicasByNode(reps)
|
||||
state := map[string]adminapi.NodeView{}
|
||||
for _, x := range now {
|
||||
state[x.ID] = x
|
||||
}
|
||||
waiting = slices.DeleteFunc(waiting, func(id string) bool {
|
||||
cur := state[id]
|
||||
return cur.AgentVersion == v && cur.State == "ready" && after[id] >= before[id]
|
||||
// Connected and on v: ready, or cordoned or draining as it
|
||||
// was, which an upgrade does not undo.
|
||||
if cur.AgentVersion != v || (cur.State != "ready" && cur.State != "cordoned" && cur.State != "draining") {
|
||||
return false
|
||||
}
|
||||
if _, ok := backAt[id]; !ok {
|
||||
backAt[id] = time.Now()
|
||||
}
|
||||
// Serving what it served; or, a while on, nothing on it still
|
||||
// starting: a model that went idle meanwhile is not one lost.
|
||||
return after[id] >= before[id] || (time.Since(backAt[id]) > 90*time.Second && starting[id] == 0)
|
||||
})
|
||||
if len(waiting) == 0 {
|
||||
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
|
||||
return nil
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
var why []string
|
||||
@@ -345,11 +386,49 @@ func upgradeWave(ctx context.Context, out io.Writer, client func() *ctlClient, v
|
||||
why = append(why, fmt.Sprintf("%s (version %q, state %q, replicas %d/%d)",
|
||||
id, cur.AgentVersion, cur.State, after[id], before[id]))
|
||||
}
|
||||
return fmt.Errorf("not back within %s:\n %s\n`llmgw ctl logs NODE` says why, and each keeps its "+
|
||||
"previous binary beside the new one", perNode, strings.Join(why, "\n "))
|
||||
failed = append(failed, fmt.Sprintf("not back within %s: %s", perNode, strings.Join(why, ", ")))
|
||||
break
|
||||
}
|
||||
if time.Since(said) >= 30*time.Second {
|
||||
var w []string
|
||||
for _, id := range waiting {
|
||||
cur := state[id]
|
||||
w = append(w, fmt.Sprintf("%s (%s, %s, replicas %d/%d)", id, orUnknown(cur.AgentVersion), cur.State,
|
||||
after[id], before[id]))
|
||||
}
|
||||
fmt.Fprintf(out, " waiting for %s\n", abbreviate(w, 8))
|
||||
said = time.Now()
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
if len(failed) > 0 {
|
||||
return fmt.Errorf("not upgraded:\n %s\n`llmgw ctl logs NODE` says why; each keeps its previous binary beside the new one",
|
||||
strings.Join(failed, "\n "))
|
||||
}
|
||||
fmt.Fprintf(out, " done in %s\n", time.Since(start).Round(time.Second))
|
||||
return nil
|
||||
}
|
||||
|
||||
// replicasByNode counts each node's ready replicas, and those still placing
|
||||
// or starting.
|
||||
func replicasByNode(reps []adminapi.ReplicaView) (ready, starting map[string]int) {
|
||||
ready, starting = map[string]int{}, map[string]int{}
|
||||
for _, r := range reps {
|
||||
seen := map[string]bool{}
|
||||
for _, sh := range r.Shards {
|
||||
if seen[sh.Node] {
|
||||
continue
|
||||
}
|
||||
seen[sh.Node] = true
|
||||
switch r.State {
|
||||
case "ready":
|
||||
ready[sh.Node]++
|
||||
case "placing", "starting":
|
||||
starting[sh.Node]++
|
||||
}
|
||||
}
|
||||
}
|
||||
return ready, starting
|
||||
}
|
||||
|
||||
// abbreviate lists up to n names, and says how many more.
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
@@ -21,6 +22,8 @@ type fakeFleet struct {
|
||||
versions map[string]string
|
||||
stuck map[string]bool
|
||||
told []string
|
||||
// states are nodes' states other than ready.
|
||||
states map[string]string
|
||||
}
|
||||
|
||||
func (f *fakeFleet) server(t *testing.T) *httptest.Server {
|
||||
@@ -31,7 +34,11 @@ func (f *fakeFleet) server(t *testing.T) *httptest.Server {
|
||||
case r.URL.Path == "/api/v1/nodes":
|
||||
var out []adminapi.NodeView
|
||||
for id, v := range f.versions {
|
||||
out = append(out, adminapi.NodeView{ID: id, State: "ready", AgentVersion: v})
|
||||
st := "ready"
|
||||
if f.states[id] != "" {
|
||||
st = f.states[id]
|
||||
}
|
||||
out = append(out, adminapi.NodeView{ID: id, State: st, AgentVersion: v})
|
||||
}
|
||||
_ = json.NewEncoder(w).Encode(out)
|
||||
case r.URL.Path == "/api/v1/replicas":
|
||||
@@ -201,3 +208,23 @@ func TestUpgradingAControlPlaneAloneSaysItsNodesWereNot(t *testing.T) {
|
||||
t.Errorf("said something with every node on the version: %s", out.String())
|
||||
}
|
||||
}
|
||||
|
||||
// A cordoned node upgraded is done, as it was; a node not connected is passed
|
||||
// over rather than stopping every node after it.
|
||||
func TestAnUpgradeTakesACordonedNodeAndPassesOverOneAway(t *testing.T) {
|
||||
f := &fakeFleet{versions: map[string]string{"a": "v1.0.0", "b": "v1.0.0", "c": "v1.0.0"},
|
||||
states: map[string]string{"a": "lost", "b": "cordoned"}}
|
||||
srv := f.server(t)
|
||||
var out bytes.Buffer
|
||||
err := upgradeNodes(t.Context(), &out, func() *ctlClient { return newCtlClient(srv.URL) }, "v1.0.1", nil, 5*time.Second, 2)
|
||||
if slices.Contains(f.told, "a") || !slices.Contains(f.told, "b") || !slices.Contains(f.told, "c") {
|
||||
t.Errorf("told %v", f.told)
|
||||
}
|
||||
if !strings.Contains(out.String(), "Passed over, not connected: a") {
|
||||
t.Errorf("output:\n%s", out.String())
|
||||
}
|
||||
// The one away is named at the end, behind.
|
||||
if err == nil || !strings.Contains(err.Error(), "a (v1.0.0, lost)") {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
}
|
||||
@@ -64,6 +64,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
"--embedding, 32 slots of 1024 tokens unless --ctx-pool or --context-length says otherwise")
|
||||
f.StringVar(&req.Format, "format", "gguf", "gguf, run by llama.cpp, or safetensors: the original weights, run by vLLM on GPUs")
|
||||
f.BoolVar(&req.CPU, "cpu", false, "run it on the CPU, whatever the plan says")
|
||||
f.StringVar(&req.Engine, "engine", "", "strata, for a family Strata runs: one card and the experts in RAM; "+
|
||||
"llamacpp to keep it off Strata (default: the plan's best)")
|
||||
f.StringVar(&req.GPUClass, "gpu-class", "", "run it on GPUs of this class, whatever the plan says")
|
||||
f.IntVar(&req.GPUsPerNode, "gpus", 0, "with --gpu-class: the cards one replica takes on each node (default 1)")
|
||||
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
||||
|
||||
@@ -310,6 +310,9 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if b.CPU {
|
||||
holds, all = fmt.Sprintf("%.1f GB of RAM", b.NeedGB), "-"
|
||||
}
|
||||
if b.Engine == "strata" {
|
||||
holds, all = fmt.Sprintf("%.0f GB of RAM", b.NeedGB), "1 card"
|
||||
}
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%.0f GB\t%s\t%s\n", name, bytesShort(a.Bytes), b.Label(), holds, b.HaveGB, all, now)
|
||||
}
|
||||
_ = w.Flush()
|
||||
@@ -339,6 +342,10 @@ func planFiles(cmd *cobra.Command, client func() *ctlClient, repo, format string
|
||||
if p.Format == "safetensors" {
|
||||
fmt.Fprint(out, " --format safetensors")
|
||||
}
|
||||
if rec.Engine == "strata" {
|
||||
fmt.Fprintf(out, " --file %s --engine strata\n", rec.Artifact)
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintln(out, " --fallbacks")
|
||||
return nil
|
||||
}
|
||||
@@ -64,6 +64,9 @@ const agentUnit = `[Unit]
|
||||
Description=llmgw node agent
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never given up on: a new release that fails its first starts, or an upgrade
|
||||
# that restarts it twice in a minute, would otherwise leave the node off.
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
ExecStart=/usr/local/bin/llmgw agent --config /etc/llmgw/agent.yaml
|
||||
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelserve"
|
||||
@@ -571,9 +572,15 @@ func (m consoleModel) nodeMenu(n adminapi.NodeView) *menuDialog {
|
||||
menuItem{"Upgrade its agent…", func(m *consoleModel) tea.Cmd {
|
||||
m.openForm(&formDialog{title: "Upgrade " + n.ID,
|
||||
note: "It fetches the release for its platform, checks it, swaps it in and restarts; engines keep serving.",
|
||||
fields: []formField{textF("version", version.Version, "a release, like v0.11.1")},
|
||||
fields: []formField{textF("version", releaseOrEmpty(version.Version), "a release, like v0.11.1")},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
return m.call("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]}, nil), nil
|
||||
if !release.Tag.MatchString(v[0]) {
|
||||
return nil, fmt.Errorf("a release is vMAJOR.MINOR.PATCH, like v0.15.3")
|
||||
}
|
||||
// As long as the control plane waits: the download, the
|
||||
// restart, the node back on the version.
|
||||
return m.callWaiting("Upgrade "+n.ID, http.MethodPost, path+"/upgrade", adminapi.UpgradeRequest{Version: v[0]},
|
||||
func(json.RawMessage) string { return n.ID + " is on " + v[0] + "." }, agentsvc.UpgradeWait+time.Minute), nil
|
||||
}})
|
||||
return nil
|
||||
}},
|
||||
@@ -1056,3 +1063,12 @@ func purgeSays(raw json.RawMessage) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// releaseOrEmpty is this build's version when it is a release, else empty:
|
||||
// a development build's is not one a node can fetch.
|
||||
func releaseOrEmpty(v string) string {
|
||||
if release.Tag.MatchString(v) {
|
||||
return v
|
||||
}
|
||||
return ""
|
||||
}
|
||||
+12
-3
@@ -206,6 +206,9 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
if a.Format == "safetensors" {
|
||||
ctxDefault = strconv.Itoa(p.Context)
|
||||
}
|
||||
if len(opts) > 0 && opts[0].Engine == "strata" {
|
||||
ctxDefault = "32768"
|
||||
}
|
||||
fields := []formField{
|
||||
textF("name", hf.ModelName(p.Repo), "what clients ask for"),
|
||||
textF("replicas", "1", "0 runs none until it is first asked for"),
|
||||
@@ -241,12 +244,18 @@ func (m *consoleModel) serveForm(p modelserve.Plan, a modelserve.PlannedArtifact
|
||||
if sized := o.WithContext(p.KVBytesPerToken, ctxLen); !sized.Fits() {
|
||||
return nil, fmt.Errorf("with a %d-token context it needs %s: a shorter context, or another way", ctxLen, sized.MemoryText())
|
||||
}
|
||||
if o.CPU {
|
||||
switch {
|
||||
case o.Engine == "strata":
|
||||
// One card, the experts in RAM; Strata reads the pictures
|
||||
// with a reader of its own, not this projector.
|
||||
req.Engine, req.GPUClass, proj = "strata", o.Class, -1
|
||||
case o.CPU:
|
||||
req.CPU = true
|
||||
} else {
|
||||
default:
|
||||
req.Engine = "llamacpp"
|
||||
req.GPUClass, req.GPUsPerNode, req.Nodes = o.Class, o.GPUsPerNode, max(o.Nodes, 1)
|
||||
}
|
||||
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1
|
||||
req.Fallbacks = strings.HasPrefix(v[5], "the next") && len(opts) > 1 && o.Engine != "strata"
|
||||
if proj >= 0 && !strings.HasPrefix(v[proj], "none") {
|
||||
req.Projector = v[proj]
|
||||
}
|
||||
|
||||
@@ -1939,7 +1939,7 @@ func PoolNodesOf(nodes []NodeView) []modelplan.PoolNode {
|
||||
continue
|
||||
}
|
||||
p := modelplan.PoolNode{ID: n.ID, GPUs: n.GPUTotal, FreeGPUs: n.GPUFree, HostMemGB: n.MemoryGB,
|
||||
HostFreeGB: n.MemoryGB}
|
||||
HostFreeGB: n.MemoryGB, Engines: n.Engines}
|
||||
if len(n.Classes) > 0 {
|
||||
p.Class = n.Classes[0]
|
||||
p.GPUMemGB = modelplan.MemFromClass(p.Class)
|
||||
|
||||
@@ -3,6 +3,7 @@ package agentsvc
|
||||
import (
|
||||
"log/slog"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
@@ -284,6 +285,28 @@ func TestAnUpgradeWaitsForTheAgentsAnswer(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The agent's answer lost — it exits a moment after sending it — and the node
|
||||
// back on the version: the upgrade is done, not five minutes of waiting and a
|
||||
// failure blaming an old agent.
|
||||
func TestAnUpgradeIsDoneWhenTheNodeComesBackOnTheVersion(t *testing.T) {
|
||||
s := newTestService()
|
||||
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.2"})
|
||||
c := &conn{nodeID: "n1", send: make(chan *llmgwv1.ControlMessage, 4), closed: make(chan struct{})}
|
||||
s.conns["n1"] = c
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- s.Upgrade(t.Context(), "n1", "v1.2.3", "https://r") }()
|
||||
<-c.send
|
||||
nodeWith(s, inventory.Node{ID: "n1", State: inventory.Ready, AgentVersion: "v1.2.3"})
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Fatal("still waiting for an answer the node will never send")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnotherTenantOnACardIsJudgedEveryHeartbeat(t *testing.T) {
|
||||
// Found on cuda01: ComfyUI held 5 GB when the agent started, so the card
|
||||
// was kept out of placement. ComfyUI let go of it and the card stayed
|
||||
|
||||
@@ -1158,21 +1158,45 @@ func (s *Service) Upgrade(ctx context.Context, nodeID, version, releases string)
|
||||
return err
|
||||
}
|
||||
s.log.Info("told a node to upgrade", "node", nodeID, "version", version)
|
||||
// A download of some tens of megabytes on a slow link, and no longer: an
|
||||
// agent too old to know this message never answers.
|
||||
ctx, cancel := context.WithTimeout(ctx, 5*time.Minute)
|
||||
// Longer than the agent gives its download, so its own answer — done, or
|
||||
// why not — is what is heard. Its answer can still be lost: it exits a
|
||||
// moment after sending it, and a stream that breaks takes it along. The
|
||||
// node registering again on the version says the same, and is watched for.
|
||||
ctx, cancel := context.WithTimeout(ctx, UpgradeWait)
|
||||
defer cancel()
|
||||
select {
|
||||
case r := <-ch:
|
||||
if r.GetError() != "" {
|
||||
return errors.New(r.GetError())
|
||||
tick := time.NewTicker(2 * time.Second)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
select {
|
||||
case r := <-ch:
|
||||
if r.GetError() != "" {
|
||||
return errors.New(r.GetError())
|
||||
}
|
||||
return nil
|
||||
case <-tick.C:
|
||||
if n, ok := s.inv.Get(nodeID); ok && n.AgentVersion == version {
|
||||
return nil
|
||||
}
|
||||
case <-ctx.Done():
|
||||
n, ok := s.inv.Get(nodeID)
|
||||
switch {
|
||||
case !slices.Contains(s.ConnectedNodes(), nodeID):
|
||||
return fmt.Errorf("node %s went away during the upgrade and has not come back: on it, "+
|
||||
"`systemctl status llmgw-agent` and `journalctl -u llmgw-agent` say why", nodeID)
|
||||
case ok && n.AgentVersion != "":
|
||||
return fmt.Errorf("node %s did not answer the upgrade and still runs %s: its download is slow or stuck, "+
|
||||
"or its agent is older than v0.10.4, which cannot upgrade itself", nodeID, n.AgentVersion)
|
||||
default:
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return fmt.Errorf("node %s did not answer the upgrade; an agent older than v0.10.4 cannot upgrade itself", nodeID)
|
||||
}
|
||||
}
|
||||
|
||||
// UpgradeWait is how long the control plane waits for an agent to upgrade:
|
||||
// longer than the agent gives its download (agent.upgradeDownload).
|
||||
const UpgradeWait = 11 * time.Minute
|
||||
|
||||
// wearFrom is a card's wear, when the agent read it.
|
||||
func wearFrom(g *llmgwv1.GPUSensor) *inventory.GPUWear {
|
||||
if !g.GetHasWear() {
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// ErrExists is a model added only if absent that was there already.
|
||||
@@ -1201,10 +1202,14 @@ func (m Model) validate() error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("model %q profile %q: strata runs a replica on one GPU of one node", m.Name, p.Name)
|
||||
}
|
||||
if !slices.ContainsFunc(p.Args, func(a string) bool { return a == "--family" || strings.HasPrefix(a, "--family=") }) {
|
||||
family, size := strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size")
|
||||
if family == "" || size == "" {
|
||||
return fmt.Errorf("model %q profile %q: strata needs the model's family and size in args "+
|
||||
"(--family qwen --size IQ2_XS)", m.Name, p.Name)
|
||||
}
|
||||
if _, err := strata.SizeOf(family, size); err != nil {
|
||||
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
|
||||
return fmt.Errorf("model %q profile %q: strata prepares a model from all of its GGUF files; "+
|
||||
"give the directory they are in as the source", m.Name, p.Name)
|
||||
|
||||
@@ -20,11 +20,17 @@ func TestAStrataProfileNamesItsFamilyAndRunsOnOneCard(t *testing.T) {
|
||||
"family and size": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--size", "IQ2_XS"}}}},
|
||||
"all of its GGUF files": {Name: "x", Source: "http://mirror:8081/x/1/x-00001-of-00002.gguf", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}},
|
||||
"no family \"llama\"": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "llama", "--size", "IQ2_XS"}}}},
|
||||
"comes in IQ1_M": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "coder", "--size", "IQ2_XS"}}}},
|
||||
"family and size ": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen"}}}},
|
||||
"--gguf-dir": {Name: "x", Source: "http://mirror:8081/x/1/", Profiles: []Profile{{
|
||||
Name: "p", Engine: "strata", GPUClass: "6000-24g", Args: []string{"--family", "qwen", "--gguf-dir", "/etc"}}}},
|
||||
} {
|
||||
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), want) {
|
||||
if _, err := New([]Model{bad}); err == nil || !strings.Contains(err.Error(), strings.TrimSpace(want)) {
|
||||
t.Errorf("expected %q, got %v", want, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -134,6 +134,9 @@ type Request struct {
|
||||
// Fallbacks adds the next best ways, lower in priority, so a model whose
|
||||
// first choice is busy runs on another rather than waiting.
|
||||
Fallbacks bool `json:"fallbacks,omitempty"`
|
||||
// Engine is the engine the file runs on: "strata" for a family Strata
|
||||
// runs, "llamacpp" to keep it off Strata; empty, the plan's best.
|
||||
Engine string `json:"engine,omitempty"`
|
||||
// Name is the catalog name; the repository suggests one otherwise.
|
||||
Name string `json:"name,omitempty"`
|
||||
Aliases []string `json:"aliases,omitempty"`
|
||||
@@ -488,6 +491,23 @@ func (s *Server) run(j *Job) {
|
||||
}
|
||||
|
||||
// 3. Whether the nodes' engine loads it, and how much memory it needs.
|
||||
if layout[0].Engine == "strata" {
|
||||
// Strata prepares it on the node from the directory of its files,
|
||||
// and keeps the experts in the RAM the plan counted.
|
||||
if !strings.HasSuffix(source, "/") {
|
||||
source = strings.TrimSuffix(source, mirror.LastPathElement(source))
|
||||
}
|
||||
if !req.sized {
|
||||
req.ContextLength = strataContext
|
||||
}
|
||||
s.step(j, "check", Done, fmt.Sprintf("Strata's %s %s, prepared on the node; %.0f GB of RAM for its experts",
|
||||
layout[0].StrataFamily, layout[0].StrataSize, layout[0].NeedGB))
|
||||
s.mu.Lock()
|
||||
j.Request = req
|
||||
s.mu.Unlock()
|
||||
s.addAndStart(ctx, j, repo, set, name, source, profilesFor(layout, req, 0, 0), hardwareExists(layout))
|
||||
return
|
||||
}
|
||||
if req.Format == "safetensors" {
|
||||
// vLLM reads the model's shape from its config.json, which came
|
||||
// with the weights.
|
||||
@@ -566,10 +586,21 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
|
||||
}
|
||||
var opts []modelplan.Option
|
||||
for _, o := range plan.Options {
|
||||
if o.Artifact == artifact {
|
||||
onStrata := o.Engine == "strata"
|
||||
if o.Artifact == artifact && (req.Engine == "" || (req.Engine == "strata") == onStrata) {
|
||||
opts = append(opts, o)
|
||||
}
|
||||
}
|
||||
if req.Engine == "strata" {
|
||||
// Strata's way only, on the class asked for if one was.
|
||||
i := slices.IndexFunc(opts, func(o modelplan.Option) bool { return req.GPUClass == "" || o.Class == req.GPUClass })
|
||||
if i < 0 {
|
||||
return nil, fmt.Errorf("Strata cannot run %s here: it runs a few families only (`llmgw ctl model plan %s` "+
|
||||
"says whether this is one), on a node with Strata installed, a card of 12 GB or more and the RAM for its experts",
|
||||
set.Name, plan.Repo)
|
||||
}
|
||||
return []modelplan.Option{opts[i]}, nil
|
||||
}
|
||||
var out []modelplan.Option
|
||||
switch {
|
||||
case req.CPU || req.GPUClass != "":
|
||||
@@ -600,7 +631,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model
|
||||
default:
|
||||
out = append(out, modelplan.Option{Artifact: artifact, Format: req.Format, Engine: engine, CPU: true})
|
||||
}
|
||||
if req.Fallbacks {
|
||||
if req.Fallbacks && out[0].Engine != "strata" {
|
||||
shape := func(o modelplan.Option) string {
|
||||
return fmt.Sprintf("%v/%s/%d/%d", o.CPU, o.Class, o.GPUsPerNode, o.Nodes)
|
||||
}
|
||||
@@ -625,6 +656,12 @@ func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []cat
|
||||
for i, o := range layout {
|
||||
p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env}
|
||||
switch {
|
||||
case o.Engine == "strata":
|
||||
p.Name = "strata-" + o.Class
|
||||
p.GPUClass, p.GPUsPerNode = o.Class, 1
|
||||
p.Args = []string{"--family", o.StrataFamily, "--size", o.StrataSize,
|
||||
"--context", strconv.Itoa(req.ContextLength), "--vision", "no"}
|
||||
p.MinMemoryGB = int(o.NeedGB)
|
||||
case o.CPU:
|
||||
p.Name = "cpu"
|
||||
p.Args = llamaArgs(req)
|
||||
@@ -674,6 +711,10 @@ func llamaArgs(req Request) []string {
|
||||
return args
|
||||
}
|
||||
|
||||
// strataContext is a Strata model's context when its caller did not say:
|
||||
// Strata's own default.
|
||||
const strataContext = 32768
|
||||
|
||||
// An embedding model's slots, when its caller did not size them: passages
|
||||
// of up to a thousand tokens, thirty-two at a time.
|
||||
const (
|
||||
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// Planning: which of a repository's files suits the cluster's hardware, and
|
||||
@@ -168,6 +169,18 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan,
|
||||
if !p.Servable {
|
||||
p.Options = nil
|
||||
}
|
||||
// A family Strata runs: its files on one card with the experts in RAM.
|
||||
// Strata reads them itself, whether or not llama.cpp could.
|
||||
if fam, ok := strata.ForRepo(repo.ID); ok && p.Format == "gguf" {
|
||||
if so := modelplan.StrataOptions(fam.Name, arts, s.deps.Pool()); len(so) > 0 {
|
||||
p.Options = append(p.Options, so...)
|
||||
modelplan.Rank(p.Options)
|
||||
if !p.Servable {
|
||||
p.Note = strings.TrimSpace(p.Note + " llama.cpp cannot run it (" + p.Reason + "); Strata can.")
|
||||
p.Servable, p.Reason = true, ""
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, a := range arts {
|
||||
pa := PlannedArtifact{Artifact: a}
|
||||
for i := range p.Options {
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
package modelserve
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
)
|
||||
|
||||
func TestAStrataWayIsChosenAndMadeAProfile(t *testing.T) {
|
||||
set := hf.FileSet{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf"}
|
||||
strataWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "strata", Class: "6000-24g",
|
||||
GPUsPerNode: 1, Nodes: 1, NeedGB: 52, HaveGB: 213, Hosts: 1, StrataFamily: "qwen", StrataSize: "IQ2_XS"}
|
||||
cpuWay := modelplan.Option{Artifact: set.Name, Format: "gguf", Engine: "llamacpp", CPU: true, NeedGB: 75, HaveGB: 213, Hosts: 1}
|
||||
plan := Plan{Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Options: []modelplan.Option{strataWay, cpuWay}}
|
||||
|
||||
got, err := chooseLayout(Request{Format: "gguf", Fallbacks: true}, plan, true, set)
|
||||
if err != nil || len(got) != 1 || got[0].Engine != "strata" {
|
||||
t.Fatalf("the plan's best, without llama.cpp fallbacks: %+v %v", got, err)
|
||||
}
|
||||
if got, _ := chooseLayout(Request{Format: "gguf", Engine: "llamacpp"}, plan, true, set); len(got) != 1 || !got[0].CPU {
|
||||
t.Errorf("kept off Strata: %+v", got)
|
||||
}
|
||||
if _, err := chooseLayout(Request{Format: "gguf", Engine: "strata", GPUClass: "h100-80g"}, plan, true, set); err == nil {
|
||||
t.Error("Strata on a class it was not planned for")
|
||||
}
|
||||
|
||||
ps := profilesFor(got, Request{ContextLength: 32768}, 0, 0)
|
||||
p := ps[0]
|
||||
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
|
||||
!slices.Equal(p.Args, []string{"--family", "qwen", "--size", "IQ2_XS", "--context", "32768", "--vision", "no"}) ||
|
||||
!strings.HasPrefix(p.Name, "strata-") {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// Store is the persistence this needs.
|
||||
@@ -129,6 +130,20 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b
|
||||
// fits everywhere as far as placement can tell — however many of it
|
||||
// there already are. Required of what is added now; models stored before
|
||||
// this still load, and say what they are missing.
|
||||
m.Profiles = slices.Clone(m.Profiles)
|
||||
for i := range m.Profiles {
|
||||
// Strata keeps a model's experts in the host's memory, tens of
|
||||
// gigabytes of it: placed as if it took none, two replicas, or one
|
||||
// beside a CPU model, filled a node until one was killed. What its
|
||||
// size takes, from Strata's table, unless the model says.
|
||||
if p := &m.Profiles[i]; p.Engine == "strata" && p.MinMemoryGB <= 0 {
|
||||
size, err := strata.SizeOf(strata.ArgValue(p.Args, "--family"), strata.ArgValue(p.Args, "--size"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("model %q profile %q: %w", m.Name, p.Name, err)
|
||||
}
|
||||
p.MinMemoryGB = size.MemoryGB()
|
||||
}
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.CPUOnly() && p.MinMemoryGB <= 0 {
|
||||
return fmt.Errorf("model %q profile %q runs without a GPU and declares no memory (min_memory_gb): "+
|
||||
|
||||
@@ -438,3 +438,23 @@ func TestAStoppedModelRunsNoneAndStartsAsItWas(t *testing.T) {
|
||||
t.Error("still stopped after a scale")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStrataModelReservesTheMemoryItsExpertsTake(t *testing.T) {
|
||||
s, _, rec := newService(t)
|
||||
m := model("big")
|
||||
m.Source = "http://mirror:8081/big/v1/"
|
||||
m.Profiles = []catalog.Profile{{Name: "gpu", Engine: "strata", GPUClass: "6000-24g", GPUsPerNode: 1,
|
||||
Args: []string{"--family", "qwen", "--size", "IQ2_XS"}}}
|
||||
if err := s.Add(t.Context(), m, "ilker"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _ := rec.last.Resolve("big")
|
||||
if got.Profiles[0].MinMemoryGB != 52 {
|
||||
t.Errorf("IQ2_XS reserves %d GB, want Strata's 48 and 4 for the engine", got.Profiles[0].MinMemoryGB)
|
||||
}
|
||||
m.Name = "big2"
|
||||
m.Profiles[0].Args = []string{"--family", "qwen", "--size", "IQ9"}
|
||||
if err := s.Add(t.Context(), m, "ilker"); err == nil || !strings.Contains(err.Error(), "comes in") {
|
||||
t.Errorf("an unknown size: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -53,6 +53,8 @@ type PoolNode struct {
|
||||
// card beside it another. Empty means Class, GPUMemGB, GPUs and FreeGPUs
|
||||
// say it all.
|
||||
Kinds []PoolKind
|
||||
// Engines are the engines the node has installed.
|
||||
Engines []string
|
||||
}
|
||||
|
||||
// PoolKind is one class of a node's devices.
|
||||
@@ -110,6 +112,11 @@ type Option struct {
|
||||
// cards, whole, rather than slicing each across them: vLLM's
|
||||
// --enable-expert-parallel, for a model of experts on more than one card.
|
||||
ExpertParallel bool `json:"expert_parallel,omitempty"`
|
||||
// StrataFamily and StrataSize are Strata's --family and --size, for an
|
||||
// option on Strata: one card, and the experts in the node's RAM, which
|
||||
// NeedGB and HaveGB are then about.
|
||||
StrataFamily string `json:"strata_family,omitempty"`
|
||||
StrataSize string `json:"strata_size,omitempty"`
|
||||
}
|
||||
|
||||
// GPUs is every card one replica takes.
|
||||
@@ -130,7 +137,7 @@ func (o Option) Fits() bool { return o.NeedGB <= o.HaveGB }
|
||||
// WithContext is the option for another context: kvPerToken a token's KV
|
||||
// cache, the whole model; the weights and the engine's share stay.
|
||||
func (o Option) WithContext(kvPerToken int64, tokens int) Option {
|
||||
if kvPerToken <= 0 || tokens <= 0 {
|
||||
if kvPerToken <= 0 || tokens <= 0 || o.Engine == "strata" {
|
||||
return o
|
||||
}
|
||||
ctx := GB(int64(tokens) * kvPerToken)
|
||||
@@ -144,6 +151,9 @@ func (o Option) WithContext(kvPerToken int64, tokens int) Option {
|
||||
|
||||
// MemoryText says what it needs, part by part.
|
||||
func (o Option) MemoryText() string {
|
||||
if o.Engine == "strata" {
|
||||
return fmt.Sprintf("%.0f GB of the node's RAM for its experts, of %.0f usable, and one card", o.NeedGB, o.HaveGB)
|
||||
}
|
||||
parts := fmt.Sprintf("weights %.1f + context %.1f + engine %.1f", o.WeightsGB, o.ContextGB, o.EngineGB)
|
||||
if o.WeightsGB == 0 && o.ContextGB == 0 {
|
||||
parts = ""
|
||||
@@ -174,6 +184,9 @@ func (o Option) Label() string {
|
||||
if o.Nodes > 1 {
|
||||
return fmt.Sprintf("%d nodes × %d %s on %s", o.Nodes, o.GPUsPerNode, o.Class, o.Fabric)
|
||||
}
|
||||
if o.Engine == "strata" {
|
||||
return fmt.Sprintf("1 × %s with Strata, experts in RAM", o.Class)
|
||||
}
|
||||
return fmt.Sprintf("%d × %s", o.GPUsPerNode, o.Class)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
package modelplan
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// Strata runs a few mixture-of-experts models with their experts in the
|
||||
// node's RAM and one card for the rest: a model that fits no card, or only
|
||||
// many, served from one. Offered for the files of a family it runs, on nodes
|
||||
// that have it installed, a card of at least strataMinCardGB and the RAM.
|
||||
|
||||
// strataMinCardGB is the smallest card Strata is worth running on: what its
|
||||
// guide recommends.
|
||||
const strataMinCardGB = 12
|
||||
|
||||
// StrataOptions are the ways a family's files could run on Strata here.
|
||||
func StrataOptions(family string, arts []Artifact, nodes []PoolNode) []Option {
|
||||
var out []Option
|
||||
for _, a := range arts {
|
||||
size, ok := strata.SizeForFile(family, a.Name)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
need := float64(size.MemoryGB())
|
||||
byClass := map[string]*Option{}
|
||||
var order []string
|
||||
for _, n := range nodes {
|
||||
if !slices.Contains(n.Engines, "strata") || n.HostMemGB <= 0 {
|
||||
continue
|
||||
}
|
||||
have := float64(n.HostMemGB) * hostShare
|
||||
if need > have {
|
||||
continue
|
||||
}
|
||||
for _, k := range n.kinds() {
|
||||
if k.Class == "" || isMIG(k.Class) || k.GPUMemGB < strataMinCardGB || k.GPUs < 1 {
|
||||
continue
|
||||
}
|
||||
o := byClass[k.Class]
|
||||
if o == nil {
|
||||
o = &Option{Artifact: a.Name, Format: a.Format, Engine: "strata", Class: k.Class,
|
||||
GPUsPerNode: 1, Nodes: 1, NeedGB: need, StrataFamily: family, StrataSize: size.Name}
|
||||
byClass[k.Class] = o
|
||||
order = append(order, k.Class)
|
||||
}
|
||||
o.Hosts++
|
||||
o.HaveGB = max(o.HaveGB, have)
|
||||
if k.FreeGPUs > 0 && float64(n.HostFreeGB) >= need {
|
||||
o.FreeNow = true
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, c := range order {
|
||||
out = append(out, *byClass[c])
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
package modelplan
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestStrataIsOfferedOnANodeWithItTheCardAndTheRAM(t *testing.T) {
|
||||
arts := []Artifact{{Name: "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf", Format: "gguf", Bytes: 68e9},
|
||||
{Name: "something-else-Q4_K_M.gguf", Format: "gguf", Bytes: 5e9}}
|
||||
nodes := []PoolNode{
|
||||
{ID: "cuda01", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, HostFreeGB: 200,
|
||||
Engines: []string{"llamacpp", "strata"}},
|
||||
{ID: "no-strata", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"llamacpp"}},
|
||||
{ID: "small-ram", Class: "6000-24g", GPUMemGB: 24, GPUs: 1, FreeGPUs: 1, HostMemGB: 32, Engines: []string{"strata"}},
|
||||
{ID: "small-card", Class: "t4-8g", GPUMemGB: 8, GPUs: 1, FreeGPUs: 1, HostMemGB: 251, Engines: []string{"strata"}},
|
||||
}
|
||||
opts := StrataOptions("qwen", arts, nodes)
|
||||
if len(opts) != 1 {
|
||||
t.Fatalf("%+v", opts)
|
||||
}
|
||||
o := opts[0]
|
||||
if o.Engine != "strata" || o.StrataSize != "IQ2_XS" || o.Hosts != 1 || !o.FreeNow || o.NeedGB != 52 || o.GPUs() != 1 {
|
||||
t.Errorf("%+v", o)
|
||||
}
|
||||
// It ranks before llama.cpp on the CPU, and says what it needs.
|
||||
all := append(Options(arts[:1], Shape{}, nodes, 16384), opts...)
|
||||
Rank(all)
|
||||
if all[0].Engine != "strata" {
|
||||
t.Errorf("first is %+v", all[0])
|
||||
}
|
||||
if o.Label() != "1 × 6000-24g with Strata, experts in RAM" || o.MemoryText() == "" {
|
||||
t.Errorf("%q %q", o.Label(), o.MemoryText())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package strata
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// What Strata runs, as its setup (0.2.7, the version llmgw installs) lists
|
||||
// it: a few model families, each in a few sizes, with what a size takes.
|
||||
// Written down here so a model is checked when it is added rather than when
|
||||
// a node's setup refuses it minutes later, and so the plan can offer Strata
|
||||
// for a model it runs, with the memory to reserve.
|
||||
|
||||
// Family is a model Strata runs, from one publisher's files.
|
||||
type Family struct {
|
||||
// Name is setup's --family.
|
||||
Name string
|
||||
Title string
|
||||
// Repo is the Hugging Face repository its files are in; empty when
|
||||
// Strata fetches them itself from several.
|
||||
Repo string
|
||||
// Subdir is whether each size's files are in a directory of its own,
|
||||
// named as the size, in the repository.
|
||||
Subdir bool
|
||||
// Engine is a family that runs on an engine path of its own, with a
|
||||
// download of its own (DeepSeek V4.1).
|
||||
Engine bool
|
||||
}
|
||||
|
||||
// Size is one of a family's sizes: setup's --model, which llmgw's profile
|
||||
// says as --size.
|
||||
type Size struct {
|
||||
Name string
|
||||
Families []string
|
||||
// DownloadGB is what is fetched; DiskGB what it takes on the node once
|
||||
// prepared, when more than the download.
|
||||
DownloadGB, DiskGB float64
|
||||
// RAMGB is the host memory that keeps every expert in RAM: what a
|
||||
// replica reserves. Strata can read the rest from the disk with less,
|
||||
// much slower.
|
||||
RAMGB float64
|
||||
// NVIDIAOnly is a size whose kernels only NVIDIA cards have.
|
||||
NVIDIAOnly bool
|
||||
}
|
||||
|
||||
// Families are the families Strata 0.2.7 runs.
|
||||
var Families = []Family{
|
||||
{Name: "qwen", Title: "Qwen3.8-Flash-Next", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF", Subdir: true},
|
||||
{Name: "swift", Title: "Swift 1.5 (Qwen3.8-Flash-Next, shorter thinking)", Repo: "ukisai/Swift-1.5-Qwen3.8-Flash-Next-GSQ-RCO-GGUF"},
|
||||
{Name: "coder", Title: "Qwen3.8-Flash-Next Coder", Repo: "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF", Subdir: true},
|
||||
{Name: "unsloth", Title: "Qwen3.8-Flash-Next (Unsloth)", Repo: "unsloth/Qwen3.8-Flash-Next-GGUF", Subdir: true},
|
||||
{Name: "gyro", Title: "Qwen3.8-Flash-Next (Gyro)", Repo: "agentionai/Qwen3.8-Flash-Next-Gyro-GGUF"},
|
||||
{Name: "nvfp4", Title: "Qwen3.8-Flash-Next (NVFP4)"},
|
||||
{Name: "qwen36", Title: "Qwen3.6-35B-A3B", Repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"},
|
||||
{Name: "deepseek41", Title: "DeepSeek V4.1-Flash", Repo: "ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF", Engine: true},
|
||||
}
|
||||
|
||||
// Sizes are the sizes Strata 0.2.7 runs, and which families have each.
|
||||
var Sizes = []Size{
|
||||
{Name: "Q2_0", Families: []string{"qwen"}, DownloadGB: 66.4, RAMGB: 48},
|
||||
{Name: "IQ2_XS", Families: []string{"qwen", "swift"}, DownloadGB: 68.0, RAMGB: 48},
|
||||
{Name: "IQ3_XXS", Families: []string{"qwen", "swift"}, DownloadGB: 75.8, RAMGB: 60},
|
||||
{Name: "IQ3_S", Families: []string{"qwen", "swift"}, DownloadGB: 83.6, RAMGB: 62},
|
||||
{Name: "IQ1_M", Families: []string{"coder"}, DownloadGB: 58.4, RAMGB: 32},
|
||||
{Name: "UD-IQ4_XS", Families: []string{"unsloth"}, DownloadGB: 93.7, RAMGB: 80},
|
||||
{Name: "UD-Q4_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 111.3, RAMGB: 94},
|
||||
{Name: "UD-Q5_K_XL", Families: []string{"unsloth"}, DownloadGB: 158.3, RAMGB: 108},
|
||||
{Name: "UD-Q6_K_XL", Families: []string{"unsloth", "qwen36"}, DownloadGB: 169.2, RAMGB: 118},
|
||||
{Name: "Q8_0", Families: []string{"unsloth", "qwen36"}, DownloadGB: 188.2, RAMGB: 136},
|
||||
{Name: "Gyro-S", Families: []string{"gyro"}, DownloadGB: 58.5, RAMGB: 27.6},
|
||||
{Name: "Gyro-M", Families: []string{"gyro"}, DownloadGB: 92.0, RAMGB: 35},
|
||||
{Name: "NVFP4", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "NVFP4-NVIDIA", Families: []string{"nvfp4"}, DownloadGB: 128, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "NVFP4-QAD", Families: []string{"nvfp4"}, DownloadGB: 102.2, DiskGB: 212.2, RAMGB: 80, NVIDIAOnly: true},
|
||||
{Name: "IST-DAS-1.75bpw", Families: []string{"deepseek41"}, DownloadGB: 239, RAMGB: 34},
|
||||
{Name: "IST-DAS-2.00bpw", Families: []string{"deepseek41"}, DownloadGB: 257, RAMGB: 40},
|
||||
{Name: "IST-DAS-2.25bpw", Families: []string{"deepseek41"}, DownloadGB: 274, RAMGB: 44},
|
||||
{Name: "UD-Q8_K_XL", Families: []string{"deepseek41"}, DownloadGB: 520, RAMGB: 306},
|
||||
}
|
||||
|
||||
// qwen36 is smaller than the Flash-Next sizes of the same names: its own
|
||||
// figures.
|
||||
var qwen36RAM = map[string]float64{"UD-Q4_K_XL": 32, "UD-Q6_K_XL": 44, "Q8_0": 52}
|
||||
var qwen36Download = map[string]float64{"UD-Q4_K_XL": 22.9, "UD-Q6_K_XL": 32.6, "Q8_0": 37.8}
|
||||
|
||||
// FamilyNamed is a family by its --family.
|
||||
func FamilyNamed(name string) (Family, bool) {
|
||||
i := slices.IndexFunc(Families, func(f Family) bool { return f.Name == name })
|
||||
if i < 0 {
|
||||
return Family{}, false
|
||||
}
|
||||
return Families[i], true
|
||||
}
|
||||
|
||||
// SizeOf is a family's size by its --size, with that family's figures.
|
||||
func SizeOf(family, size string) (Size, error) {
|
||||
if _, ok := FamilyNamed(family); !ok {
|
||||
return Size{}, fmt.Errorf("strata runs no family %q; it runs %s", family, strings.Join(familyNames(), ", "))
|
||||
}
|
||||
var names []string
|
||||
for _, s := range Sizes {
|
||||
if !slices.Contains(s.Families, family) {
|
||||
continue
|
||||
}
|
||||
names = append(names, s.Name)
|
||||
if s.Name == size {
|
||||
if family == "qwen36" {
|
||||
s.RAMGB, s.DownloadGB = qwen36RAM[size], qwen36Download[size]
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
return Size{}, fmt.Errorf("strata's %s comes in %s, not %q", family, strings.Join(names, ", "), size)
|
||||
}
|
||||
|
||||
// ForRepo is the family whose files a Hugging Face repository holds.
|
||||
func ForRepo(repo string) (Family, bool) {
|
||||
i := slices.IndexFunc(Families, func(f Family) bool { return f.Repo != "" && strings.EqualFold(f.Repo, repo) })
|
||||
if i < 0 {
|
||||
return Family{}, false
|
||||
}
|
||||
return Families[i], true
|
||||
}
|
||||
|
||||
// SizesOf are a family's sizes, with its figures.
|
||||
func SizesOf(family string) []Size {
|
||||
var out []Size
|
||||
for _, s := range Sizes {
|
||||
if slices.Contains(s.Families, family) {
|
||||
got, _ := SizeOf(family, s.Name)
|
||||
out = append(out, got)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MemoryGB is what a replica of a size reserves of the host: its experts,
|
||||
// and the engine and server beside them.
|
||||
func (s Size) MemoryGB() int { return int(s.RAMGB+0.999) + 4 }
|
||||
|
||||
func familyNames() []string {
|
||||
out := make([]string, len(Families))
|
||||
for i, f := range Families {
|
||||
out[i] = f.Name
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// ArgValue is a flag's value in a profile's arguments, "" when absent.
|
||||
func ArgValue(args []string, flag string) string {
|
||||
for i, a := range args {
|
||||
if a == flag && i+1 < len(args) {
|
||||
return args[i+1]
|
||||
}
|
||||
if v, ok := strings.CutPrefix(a, flag+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// SizeForFile is the size of a family a file is, by its name: the longest
|
||||
// size name in it, so UD-Q4_K_XL is not read as Q4_K_XL's shorter cousins.
|
||||
func SizeForFile(family, file string) (Size, bool) {
|
||||
upper := strings.ToUpper(file)
|
||||
var best Size
|
||||
for _, s := range SizesOf(family) {
|
||||
if strings.Contains(upper, strings.ToUpper(s.Name)) && len(s.Name) > len(best.Name) {
|
||||
best = s
|
||||
}
|
||||
}
|
||||
return best, best.Name != ""
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package strata
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestASizeIsKnownForTheFamiliesThatHaveIt(t *testing.T) {
|
||||
if s, err := SizeOf("qwen", "IQ2_XS"); err != nil || s.RAMGB != 48 || s.MemoryGB() != 52 {
|
||||
t.Errorf("%+v %v", s, err)
|
||||
}
|
||||
if s, err := SizeOf("qwen36", "Q8_0"); err != nil || s.RAMGB != 52 {
|
||||
t.Errorf("qwen36's own figures: %+v %v", s, err)
|
||||
}
|
||||
if _, err := SizeOf("coder", "IQ2_XS"); err == nil || !strings.Contains(err.Error(), "IQ1_M") {
|
||||
t.Errorf("coder in IQ2_XS: %v", err)
|
||||
}
|
||||
if _, err := SizeOf("llama", "Q8_0"); err == nil || !strings.Contains(err.Error(), "deepseek41") {
|
||||
t.Errorf("an unknown family: %v", err)
|
||||
}
|
||||
if f, ok := ForRepo("istA-daslab/qwen3.8-flash-next-gsq-rco-gguf"); !ok || f.Name != "qwen" {
|
||||
t.Errorf("by repository: %+v", f)
|
||||
}
|
||||
for _, f := range Families {
|
||||
if len(SizesOf(f.Name)) == 0 {
|
||||
t.Errorf("%s has no sizes", f.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -823,7 +823,11 @@ func (p *AccountPages) openRepo(r *http.Request, d *adminMirrorData, repo string
|
||||
Version: got.ShortRevision(),
|
||||
}
|
||||
rs.Verdict = p.verdictFor(langOf(r), got.Architecture, set.Name)
|
||||
if fit != nil && rs.Verdict.Servable {
|
||||
// A family Strata runs: one card and the experts in RAM, whether
|
||||
// or not llama.cpp could load it.
|
||||
if layout, ok := fit.strataLayout(langOf(r), got.ID, set.Name, set.Bytes); ok {
|
||||
rs.Verdict, rs.GPULayout = gguf.Verdict{Checked: true, Servable: true}, layout
|
||||
} else if fit != nil && rs.Verdict.Servable {
|
||||
if v, ok := fit.holds(langOf(r), set.Name, set.Bytes, got.Params); !ok {
|
||||
rs.Verdict = v
|
||||
} else if gguf.IsGGUF(set.Name) {
|
||||
|
||||
@@ -4,14 +4,18 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// CatalogAdmin adds a model to the catalogue in force.
|
||||
@@ -76,6 +80,11 @@ type serveForm struct {
|
||||
// CtxPool is the engine context pool the form starts with, and the one
|
||||
// the memory figure assumes.
|
||||
CtxPool int
|
||||
// Context is the context length the form starts with.
|
||||
Context int
|
||||
// StrataFamily and StrataSize are Strata's --family and --size, for a
|
||||
// version of a family Strata runs, served on it.
|
||||
StrataFamily, StrataSize string
|
||||
|
||||
// Refusal is set when this version cannot be served as it stands, and says
|
||||
// why. Offered and then failing at placement is worse than not offered.
|
||||
@@ -122,6 +131,35 @@ func planServe(l *Lang, st mirror.Status, files []mirror.Status, base string,
|
||||
st.BytesTotal = total
|
||||
}
|
||||
|
||||
f.Context = 4096
|
||||
// Files of a family Strata runs, on a fleet that has Strata: served on
|
||||
// it, from the version's directory, whether or not llama.cpp knows them.
|
||||
if f.Engine == "llamacpp" && strataInstalled(fleet) {
|
||||
if id, _, _, ok := hf.FileSource(st.Source, hf.DefaultEndpoint); ok {
|
||||
if fam, ok := strata.ForRepo(id); ok {
|
||||
if size, ok := strata.SizeForFile(fam.Name, st.File); ok {
|
||||
f.Engine, f.StrataFamily, f.StrataSize = "strata", fam.Name, size.Name
|
||||
f.Source = strings.TrimSuffix(mirror.Entry{Model: st.Model, Version: st.Version}.URL(base), "/") + "/"
|
||||
f.Context, f.Profile = 32768, "gpu"
|
||||
f.Placements, f.FreeGPUs = placementsFor(l, fleet, f.Engine)
|
||||
for _, pc := range f.Placements {
|
||||
if pc.Selected {
|
||||
f.Profile = pc.profile()
|
||||
}
|
||||
}
|
||||
f.GPUs = 1
|
||||
f.MinMemoryGB = size.MemoryGB()
|
||||
f.MemoryBasis = l.T("serve.basis.strata", fam.Title, size.Name)
|
||||
if cat != nil && cat.Declared(st.Model) {
|
||||
f.Already = true
|
||||
f.Refusal = l.T("serve.refuse.declared", st.Model)
|
||||
}
|
||||
return f
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
switch f.Engine {
|
||||
case "":
|
||||
f.Refusal = l.T("serve.refuse.no_engine", extOf(st.File))
|
||||
@@ -220,6 +258,19 @@ func extOf(file string) string {
|
||||
return file
|
||||
}
|
||||
|
||||
// strataInstalled reports whether a node of the fleet has Strata.
|
||||
func strataInstalled(fleet Source) bool {
|
||||
if fleet == nil {
|
||||
return false
|
||||
}
|
||||
for _, n := range fleet.Nodes() {
|
||||
if slices.Contains(n.Engines, "strata") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// runsAnywhere reports whether some node has this engine.
|
||||
//
|
||||
// True when nothing is known, so a fleet of agents older than the engine probe
|
||||
@@ -304,6 +355,14 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
|
||||
}
|
||||
place := placeChoice{Value: strings.TrimSpace(get("placement"))}
|
||||
switch class, onGPU := strings.CutPrefix(place.Value, "gpu:"); {
|
||||
case onGPU && class != "" && engine == "strata":
|
||||
// One card; the model's family and size, and the context.
|
||||
p.GPUClass, p.GPUsPerNode, place.GPUs = class, 1, 1
|
||||
family, size := strings.TrimSpace(get("strata_family")), strings.TrimSpace(get("strata_size"))
|
||||
if _, err := strata.SizeOf(family, size); err != nil {
|
||||
return m, err
|
||||
}
|
||||
p.Args = []string{"--family", family, "--size", size, "--context", strconv.Itoa(ctxLen), "--vision", "no"}
|
||||
case onGPU && class != "":
|
||||
// Placement is decided by gpu_class, which is what the selector
|
||||
// sets; the count is gpus_per_node for llama.cpp, which spreads
|
||||
@@ -330,7 +389,7 @@ func buildModel(l *Lang, get func(string) string) (catalog.Model, error) {
|
||||
// The pool, not the per-request limit: llama.cpp divides it across its
|
||||
// parallel slots, so the number a client may use is smaller and is what
|
||||
// context_length above says.
|
||||
if v := strings.TrimSpace(get("ctx_size")); v != "" {
|
||||
if v := strings.TrimSpace(get("ctx_size")); v != "" && engine != "strata" {
|
||||
if _, err := positive(l, v, "serve.what.ctx_pool"); err != nil {
|
||||
return m, err
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
)
|
||||
|
||||
@@ -554,3 +555,38 @@ func (f *fakeCatalog) Stop(_ context.Context, name, _ string) (catalog.Model, er
|
||||
func (f *fakeCatalog) Start(_ context.Context, name, _ string) (catalog.Model, error) {
|
||||
return catalog.Model{Name: name, Scaling: catalog.Scaling{Min: 1, Max: 1}}, f.err
|
||||
}
|
||||
|
||||
// A version of a family Strata runs, on a fleet with Strata, is served on it:
|
||||
// one card, its family and size, the experts' memory, from the directory.
|
||||
func TestAStrataFamilysVersionIsServedOnStrata(t *testing.T) {
|
||||
st := held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf")
|
||||
st.Source = "https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF/resolve/abc/IQ2_XS/" + st.File
|
||||
files := []mirror.Status{st, held("q38", "abc123", "Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00002-of-00002.gguf")}
|
||||
f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp", "strata"}, memoryGB: 251},
|
||||
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} })
|
||||
if f.Engine != "strata" || f.StrataFamily != "qwen" || f.StrataSize != "IQ2_XS" || f.Refusal != "" ||
|
||||
f.MinMemoryGB != 52 || f.Context != 32768 || !strings.HasSuffix(f.Source, "/q38/abc123/") {
|
||||
t.Fatalf("%+v", f)
|
||||
}
|
||||
// Without Strata on any node, llama.cpp as before, and its refusal.
|
||||
if f := planServe(english, st, files, "http://mirror:8081", fakeFleet{gpus: 1, engines: []string{"llamacpp"}},
|
||||
nil, func(string) gguf.Verdict { return gguf.Verdict{Checked: true, Reason: "unknown architecture"} }); f.Engine != "llamacpp" || f.Refusal == "" {
|
||||
t.Errorf("%+v", f)
|
||||
}
|
||||
form := url.Values{"name": {"q38"}, "source": {f.Source}, "engine": {"strata"}, "replicas": {"1"},
|
||||
"context_length": {"32768"}, "placement": {"gpu:6000-24g"}, "min_memory_gb": {"52"},
|
||||
"strata_family": {"qwen"}, "strata_size": {"IQ2_XS"}}
|
||||
m, err := buildModel(english, form.Get)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := m.Profiles[0]
|
||||
if p.Engine != "strata" || p.GPUClass != "6000-24g" || p.GPUsPerNode != 1 || p.MinMemoryGB != 52 ||
|
||||
strings.Join(p.Args, " ") != "--family qwen --size IQ2_XS --context 32768 --vision no" {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
form.Set("strata_size", "IQ9")
|
||||
if _, err := buildModel(english, form.Get); err == nil {
|
||||
t.Error("an unknown size was taken")
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/gguf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/modelplan"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/strata"
|
||||
)
|
||||
|
||||
// The explorer lists what this cluster can run, judged by the nodes it has
|
||||
@@ -137,8 +138,43 @@ var pictureTasks = []string{"text-to-image", "image-to-image", "image-to-3d", "t
|
||||
// people want. A repository of Q8 only is judged kindly; its page says.
|
||||
const ggufBitsPerWeight = 4.8
|
||||
|
||||
// strataLayout is how a file of a family Strata runs would run on it here,
|
||||
// and whether it could.
|
||||
func (c *clusterFit) strataLayout(l *Lang, repo, name string, bytes int64) (string, bool) {
|
||||
if c == nil {
|
||||
return "", false
|
||||
}
|
||||
fam, ok := strata.ForRepo(repo)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
opts := modelplan.StrataOptions(fam.Name, []modelplan.Artifact{{Name: name, Format: "gguf", Bytes: bytes}}, c.pool)
|
||||
if len(opts) == 0 {
|
||||
return "", false
|
||||
}
|
||||
return l.T("explore.strata", opts[0].Class, l.Num(opts[0].NeedGB, 0)), true
|
||||
}
|
||||
|
||||
// strataRuns is the smallest of a Strata family's sizes that runs here, as
|
||||
// a layout, for a repository of the family.
|
||||
func (c *clusterFit) strataRuns(l *Lang, repo string) (string, bool) {
|
||||
fam, ok := strata.ForRepo(repo)
|
||||
if !ok {
|
||||
return "", false
|
||||
}
|
||||
for _, s := range strata.SizesOf(fam.Name) {
|
||||
if layout, ok := c.strataLayout(l, repo, s.Name+".gguf", int64(s.DownloadGB*1e9)); ok {
|
||||
return layout, true
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// runs reports whether some node here could hold the repository's model.
|
||||
func (c *clusterFit) runs(m hf.Model) bool {
|
||||
if _, ok := c.strataRuns(english, m.ID); ok {
|
||||
return true
|
||||
}
|
||||
params := m.Params()
|
||||
if slices.Contains(pictureTasks, m.PipelineTag) || m.HasTag("diffusers") {
|
||||
b := m.WeightBytes()
|
||||
@@ -329,6 +365,9 @@ func layoutLabel(l *Lang, o modelplan.Option, links modelplan.Links) string {
|
||||
// rowGPU is a listed repository's speed on the cards: its GGUF at Q4_K_M
|
||||
// on llama.cpp, its safetensors on vLLM, whichever is faster here.
|
||||
func (c *clusterFit) rowGPU(l *Lang, m hf.Model) (string, string) {
|
||||
if layout, ok := c.strataRuns(l, m.ID); ok {
|
||||
return "", layout
|
||||
}
|
||||
if !c.gpus() || m.Params() <= 0 {
|
||||
return "", ""
|
||||
}
|
||||
|
||||
@@ -256,5 +256,6 @@
|
||||
"mirror.err.not_adapter": "that file is a model, not a LoRA adapter",
|
||||
"mirror.err.wrong_arch": "that adapter is for %s models and %s is %s",
|
||||
"mirror.msg.adapter_added": "%[1]s: adapter %[2]s added; ask for it by that name once the replicas are restarted (llmgw ctl restart --model %[1]s)",
|
||||
"mirror.err.adapter_op": "add or remove an adapter"
|
||||
"mirror.err.adapter_op": "add or remove an adapter",
|
||||
"explore.strata": "1× %s with Strata, %s GB of RAM"
|
||||
}
|
||||
@@ -81,5 +81,7 @@
|
||||
"serve.replicas_n.one": "%d replica",
|
||||
"serve.replicas_n.other": "%d replicas",
|
||||
"serve.msg.judge": "It is the guardrails' judge (role: guard): policies naming %s ask it; clients do not see it.",
|
||||
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s."
|
||||
"serve.msg.judge_renamed": "Its weights are a judge's, so it is the guardrails' judge (role: guard), named %[2]s as the policies ask rather than %[1]s.",
|
||||
"serve.basis.strata": "what Strata keeps of %s %s's experts in RAM, and 4 GB for its engine and server",
|
||||
"serve.strata": "Strata runs this (%s, %s): one card, and the experts in the node's RAM. The first start prepares the model on the node, which takes minutes."
|
||||
}
|
||||
@@ -249,5 +249,6 @@
|
||||
"mirror.err.not_adapter": "bu dosya bir LoRA adaptörü değil, bir model",
|
||||
"mirror.err.wrong_arch": "bu adaptör %s modelleri için; %s ise %s",
|
||||
"mirror.msg.adapter_added": "%[1]s: %[2]s adaptörü eklendi; replikalar yeniden başlatılınca bu adla isteyin (llmgw ctl restart --model %[1]s)",
|
||||
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın"
|
||||
"mirror.err.adapter_op": "bir adaptör ekleyin ya da kaldırın",
|
||||
"explore.strata": "Strata ile 1× %s, %s GB RAM"
|
||||
}
|
||||
@@ -78,5 +78,7 @@
|
||||
"serve.msg.added": "%s kataloğa eklendi, %s: %s; bir sonraki geçişte yerleştirilecek",
|
||||
"serve.replicas_n.other": "%d replika",
|
||||
"serve.msg.judge": "Guardrail'lerin judge modeli olarak eklendi (role: guard): %s adını veren politikalar ona soruyor; istemciler görmüyor.",
|
||||
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s."
|
||||
"serve.msg.judge_renamed": "Ağırlıkları bir judge modelinin olduğu için guardrail judge'ı olarak eklendi (role: guard); adı %[1]s yerine politikaların istediği gibi %[2]s.",
|
||||
"serve.basis.strata": "Strata'nın %s %s için RAM'de tuttuğu uzmanlar, artı motoru ve sunucusu için 4 GB",
|
||||
"serve.strata": "Bunu Strata çalıştırır (%s, %s): tek kart, uzmanlar node'un RAM'inde. İlk açılışta model node üzerinde hazırlanır; bu birkaç dakika sürer."
|
||||
}
|
||||
@@ -133,8 +133,11 @@
|
||||
<input type="hidden" name="csrf" value="{{$.CSRF}}">
|
||||
<input type="hidden" name="source" value="{{.Source}}">
|
||||
<input type="hidden" name="engine" value="{{.Engine}}">
|
||||
{{if .StrataFamily}}<input type="hidden" name="strata_family" value="{{.StrataFamily}}">
|
||||
<input type="hidden" name="strata_size" value="{{.StrataSize}}">{{end}}
|
||||
<p class="sub">{{th "serve.serving" .Source .Engine}}
|
||||
{{if .Files}}{{t "serve.whole_version" .Files}}{{end}}</p>
|
||||
{{if .StrataFamily}}<p class="note">{{t "serve.strata" .StrataFamily .StrataSize}}</p>{{end}}
|
||||
<div class="field-row">
|
||||
<label>{{t "serve.name"}} <input type="text" name="name" value="{{.Model}}" required></label>
|
||||
<label>{{t "serve.aliases"}} <span class="opt">{{t "serve.aliases_hint"}}</span>
|
||||
@@ -154,11 +157,11 @@
|
||||
placeholder="{{t "serve.description_example"}}"></label>
|
||||
<div class="field-row">
|
||||
<label>{{t "serve.context_length"}}
|
||||
<input type="text" name="context_length" value="4096" required></label>
|
||||
<input type="text" name="context_length" value="{{.Context}}" required></label>
|
||||
<label>{{t "serve.max_output"}} <span class="opt">{{t "serve.optional"}}</span>
|
||||
<input type="text" name="max_output" value="1024"></label>
|
||||
<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
|
||||
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>
|
||||
{{if ne .Engine "strata"}}<label>{{t "serve.ctx_pool"}} <span class="opt">{{t "serve.ctx_pool_hint"}}</span>
|
||||
<input type="text" name="ctx_size" value="{{.CtxPool}}"></label>{{end}}
|
||||
</div>
|
||||
{{/* Where it runs is a choice with consequences, so it is a choice: the
|
||||
GPU classes the fleet has, and the CPU for llama.cpp. A profile name
|
||||
@@ -169,8 +172,8 @@
|
||||
<select name="placement">
|
||||
{{range .Placements}}<option value="{{.Value}}"{{if .Selected}} selected{{end}}>{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
|
||||
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>
|
||||
{{if ne .Engine "strata"}}<label>{{t "serve.gpus"}} <span class="opt">{{t "serve.gpus_hint"}}</span>
|
||||
<input type="text" name="gpus" value="{{.GPUs}}" size="3"></label>{{end}}
|
||||
{{if eq .Engine "llamacpp"}}<label>{{t "serve.parallel"}} <span class="opt">{{t "serve.parallel_hint"}}</span>
|
||||
<input type="text" name="parallel" value="{{.Parallel}}" size="3"></label>{{end}}
|
||||
</div>
|
||||
|
||||
Reference in new issue
Block a user