Strata runs as an engine: a 125B MoE on one card
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client. - engines/strata.go: the adapter. The agent starts llmgw strata-serve, which runs Strata's setup the way its desktop app does (STRATA_CONTROL_INSTALL: installs nothing, writes only under the replica's and the node's directories) on the fetched GGUF files, points the configuration it writes at the installed engine, and execs Strata's server. The key goes in STRATA_API_KEY, never on a command line. - A profile's args go to setup: --family, --size (setup's --model, which is the source everywhere else), --context, --parallel, ... Flags that move files or pick the engine or GPU are refused. - provision: the source pinned by digest in a Python environment, and the release's prebuilt engine unpacked beside it, checked against the digest GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9, driver 575). Python builds can now carry release assets. - Metrics are vLLM's names under /metrics/prometheus. - The catalog wants one card and a family, and a directory, not one .gguf. Opt-in: not in DefaultEngines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
eb9c448d68
commit
c0d5194ea0
19 files changed
+906
-4
No files matched your search
@@ -396,6 +396,9 @@ rather than repeated.
|
||||
| `timeout` | `time.Duration` | [Service](#service) |
|
||||
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `dir` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `engine` | `string` | [StrataConfig](#strataconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
| `retain_hours` | `int` | [Threads](#threads) |
|
||||
@@ -476,6 +479,7 @@ rather than repeated.
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -3415,6 +3419,27 @@ entries ask. In force on every control plane within seconds.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## StrataConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is Strata's source directory, where setup.py and serve/ are.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
### `engine`
|
||||
|
||||
`string`
|
||||
|
||||
Engine is the directory of its prebuilt engine, where the strata
|
||||
binary and the CUDA libraries it ships are; Dir's engine/ when unset.
|
||||
|
||||
## TRTLLMConfig
|
||||
|
||||
### `python`
|
||||
@@ -3987,6 +4012,15 @@ ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
### `strata`
|
||||
|
||||
`StrataConfig`
|
||||
|
||||
Strata serves a few mixture-of-experts models from RAM and one card;
|
||||
offered by a node whose Strata is named here, or that installed it.
|
||||
|
||||
Takes a [StrataConfig](#strataconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
Reference in new issue
Block a user