Strata runs as an engine: a 125B MoE on one card

Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.

- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
  which runs Strata's setup the way its desktop app does
  (STRATA_CONTROL_INSTALL: installs nothing, writes only under the
  replica's and the node's directories) on the fetched GGUF files, points
  the configuration it writes at the installed engine, and execs Strata's
  server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
  is the source everywhere else), --context, --parallel, ... Flags that
  move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
  release's prebuilt engine unpacked beside it, checked against the digest
  GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
  driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.

Opt-in: not in DefaultEngines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-29 15:09:13 +01:00
1 parent eb9c448d68
commit c0d5194ea0
19 files changed
+906 -4

No files matched your search

+34
View File
@@ -396,6 +396,9 @@ rather than repeated.
| `timeout` | `time.Duration` | [Service](#service) |
| `stopped` | `bool` | [ServingSettings](#servingsettings) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `dir` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [StrataConfig](#strataconfig) |
| `engine` | `string` | [StrataConfig](#strataconfig) |
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
| `store` | `string` | [Threads](#threads) |
| `retain_hours` | `int` | [Threads](#threads) |
@@ -476,6 +479,7 @@ rather than repeated.
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
| `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -3415,6 +3419,27 @@ entries ask. In force on every control plane within seconds.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## StrataConfig
### `dir`
`string`
Dir is Strata's source directory, where setup.py and serve/ are.
### `python`
`string`
Python is the interpreter of its environment; Dir's .venv when unset.
### `engine`
`string`
Engine is the directory of its prebuilt engine, where the strata
binary and the CUDA libraries it ships are; Dir's engine/ when unset.
## TRTLLMConfig
### `python`
@@ -3987,6 +4012,15 @@ ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
Takes a [ComfyUIConfig](#comfyuiconfig) block.
### `strata`
`StrataConfig`
Strata serves a few mixture-of-experts models from RAM and one card;
offered by a node whose Strata is named here, or that installed it.
Takes a [StrataConfig](#strataconfig) block.
## guard.Config
### `enabled`