CHANGELOG: one list
This commit is contained in:
1 parent
7465c24830
commit
d6f8f433f4
24 files changed
+437
-22
No files matched your search
@@ -11186,6 +11186,13 @@
|
||||
"expert_parallel": {
|
||||
"type": "boolean",
|
||||
"description": "ExpertParallel spreads a mixture-of-experts model's experts over the cards, whole, rather than slicing each across them: vLLM's --enable-expert-parallel, for a model of experts on more than one card."
|
||||
},
|
||||
"strata_family": {
|
||||
"type": "string",
|
||||
"description": "StrataFamily and StrataSize are Strata's --family and --size, for an option on Strata: one card, and the experts in the node's RAM, which NeedGB and HaveGB are then about."
|
||||
},
|
||||
"strata_size": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -11536,6 +11543,10 @@
|
||||
"type": "boolean",
|
||||
"description": "Fallbacks adds the next best ways, lower in priority, so a model whose first choice is busy runs on another rather than waiting."
|
||||
},
|
||||
"engine": {
|
||||
"type": "string",
|
||||
"description": "Engine is the engine the file runs on: \"strata\" for a family Strata runs, \"llamacpp\" to keep it off Strata; empty, the plan's best."
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Name is the catalog name; the repository suggests one otherwise."
|
||||
|
||||
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -745,6 +746,28 @@ you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Strata: a large model of experts on one card
|
||||
|
||||
Strata runs a few mixture-of-experts models — Qwen3.8-Flash-Next (125B) and its
|
||||
Swift, Coder, Unsloth, Gyro and NVFP4 versions, Qwen3.6-35B-A3B, DeepSeek
|
||||
V4.1-Flash — with the experts in the node's RAM and the rest on one card of 12
|
||||
GB or more. Where it is installed, the plan offers it for a repository of one
|
||||
of those families:
|
||||
|
||||
```
|
||||
llmgw ctl model plan ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF
|
||||
llmgw ctl model serve ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF \
|
||||
--file Qwen3.8-Flash-Next-GSQ-RCO-IQ2_XS-00001-of-00002.gguf --engine strata
|
||||
```
|
||||
|
||||
The Hugging Face explorer shows the same ("1× … with Strata, N GB of RAM"),
|
||||
and the mirror's serve form serves such a version on Strata. A Strata profile
|
||||
says its family and size (`--family qwen --size IQ2_XS`), which the catalog
|
||||
checks against what Strata runs, and reserves the RAM the size keeps its experts
|
||||
in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
|
||||
first start prepares the model on the node, which takes minutes, and fetches a
|
||||
draft layer of about 5 GB from the internet.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
|
||||
|
||||
Reference in new issue
Block a user