A picture or 3D model is added by its name: llmgw ctl model add triposr
flux-schnell and triposr come with llmgw as presets: their workflow, custom nodes, packages and licence as they were found and tried on the PoC, and a profile that says the card it needs rather than a class, so the entry places on any site's cards. TripoSR's workflow is built in, with the LLMGWPrepareForMesh node that keeps a picture's background out of the mesh, which the copy in the document had left out. `ctl model add PRESET` adds one, and has the nodes install ComfyUI when the cluster's settings name none; `ctl model presets` lists them; the console's add-model dialog offers them beside a definition file. A ComfyUI build is now also checked to keep its order on a Turing card, so a node's installed cu13 is not replaced by the cu126 build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
52898fbb80
commit
40b8948d7d
12 files changed
+537
-36
No files matched your search
+37
-7
@@ -1300,6 +1300,22 @@ waits, and sends the pictures back as `b64_json`.
|
||||
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
|
||||
GPU node like any other, started when asked for, and stopped when unused.
|
||||
|
||||
**The short way: a preset.** Some models come with llmgw, with their
|
||||
workflow, custom nodes, packages and licence already found and tried:
|
||||
|
||||
```
|
||||
llmgw ctl model presets
|
||||
llmgw ctl model add flux-schnell # pictures, Apache 2.0, a card of 24 GB, Volta or newer
|
||||
llmgw ctl model add triposr # 3D models from a picture, MIT, a card of 8 GB
|
||||
```
|
||||
|
||||
A preset's profile says the card it needs, not a class, so it places on
|
||||
whatever cards the site has. When the cluster's settings name no ComfyUI,
|
||||
adding one sets `engines.comfyui` to the version this release installs, and
|
||||
every node with an NVIDIA GPU installs it; a cluster that names a version,
|
||||
or `""` for none, keeps it. In the console it is **A** on the models screen.
|
||||
The rest of this section is for a model of your own.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
|
||||
@@ -1312,8 +1328,10 @@ digest llmgw holds for that version. GitHub publishes none for a tag's archive,
|
||||
so a version llmgw does not know is refused. The agent then builds a Python
|
||||
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
|
||||
dependencies are resolved as PyPI stood the day after the release. torch comes
|
||||
from PyPI (CUDA 13), or from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580. The install is checked on the node's GPU, and takes about 10 GB.
|
||||
from PyPI (CUDA 13), from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580, or from its CUDA 12.6 build on cards older than Turing (a V100):
|
||||
torch's CUDA 12.8 and newer builds have no kernels for them. The install is
|
||||
checked on the node's GPU, and takes about 10 GB.
|
||||
`engine_overrides` can keep it off nodes that should not have it.
|
||||
|
||||
Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
|
||||
@@ -2114,38 +2132,50 @@ Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the
|
||||
uploaded picture, which a `LoadImage` node reads; `{{seed}}`,
|
||||
`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that
|
||||
saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB,
|
||||
vertex colours kept. TripoSR (MIT) on any card of 8 GB:
|
||||
vertex colours kept.
|
||||
|
||||
TripoSR (MIT), on any card of 8 GB, comes with llmgw: `llmgw ctl model add
|
||||
triposr`. What that adds is this entry, and its workflow is built in:
|
||||
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: hf:stabilityai/TripoSR/model.ckpt
|
||||
comfyui:
|
||||
workflow: /etc/llmgw/workflows/triposr.json # on every control plane
|
||||
workflow: triposr # built in; a model of your own names a JSON file on every control plane
|
||||
output: meshes
|
||||
nodes:
|
||||
- repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR
|
||||
commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3
|
||||
requirements: [omegaconf, trimesh, scikit-image, imageio,
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers]
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers,
|
||||
rembg, onnxruntime]
|
||||
license:
|
||||
id: MIT
|
||||
name: MIT
|
||||
commercial: true
|
||||
url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE
|
||||
notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..."
|
||||
scaling: {min: 0, max: 1, idle_timeout: 10m}
|
||||
profiles:
|
||||
- {name: gpu, engine: comfyui, gpu_class: 6000-24g, tp: 1}
|
||||
- {name: gpu, engine: comfyui, tp: 1, gpu: {min_compute_capability: 7.0, min_memory_gb: 8}}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"5": {"class_type": "LLMGWPrepareForMesh", "inputs": {"image": ["1", 0], "mask": ["1", 1],
|
||||
"foreground_ratio": 0.85, "background": 0.5, "remove_background": "auto", "rembg_model": "u2net"}},
|
||||
"2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["1", 0],
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["5", 0],
|
||||
"geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
|
||||
"4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
|
||||
}
|
||||
```
|
||||
|
||||
`LLMGWPrepareForMesh` comes with the gate: it cuts the object out — by the
|
||||
picture's alpha, or by rembg where it has none — and centres it on a plain
|
||||
background, without which the background becomes geometry.
|
||||
|
||||
- **Custom nodes** are named by https repository and full commit (a branch
|
||||
is refused: it moves). The gate puts each in the replica's own
|
||||
`custom_nodes` and installs its `requirements` beside the replica, never
|
||||
|
||||
@@ -149,3 +149,29 @@ func TestARequirementsFileIsReadForPackagesAndNothingElse(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A card every build runs on keeps the order it had: a Turing node's
|
||||
// installed cu13 is still the first found, and is not installed again
|
||||
// because a build for older cards was added after it.
|
||||
func TestComfyUIBuildsAreTriedBestFirst(t *testing.T) {
|
||||
variants := func(h discovery.Host) string {
|
||||
builds, err := comfyuiCandidates("0.39.2", "", h)
|
||||
if err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
var v []string
|
||||
for _, b := range builds {
|
||||
v = append(v, b.Variant)
|
||||
}
|
||||
return strings.Join(v, ",")
|
||||
}
|
||||
volta := turingNode
|
||||
volta.MinComputeCapability = 7.0
|
||||
older := turingNode
|
||||
older.NVIDIADriver = "550.163.01"
|
||||
for h, want := range map[*discovery.Host]string{&turingNode: "cu13,cu129,cu126", &volta: "cu126", &older: "cu129,cu126"} {
|
||||
if got := variants(*h); got != want {
|
||||
t.Errorf("cc %.1f driver %s: %s, want %s", h.MinComputeCapability, h.NVIDIADriver, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -592,6 +592,18 @@ var comfyuiReleases = map[string]struct {
|
||||
"0.39.2": {"1d655e22cf546700fc1d6d2536e27701695c5832a51f63bb16c41866ac09be0b", "2026-10-07T21:58:07Z"},
|
||||
}
|
||||
|
||||
// ComfyUIVersion is the newest ComfyUI this agent installs: what a cluster
|
||||
// is set to when a model first needs it.
|
||||
func ComfyUIVersion() string {
|
||||
newest := ""
|
||||
for v := range comfyuiReleases {
|
||||
if newest == "" || versionBefore(newest, v) {
|
||||
newest = v
|
||||
}
|
||||
}
|
||||
return newest
|
||||
}
|
||||
|
||||
// ComfyUIRepo is where ComfyUI's source is published.
|
||||
const ComfyUIRepo = "Comfy-Org/ComfyUI"
|
||||
|
||||
|
||||
+52
-22
@@ -32,7 +32,7 @@ func newModelCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
Use: "model",
|
||||
Short: "Add or remove models while the cluster is running",
|
||||
}
|
||||
c.AddCommand(newModelAddCmd(client), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
c.AddCommand(newModelAddCmd(client), newModelPresetsCmd(), newModelRemoveCmd(client), newModelPlanCmd(client),
|
||||
newResourcesCmd(client), newClassCmd(client), newOwnerCmd(client),
|
||||
newTargetCmd(client, asJSON), newAdaptersCmd(client),
|
||||
newModelServeCmd(client), newModelUpdatesCmd(client, asJSON), newModelUpdateCmd(client),
|
||||
@@ -46,34 +46,42 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
var path string
|
||||
var skipCheck bool
|
||||
c := &cobra.Command{
|
||||
Use: "add -f <file.yaml>",
|
||||
Short: "Add a model from a definition file",
|
||||
Long: "The definition is the same YAML a model takes in server.yaml, so an\n" +
|
||||
Use: "add PRESET | -f <file.yaml>",
|
||||
Short: "Add a model that comes with llmgw, or one from a definition file",
|
||||
Long: "A preset is a model that comes with llmgw, its workflow, nodes and licence\n" +
|
||||
"already found and tried: " + strings.Join(catalog.PresetNames(), ", ") + " (llmgw ctl model presets).\n" +
|
||||
"Adding a ComfyUI one also has the nodes install ComfyUI, when the cluster\n" +
|
||||
"has not been told otherwise.\n\n" +
|
||||
"A definition is the same YAML a model takes in server.yaml, so an\n" +
|
||||
"entry can be moved between the two by copying it.\n\n" +
|
||||
"The catalog is rebuilt and validated before anything is written, so a\n" +
|
||||
"definition that collides with an existing name, or names a fallback that\n" +
|
||||
"does not exist, is refused rather than stored and then ignored on every\n" +
|
||||
"start. Once accepted it takes effect on the reconciler's next pass — no\n" +
|
||||
"restart — and survives one.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
if path == "" {
|
||||
return fmt.Errorf("--file is required: the model definition, as YAML")
|
||||
}
|
||||
// Read by koanf, the same parser and the same tags that read
|
||||
// server.yaml — so an entry can be moved between the two by copying
|
||||
// it, and a field that works in one cannot silently mean nothing in
|
||||
// the other.
|
||||
k := koanf.New(".")
|
||||
if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
|
||||
return fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
Example: " llmgw ctl model add triposr\n" +
|
||||
" llmgw ctl model add -f qwen.yaml",
|
||||
Args: cobra.MaximumNArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var m catalog.Model
|
||||
if err := k.UnmarshalWithConf("", &m, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||
return fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
if m.Name == "" {
|
||||
return fmt.Errorf("%s: the definition has no name", path)
|
||||
switch {
|
||||
case len(args) == 1 && path != "":
|
||||
return fmt.Errorf("a preset or a file, not both")
|
||||
case len(args) == 1:
|
||||
p, ok := catalog.PresetNamed(args[0])
|
||||
if !ok {
|
||||
return fmt.Errorf("no preset %q: there are %s; any other model is added from a file (-f)",
|
||||
args[0], strings.Join(catalog.PresetNames(), ", "))
|
||||
}
|
||||
m = p.Model
|
||||
case path == "":
|
||||
return fmt.Errorf("a preset (%s) or --file, the model definition as YAML",
|
||||
strings.Join(catalog.PresetNames(), ", "))
|
||||
default:
|
||||
var err error
|
||||
if m, err = readModelFile(path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
// Before the catalog is touched, because a source that does not
|
||||
@@ -107,6 +115,9 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
fmt.Fprintf(cmd.OutOrStdout(),
|
||||
"%s added. It will be placed on the next reconcile pass; watch it with:\n"+
|
||||
" llmgw ctl models\n", m.Name)
|
||||
if m.ServesComfyUI() && !m.IsExternal() {
|
||||
return ensureComfyUI(cmd, client, m.Name)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
@@ -116,6 +127,25 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
return c
|
||||
}
|
||||
|
||||
// readModelFile reads a model's definition. By koanf, the same parser and
|
||||
// the same tags that read server.yaml — so an entry can be moved between the
|
||||
// two by copying it, and a field that works in one cannot silently mean
|
||||
// nothing in the other.
|
||||
func readModelFile(path string) (catalog.Model, error) {
|
||||
var m catalog.Model
|
||||
k := koanf.New(".")
|
||||
if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
|
||||
return m, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
if err := k.UnmarshalWithConf("", &m, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||
return m, fmt.Errorf("%s: %w", path, err)
|
||||
}
|
||||
if m.Name == "" {
|
||||
return m, fmt.Errorf("%s: the definition has no name", path)
|
||||
}
|
||||
return m, nil
|
||||
}
|
||||
|
||||
func newModelRemoveCmd(client func() *ctlClient) *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "remove <name>",
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/settings"
|
||||
)
|
||||
|
||||
func newModelPresetsCmd() *cobra.Command {
|
||||
return &cobra.Command{
|
||||
Use: "presets",
|
||||
Short: "The models that come with llmgw, added by name",
|
||||
Long: "Each is a catalog entry whose workflow, custom nodes, packages and licence\n" +
|
||||
"were found and tried, and whose profile says the card it needs rather than\n" +
|
||||
"a class, so it places on any site's cards. Add one with\n" +
|
||||
"`llmgw ctl model add NAME`.",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "NAME\tMAKES\tCARD\tLICENCE\tWHAT IT IS FOR")
|
||||
for _, p := range catalog.Presets() {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", p.Model.Name, p.Makes, presetCard(p.Model),
|
||||
presetLicence(p.Model), p.Model.Description)
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func presetCard(m catalog.Model) string {
|
||||
if len(m.Profiles) == 0 || m.Profiles[0].GPU == nil {
|
||||
return "—"
|
||||
}
|
||||
return m.Profiles[0].GPU.String()
|
||||
}
|
||||
|
||||
func presetLicence(m catalog.Model) string {
|
||||
if m.License == nil {
|
||||
return "—"
|
||||
}
|
||||
l := m.License.Name
|
||||
if c := m.License.Commercial; c != nil && !*c {
|
||||
l += ", not commercial"
|
||||
}
|
||||
return l
|
||||
}
|
||||
|
||||
// ensureComfyUI has the nodes install ComfyUI for a model that needs it,
|
||||
// unless the cluster names a version already, or none on purpose.
|
||||
func ensureComfyUI(cmd *cobra.Command, client func() *ctlClient, model string) error {
|
||||
out := cmd.OutOrStdout()
|
||||
path := []string{"engines", catalog.EngineComfyUI}
|
||||
var cur adminapi.SettingsView
|
||||
if err := client().get(cmd.Context(), "/api/v1/settings", &cur); err != nil {
|
||||
return err
|
||||
}
|
||||
doc, err := settings.FromBytes("settings", []byte(cur.Body))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
version := provision.ComfyUIVersion()
|
||||
if n := doc.Node(path); n != nil {
|
||||
if n.Value == "" {
|
||||
fmt.Fprintf(out, "engines.comfyui is set to none, so the nodes install no ComfyUI for it: set a version "+
|
||||
"(llmgw ctl config set engines.comfyui %s) or name one installed by hand in agent.yaml\n", version)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintf(out, "Nodes with an NVIDIA GPU install ComfyUI %s for it, about 10 GB each:\n", version)
|
||||
if err := doc.SetYAML(path, version); err != nil {
|
||||
return err
|
||||
}
|
||||
b, err := doc.Bytes()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return putSettings(cmd, client, cur, b, "ComfyUI for "+model)
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
)
|
||||
|
||||
// presetAdmin is an admin API holding settings, which records the models
|
||||
// added and the settings put.
|
||||
type presetAdmin struct {
|
||||
mu sync.Mutex
|
||||
settings string
|
||||
added []catalog.Model
|
||||
put []string
|
||||
}
|
||||
|
||||
func (a *presetAdmin) client(t *testing.T) func() *ctlClient {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
body, _ := io.ReadAll(r.Body)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch {
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/api/v1/models":
|
||||
var m catalog.Model
|
||||
_ = json.Unmarshal(body, &m)
|
||||
a.added = append(a.added, m)
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"name": m.Name})
|
||||
case r.Method == http.MethodGet && r.URL.Path == "/api/v1/settings":
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 1, Body: a.settings})
|
||||
case r.Method == http.MethodPut && r.URL.Path == "/api/v1/settings":
|
||||
var req adminapi.SettingsRequest
|
||||
_ = json.Unmarshal(body, &req)
|
||||
a.put = append(a.put, req.Body)
|
||||
a.settings = req.Body
|
||||
_ = json.NewEncoder(w).Encode(adminapi.SettingsView{Version: 2, Body: req.Body})
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return func() *ctlClient { return newCtlClient(strings.TrimPrefix(srv.URL, "http://"), "") }
|
||||
}
|
||||
|
||||
func runModelAdd(t *testing.T, client func() *ctlClient, args ...string) (string, error) {
|
||||
t.Helper()
|
||||
cmd := newModelAddCmd(client)
|
||||
var out bytes.Buffer
|
||||
cmd.SetOut(&out)
|
||||
cmd.SetErr(&out)
|
||||
cmd.SetArgs(append(args, "--no-source-check"))
|
||||
err := cmd.ExecuteContext(t.Context())
|
||||
return out.String(), err
|
||||
}
|
||||
|
||||
func TestAPresetIsAddedByNameAndHasTheNodesInstallComfyUI(t *testing.T) {
|
||||
a := &presetAdmin{settings: "router:\n retries: 2\n"}
|
||||
out, err := runModelAdd(t, a.client(t), "triposr")
|
||||
if err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
if len(a.added) != 1 || a.added[0].Name != "triposr" || a.added[0].ComfyUI.Workflow != "triposr" {
|
||||
t.Fatalf("added %+v", a.added)
|
||||
}
|
||||
if len(a.put) != 1 || !strings.Contains(a.put[0], "comfyui: 0.39.2") || !strings.Contains(a.put[0], "retries: 2") {
|
||||
t.Fatalf("settings put %q", a.put)
|
||||
}
|
||||
if !strings.Contains(out, "install ComfyUI 0.39.2") {
|
||||
t.Errorf("not said:\n%s", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClusterThatNamesItsComfyUIKeepsIt(t *testing.T) {
|
||||
for _, settings := range []string{"engines:\n comfyui: 0.38.0\n", "engines:\n comfyui: \"\"\n"} {
|
||||
a := &presetAdmin{settings: settings}
|
||||
out, err := runModelAdd(t, a.client(t), "flux-schnell")
|
||||
if err != nil {
|
||||
t.Fatalf("%v\n%s", err, out)
|
||||
}
|
||||
if len(a.put) != 0 {
|
||||
t.Errorf("%q: settings changed to %q", settings, a.put)
|
||||
}
|
||||
if none := strings.Contains(settings, `""`); none != strings.Contains(out, "set to none") {
|
||||
t.Errorf("%q: %s", settings, out)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnknownPresetNamesTheOnesThereAre(t *testing.T) {
|
||||
a := &presetAdmin{}
|
||||
_, err := runModelAdd(t, a.client(t), "sdxl")
|
||||
if err == nil || !strings.Contains(err.Error(), "flux-schnell, triposr") || len(a.added) != 0 {
|
||||
t.Errorf("%v, added %v", err, a.added)
|
||||
}
|
||||
if _, err := runModelAdd(t, a.client(t), "triposr", "-f", "x.yaml"); err == nil {
|
||||
t.Error("a preset and a file were both taken")
|
||||
}
|
||||
}
|
||||
@@ -1237,7 +1237,7 @@ func (m consoleModel) panel() (title, body, keys string) {
|
||||
strings.Join(md.Profiles, ","), upd, warn})
|
||||
}
|
||||
if m.inPanel {
|
||||
keys = "↑↓ row · Enter actions · A add from a file · ← back · Q leave"
|
||||
keys = "↑↓ row · Enter actions · A add a model · ← back · Q leave"
|
||||
}
|
||||
return "Models", "\n" + table([]string{"MODEL", "READY", "STATUS", "CLASS", "PROFILES", "UPDATE", ""}, rows, m.row, pw, m.inPanel), keys
|
||||
case cReplicas:
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
tea "github.com/charmbracelet/bubbletea"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
)
|
||||
|
||||
@@ -253,19 +254,37 @@ func (m consoleModel) mirrorPanel(pw int) (string, string, string) {
|
||||
return "Mirror", head + table([]string{"MODEL", "VERSION", "FILE", "STATE", "SIZE", "USED BY"}, rows, m.row, pw, m.inPanel), keys
|
||||
}
|
||||
|
||||
// addModelForm adds a model from a definition file on this machine: the
|
||||
// same YAML a model takes in server.yaml.
|
||||
// fromFile is the add-model choice that reads a definition file rather
|
||||
// than naming a preset.
|
||||
const fromFile = "from a file"
|
||||
|
||||
// addModelForm adds a model that comes with llmgw, or one from a definition
|
||||
// file on this machine: the same YAML a model takes in server.yaml.
|
||||
func (m *consoleModel) addModelForm() {
|
||||
server := "--server=" + m.client.base
|
||||
var note strings.Builder
|
||||
note.WriteString("A model that comes with llmgw, tried and with its licence:\n")
|
||||
for _, p := range catalog.Presets() {
|
||||
fmt.Fprintf(¬e, " %s: %s, %s, %s\n", p.Model.Name, p.Makes, presetCard(p.Model), presetLicence(p.Model))
|
||||
}
|
||||
note.WriteString("A ComfyUI one has the nodes install ComfyUI if the cluster has none.\n" +
|
||||
"Or a definition file: the YAML a model takes in server.yaml's models.")
|
||||
m.openForm(&formDialog{title: "Add a model",
|
||||
note: "From a definition file on this machine: the YAML a model takes in\nserver.yaml's models. Serve a model, under Do, needs none.",
|
||||
note: note.String(),
|
||||
fields: []formField{
|
||||
textF("file", "", "like /root/qwen.yaml"),
|
||||
choiceF("model", append(catalog.PresetNames(), fromFile), catalog.PresetNames()[0]),
|
||||
textF("file", "", "for "+fromFile+": like /root/qwen.yaml"),
|
||||
},
|
||||
submit: func(m *consoleModel, v []string) (tea.Cmd, error) {
|
||||
if _, err := os.Stat(v[0]); err != nil {
|
||||
if v[0] != fromFile {
|
||||
return m.runCLI("Add "+v[0], "ctl", server, "model", "add", v[0]), nil
|
||||
}
|
||||
if v[1] == "" {
|
||||
return nil, fmt.Errorf("a file, or a model that comes with llmgw")
|
||||
}
|
||||
if _, err := os.Stat(v[1]); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return m.runCLI("Add a model", "ctl", server, "model", "add", "-f", v[0]), nil
|
||||
return m.runCLI("Add a model", "ctl", server, "model", "add", "-f", v[1]), nil
|
||||
}})
|
||||
}
|
||||
@@ -15,6 +15,9 @@ import (
|
||||
|
||||
func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
for _, name := range Builtins() {
|
||||
if raw, _ := builtin.ReadFile("workflows/" + name + ".json"); strings.Contains(string(raw), "{{image}}") {
|
||||
continue // made from a picture: TestTheBuiltInMeshWorkflowFillsIn
|
||||
}
|
||||
tpl, err := Load(name)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
@@ -37,6 +40,30 @@ func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheBuiltInMeshWorkflowFillsIn(t *testing.T) {
|
||||
tpl, err := Load("triposr")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "model.ckpt", Image: "llmgw-1.png"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := json.Marshal(g)
|
||||
s := string(b)
|
||||
// The picture is cut out and centred before TripoSR sees it: without
|
||||
// that, its background becomes geometry.
|
||||
for _, want := range []string{`"image":"llmgw-1.png"`, `"model":"model.ckpt"`, `"geometry_resolution":256`,
|
||||
`"class_type":"LLMGWPrepareForMesh"`, `"reference_image":["5",0]`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("%s not in %s", want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "{{") {
|
||||
t.Errorf("a placeholder was left: %s", s)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
|
||||
tpl, _ := Load("flux-schnell")
|
||||
g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"5": {"class_type": "LLMGWPrepareForMesh", "inputs": {"image": ["1", 0], "mask": ["1", 1],
|
||||
"foreground_ratio": 0.85, "background": 0.5, "remove_background": "auto", "rembg_model": "u2net"}},
|
||||
"2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["5", 0],
|
||||
"geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
|
||||
"4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Preset is a catalog entry that comes with llmgw: a model whose workflow,
|
||||
// custom nodes, packages and licence somebody has already found and tried,
|
||||
// so that adding it is naming it.
|
||||
//
|
||||
// A ComfyUI model cannot be worked out from its repository the way a GGUF
|
||||
// can: which workflow runs it, which node at which commit, which packages at
|
||||
// which ranges, and under which licence what it makes is shipped are learned
|
||||
// by running it. Written in a document, they were copied by hand on every
|
||||
// site, and a line left out — the node that cuts the picture out — gave
|
||||
// meshes with their background as geometry.
|
||||
type Preset struct {
|
||||
// Makes is what it makes: pictures or meshes.
|
||||
Makes string
|
||||
Model Model
|
||||
}
|
||||
|
||||
// Presets are the entries that come with llmgw, by name. Each profile says
|
||||
// the card it needs rather than naming a class, so the entry places on any
|
||||
// site's cards.
|
||||
func Presets() []Preset {
|
||||
yes := true
|
||||
return []Preset{
|
||||
{
|
||||
Makes: OutputPictures,
|
||||
Model: Model{
|
||||
Name: "flux-schnell",
|
||||
Description: "FLUX.1 [schnell]: pictures from a prompt, in four steps",
|
||||
Source: "hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors",
|
||||
ComfyUI: &ComfyUI{Workflow: "flux-schnell"},
|
||||
License: &License{
|
||||
ID: "Apache-2.0", Name: "Apache-2.0", Commercial: &yes,
|
||||
URL: "https://huggingface.co/black-forest-labs/FLUX.1-schnell",
|
||||
Notice: "Made with FLUX.1 [schnell] (Black Forest Labs, Apache 2.0).",
|
||||
},
|
||||
Scaling: Scaling{Min: 0, Max: 1, IdleTimeout: 15 * time.Minute},
|
||||
// The checkpoint is 17 GB with its text encoders; a card with
|
||||
// no fp8 has its weights upcast as they load.
|
||||
Profiles: []Profile{{Name: "gpu", Engine: EngineComfyUI, TP: 1,
|
||||
GPU: &GPUNeed{MinComputeCapability: 7.0, MinMemoryGB: 24}}},
|
||||
},
|
||||
},
|
||||
{
|
||||
Makes: OutputMeshes,
|
||||
Model: Model{
|
||||
Name: "triposr",
|
||||
Description: "TripoSR: a 3D model (GLB) from a picture of one object on a plain background, as a job",
|
||||
Source: "hf:stabilityai/TripoSR/model.ckpt",
|
||||
ComfyUI: &ComfyUI{
|
||||
Workflow: "triposr",
|
||||
Output: OutputMeshes,
|
||||
Nodes: []CustomNode{{
|
||||
Repo: "https://github.com/flowtyone/ComfyUI-Flowty-TripoSR",
|
||||
Commit: "a0c94ac60a7cc062604f61aeeea6d0d493521de3",
|
||||
// transformers 5 renames the checkpoint's keys; rembg
|
||||
// and onnxruntime cut out a picture with no alpha.
|
||||
Requirements: []string{"omegaconf", "trimesh", "scikit-image", "imageio",
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", "tokenizers", "rembg", "onnxruntime"},
|
||||
}},
|
||||
},
|
||||
License: &License{
|
||||
ID: "MIT", Name: "MIT", Commercial: &yes,
|
||||
URL: "https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE",
|
||||
Notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). Its image encoder is " +
|
||||
"facebook/dino-vitb16 (Apache 2.0). TripoSR was trained on a filtered subset of " +
|
||||
"Objaverse, some of whose objects carry their own licences.",
|
||||
},
|
||||
Scaling: Scaling{Min: 0, Max: 1, IdleTimeout: 10 * time.Minute},
|
||||
Profiles: []Profile{{Name: "gpu", Engine: EngineComfyUI, TP: 1,
|
||||
GPU: &GPUNeed{MinComputeCapability: 7.0, MinMemoryGB: 8}}},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// PresetNamed is the preset of that name.
|
||||
func PresetNamed(name string) (Preset, bool) {
|
||||
ps := Presets()
|
||||
i := slices.IndexFunc(ps, func(p Preset) bool { return p.Model.Name == name })
|
||||
if i < 0 {
|
||||
return Preset{}, false
|
||||
}
|
||||
return ps[i], true
|
||||
}
|
||||
|
||||
// PresetNames are the presets' names, in order.
|
||||
func PresetNames() []string {
|
||||
var out []string
|
||||
for _, p := range Presets() {
|
||||
out = append(out, p.Model.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
)
|
||||
|
||||
func TestEveryPresetIsACatalogEntryThatLoads(t *testing.T) {
|
||||
var models []Model
|
||||
for _, p := range Presets() {
|
||||
models = append(models, p.Model)
|
||||
}
|
||||
cat, err := New(models)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, p := range Presets() {
|
||||
m, ok := cat.Resolve(p.Model.Name)
|
||||
if !ok {
|
||||
t.Fatalf("%s is not in the catalog", p.Model.Name)
|
||||
}
|
||||
if p.Makes == OutputMeshes && !m.MakesMeshes() || p.Makes == OutputPictures && !m.MakesImages() {
|
||||
t.Errorf("%s does not make %s", m.Name, p.Makes)
|
||||
}
|
||||
// A shipped entry has its workflow inside llmgw, not in a file a
|
||||
// site would have to write.
|
||||
w, ckpt := m.ComfyWorkflow()
|
||||
if _, err := comfy.Load(w); err != nil {
|
||||
t.Errorf("%s: %v", m.Name, err)
|
||||
}
|
||||
if ckpt == "" {
|
||||
t.Errorf("%s has no checkpoint", m.Name)
|
||||
}
|
||||
if m.License == nil || m.License.ID == "" || m.License.Commercial == nil {
|
||||
t.Errorf("%s does not say its licence for a machine to match", m.Name)
|
||||
}
|
||||
for _, pr := range m.Profiles {
|
||||
if pr.GPUClass != "" || pr.GPU == nil {
|
||||
t.Errorf("%s/%s names a class: a preset fits any site's cards", m.Name, pr.Name)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPresetIsFoundByName(t *testing.T) {
|
||||
if p, ok := PresetNamed("triposr"); !ok || p.Model.ComfyUI.Workflow != "triposr" {
|
||||
t.Errorf("triposr: %+v %v", p, ok)
|
||||
}
|
||||
if _, ok := PresetNamed("sdxl"); ok {
|
||||
t.Error("a preset that is not there was found")
|
||||
}
|
||||
// Each call is a fresh copy: a caller changing one changes nobody's.
|
||||
a, _ := PresetNamed("triposr")
|
||||
a.Model.ComfyUI.Nodes[0].Requirements[0] = "x"
|
||||
if b, _ := PresetNamed("triposr"); b.Model.ComfyUI.Nodes[0].Requirements[0] == "x" {
|
||||
t.Error("a preset was changed by a caller")
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user