From b66dd9e6a7d1570f6625b524e4476aefe2ce6491 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Sat, 10 Oct 2026 01:01:55 +0200 Subject: [PATCH] An on-demand model that is up reads as ready, not as not placed ctl models said "not placed" for a model with min 0 woken by a request and serving; it now says "ready (on demand)". And the docs: pictures on the cluster's own cards, with engine comfyui behind its gate. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/install.md | 61 ++++++++++++++++++++++++++++++++++--- internal/cli/ctl.go | 3 ++ internal/cli/status_test.go | 2 ++ 3 files changed, 61 insertions(+), 5 deletions(-) diff --git a/docs/install.md b/docs/install.md index 52b3190..9050a69 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1272,9 +1272,57 @@ by name. The cluster page and `ctl models` mark it `external`. ### Pictures from ComfyUI -A ComfyUI server is an external model with `api: comfyui`. Clients ask it for -pictures at OpenAI's `/v1/images/generations`; llmgw fills a ComfyUI workflow -with the request, queues it, waits, and sends the pictures back as `b64_json`: +Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them +made by ComfyUI: it fills a ComfyUI workflow with the request, queues it, +waits, and sends the pictures back as `b64_json`. + +**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a +GPU node like any other, started when asked for, and stopped when unused. The +node must offer the engine. For now ComfyUI is installed by hand: a directory +with `main.py` and its environment in `.venv`, readable by the user engines run +as (`llmgw-engine`). Name it in that node's `agent.yaml`: + +```yaml +engines: + comfyui: + dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python +``` + +Then add the model: + +```yaml +- name: flux-schnell + source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors + comfyui: + workflow: flux-schnell # or flux-dev, or a JSON file + scaling: {min: 0, max: 1, idle_timeout: 15m} + profiles: + - name: gpu + engine: comfyui + gpu_class: 6000-24g + tp: 1 +``` + +The source is fetched as for any model. It can be one file, such as the +single-file checkpoints the built-in workflows load. It can also be a mirror +set (a URL ending in `/`) for a model in several files, such as a diffusion +model, its text encoders and its VAE. ComfyUI sees every file of a set by its +own name, so a workflow of your own can load each one. A set must name the +file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file +is named by its own file name. + +ComfyUI asks nobody for a key. The agent therefore starts it behind a gate +(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate +listens on the replica's port, demands the replica's key, and lets through only +what the router asks: queueing, waiting, fetching a picture and cancelling. +Uploads and node installs are refused. Each replica has its own directory, +`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw +ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`. +`--disable-smart-memory` frees the card between pictures, at the cost of loading +the model again for each one. + +**On a ComfyUI server outside the cluster.** Make it an external model with +`api: comfyui`: ```yaml - name: flux-schnell @@ -1286,6 +1334,8 @@ with the request, queues it, waits, and sends the pictures back as `b64_json`: workflow: flux-schnell # or flux-dev, or a JSON file ``` +Like any external model, its requests count as leaving the cluster. + `flux-schnell` (the default) and `flux-dev` are built in. They suit the single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`. For any other model, export the workflow from ComfyUI with **Save (API)** and @@ -1320,8 +1370,9 @@ auth: flux-schnell: {prompt_per_1k: 0, per_image: 20000} ``` -Start ComfyUI with `--disable-smart-memory` on a card that llmgw also places -models on. Without it, ComfyUI keeps the model on the card after each picture. +An external ComfyUI on a card that llmgw also places models on should be +started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on +the card after each picture, and llmgw sees that memory as somebody else's. ### Federating clusters diff --git a/internal/cli/ctl.go b/internal/cli/ctl.go index d699094..37aac36 100644 --- a/internal/cli/ctl.go +++ b/internal/cli/ctl.go @@ -331,6 +331,9 @@ func modelStatus(m adminapi.ModelView) string { return "stopped" case m.Ready >= m.Min && m.Min > 0: return "ready" + case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total: + // Asked for, and up until it goes unused for long enough. + return "ready (on demand)" case m.Total > m.Ready: return "starting" default: diff --git a/internal/cli/status_test.go b/internal/cli/status_test.go index ea77edb..8b8ecff 100644 --- a/internal/cli/status_test.go +++ b/internal/cli/status_test.go @@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) { {"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"}, {"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"}, {"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"}, + {"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"}, + {"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"}, } { t.Run(tc.name, func(t *testing.T) { if got := modelStatus(tc.view); got != tc.want {