An on-demand model that is up reads as ready, not as not placed

ctl models said "not placed" for a model with min 0 woken by a request and
serving; it now says "ready (on demand)". And the docs: pictures on the
cluster's own cards, with engine comfyui behind its gate.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-10 01:01:55 +02:00
1 parent 354461d3bf
commit b66dd9e6a7
3 files changed
+61 -5

No files matched your search

+56 -5
View File
@@ -1272,9 +1272,57 @@ by name. The cluster page and `ctl models` mark it `external`.
### Pictures from ComfyUI
A ComfyUI server is an external model with `api: comfyui`. Clients ask it for
pictures at OpenAI's `/v1/images/generations`; llmgw fills a ComfyUI workflow
with the request, queues it, waits, and sends the pictures back as `b64_json`:
Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
waits, and sends the pictures back as `b64_json`.
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
GPU node like any other, started when asked for, and stopped when unused. The
node must offer the engine. For now ComfyUI is installed by hand: a directory
with `main.py` and its environment in `.venv`, readable by the user engines run
as (`llmgw-engine`). Name it in that node's `agent.yaml`:
```yaml
engines:
comfyui:
dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python
```
Then add the model:
```yaml
- name: flux-schnell
source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
comfyui:
workflow: flux-schnell # or flux-dev, or a JSON file
scaling: {min: 0, max: 1, idle_timeout: 15m}
profiles:
- name: gpu
engine: comfyui
gpu_class: 6000-24g
tp: 1
```
The source is fetched as for any model. It can be one file, such as the
single-file checkpoints the built-in workflows load. It can also be a mirror
set (a URL ending in `/`) for a model in several files, such as a diffusion
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
own name, so a workflow of your own can load each one. A set must name the
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
is named by its own file name.
ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
listens on the replica's port, demands the replica's key, and lets through only
what the router asks: queueing, waiting, fetching a picture and cancelling.
Uploads and node installs are refused. Each replica has its own directory,
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
`--disable-smart-memory` frees the card between pictures, at the cost of loading
the model again for each one.
**On a ComfyUI server outside the cluster.** Make it an external model with
`api: comfyui`:
```yaml
- name: flux-schnell
@@ -1286,6 +1334,8 @@ with the request, queues it, waits, and sends the pictures back as `b64_json`:
workflow: flux-schnell # or flux-dev, or a JSON file
```
Like any external model, its requests count as leaving the cluster.
`flux-schnell` (the default) and `flux-dev` are built in. They suit the
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
For any other model, export the workflow from ComfyUI with **Save (API)** and
@@ -1320,8 +1370,9 @@ auth:
flux-schnell: {prompt_per_1k: 0, per_image: 20000}
```
Start ComfyUI with `--disable-smart-memory` on a card that llmgw also places
models on. Without it, ComfyUI keeps the model on the card after each picture.
An external ComfyUI on a card that llmgw also places models on should be
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
the card after each picture, and llmgw sees that memory as somebody else's.
### Federating clusters
+3
View File
@@ -331,6 +331,9 @@ func modelStatus(m adminapi.ModelView) string {
return "stopped"
case m.Ready >= m.Min && m.Min > 0:
return "ready"
case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total:
// Asked for, and up until it goes unused for long enough.
return "ready (on demand)"
case m.Total > m.Ready:
return "starting"
default:
+2
View File
@@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) {
{"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"},
{"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"},
{"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"},
{"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"},
{"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"},
} {
t.Run(tc.name, func(t *testing.T) {
if got := modelStatus(tc.view); got != tc.want {