ComfyUI models that agree on their nodes share an environment; a licence says what a machine can match
CI / test (push) Failing after 6m21s

A ComfyUI model may name an environment: the models of one environment
agree on every custom node they both name and on every package range,
so one replica can carry all of them and ComfyUI moves between their
weights itself rather than the card being handed over with a cold start
each time. The catalog refuses an environment that cannot share a
Python, naming both models; Catalog.Environments gives each one's
models, the union of their nodes and their weight sources, for
placement to build on.

A licence gains an id (SPDX where there is one), whether its output may
be used commercially, the regions it excludes, a revenue cap and the
attribution it asks for, so a person's or a site's choice of licences
can be matched without reading prose.

BACKLOG gains section H: ComfyUI as a platform.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-11 09:19:50 +02:00
1 parent 0639edfd3b
commit b43e66c95e
7 files changed
+549

No files matched your search

+127
View File
@@ -1270,6 +1270,133 @@ the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
jobs list|get|cancel`, the console's Jobs view. The page sends down the
same path as the API, like `imagesSend` does.
## H. ComfyUI as a platform (written 2026-10-11)
G made one model at a time work: flux for pictures, TripoSR and SF3D for
meshes, each with its own endpoint, its own request shape and its own
replica. The user's direction (2026-10-11): llmgw is a generic platform,
and should serve whatever ComfyUI can run — pictures, edits, video,
audio, meshes, upscaling, segmentation — as data in the catalog rather
than code per model. Two decisions shape everything below:
- **The licence is metadata, not a filter.** llmgw does not leave a
model out because of its licence. The licence travels with the model
and with every file it makes; what a site will not serve is that
site's policy (H6). The game studio's EU constraint is one such policy,
not a decision the code makes for everyone.
- **The install fits the hardware.** The PoC has one Turing card; other
sites will have Ampere, Ada and Hopper. A catalog entry must be
writable once and placed sensibly anywhere: fp16 on Turing, bf16 or
fp8 where the card has them, flash-attention models only where it
runs, CUDA extensions built for the card that runs them (H2, H4).
The first set of models proves the structure at its corners rather than
by licence: Qwen-Image-Edit (several weight files, two pictures in),
Wan 2.2 small (video out, a long job), ACE-Step (audio out), SF3D (a
package not on PyPI, gated weights), Hunyuan3D (a licence that excludes
the EU, which H6 must be able to say no to).
### H1. A workflow says what it takes and makes (M)
- Catalog: `comfyui.inputs` — a list of `{name, type, required, default,
min, max, choices, max_mb, doc}`, type one of text, int, float, bool,
choice, image, mask, audio, video, file. Each is a placeholder of the
same name in the workflow. The template's own placeholders are checked
against the list when the catalog is built: one the workflow uses that
no input or built-in names is refused there, not at the first request.
- `comfyui.output` gains `video`, `audio` and `files` beside `pictures`
and `meshes`; `files` takes whatever the workflow saves. `comfy.Image`
becomes `comfy.File` and `typeByName` knows mp4, webm, gif, wav, flac,
mp3, ogg.
- Endpoint: `POST /v1/workflows/runs` with `{model, inputs: {...}}`, a
job like G1's (`GET /v1/workflows/runs[/{id}]`, `.../cancel`). A file
input is base64, a data URL, or a `file-…` id from `/v1/files` for
what is too big to send inline (video, long audio). Files go into the
replica's input directory through the gate's `/upload/image`, which
ComfyUI uses for any file. `/v1/images/generations` and
`/v1/3d/generations` stay as they are, as fixed shapes over the same
machinery.
- The models list (`/v1/models`) gives each ComfyUI model its inputs and
output kind, so a client can build a form without reading the catalog.
- Done when: the existing flux, triposr and sf3d entries work unchanged,
and one model with a picture and a text input (Qwen-Image-Edit) runs
through `/v1/workflows/runs` with a 400 for a missing required input
before any job is written.
### H2. Profiles that fit the hardware (M)
- A profile may say `gpu: {min_compute_capability, min_memory_gb}` in
place of `gpu_class`. When the reconciler places a model, such a
profile stands for every GPU class in the cluster that meets it, at
the profile's priority: the catalog function that turns profiles into
placement's view takes the cluster's classes (class, compute
capability, memory) and returns one concrete profile per class that
fits, named `<profile>@<class>`. Placement itself is unchanged: it
still sees exact classes, and a tensor-parallel group still never
mixes them.
- A ComfyUI profile may carry its own `comfyui` block (workflow, nodes,
checkpoint) over the model's: the fp8 workflow on compute 8.9 and up,
the fp16 one on 7.5. A Turing card is not an error, it is the profile
that fits it.
- Done when: one catalog entry with an "ampere+" profile (min 8.0) and a
"turing" fallback places the fallback on cuda01, and the same entry,
in a test with an A100 in the inventory, places the other.
### H3. Environments: which models can share a replica (S, the catalog side)
- `comfyui.environment: <name>` puts a model in a named environment.
Models in one environment must agree on every custom node they both
name (same repository, same commit) and their requirements must
resolve together; the catalog refuses an environment that cannot, with
both models' names. Models with no environment are each their own, as
today.
- `Catalog.Environments()` returns each environment with its models and
the union of their nodes: what one replica of it must install.
- Placement and eviction of a replica that serves every model of its
environment (so flux, SF3D and Qwen-Image-Edit stop handing the card to
each other with a 30 s cold start each time) is the other session's
work (llmgw-b7), built on this.
### H4. Custom nodes with packages that are built, not downloaded (S)
- `nodes[].build: [subdir, ...]`: directories of the node's repository
that are Python packages with C++/CUDA extensions (SF3D's texture_baker
and uv_unwrapper). The gate builds each as a wheel with the engine's
Python, `TORCH_CUDA_ARCH_LIST` set from the compute capability of the
card it runs on, and installs it into the replica's own `site` like
any other requirement. Built wheels are kept under the engine's home by
node commit, subdirectory and architecture, so the second replica on
the same kind of card does not build again.
- A node needing nvcc on a node without it refuses the replica with that
said, not with a compiler's traceback.
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
venv by hand (the hand install of 2026-10-10 removed first).
### H5. Licence on every output (S)
- `License` gains `id` (SPDX where there is one, else a lowercase name:
stability-community, tencent-hunyuan-community), `commercial` (unset is
not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
`revenue_cap_usd` and `attribution` ("Powered by Stability AI"). All
optional but the name. Done 2026-10-11 in the catalog, so H6 can match
on `id`.
- Written into the file where the format has a place for it: glTF
`asset.extras.license` (as now), PNG an `iTXt` chunk named `license`
(the workflow's own text chunks are still stripped), MP4/WebM and the
audio formats in their metadata where ffmpeg is at hand on the router,
otherwise beside it in the job's answer only — said in the docs.
### H6. Licence choices (S) — the other session's (llmgw-b7), at the user's request
The user asked (2026-10-11) for the choice to be a person's: on their
profile they pick the licences they will not use, and their keys and
portal session neither see those models in `/v1/models` nor may call
them (403, saying which licence). Matched on `license.id`, and on
`commercial` / `excluded_regions` where a choice is put that way. Its
store table is migration 0046. A site-wide or team-wide default can sit
on top of it later.
### H7. A job worker per model, not one for all (S)
The worker runs one job at a time: a job waiting for the card holds up a
job for another model that could run now (seen on the PoC, 2026-10-10).
Jobs are claimed and run concurrently up to a limit per model (or per
environment, after H3) and a cluster-wide cap, with the lease and
leader-only rules of G1 kept.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.