ComfyUI models that agree on their nodes share an environment; a licence says what a machine can match
CI / test (push) Failing after 6m21s
CI / test (push) Failing after 6m21s
A ComfyUI model may name an environment: the models of one environment agree on every custom node they both name and on every package range, so one replica can carry all of them and ComfyUI moves between their weights itself rather than the card being handed over with a cold start each time. The catalog refuses an environment that cannot share a Python, naming both models; Catalog.Environments gives each one's models, the union of their nodes and their weight sources, for placement to build on. A licence gains an id (SPDX where there is one), whether its output may be used commercially, the regions it excludes, a revenue cap and the attribution it asks for, so a person's or a site's choice of licences can be matched without reading prose. BACKLOG gains section H: ComfyUI as a platform. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0639edfd3b
commit
b43e66c95e
7 files changed
+549
No files matched your search
+127
@@ -1270,6 +1270,133 @@ the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
|
||||
jobs list|get|cancel`, the console's Jobs view. The page sends down the
|
||||
same path as the API, like `imagesSend` does.
|
||||
|
||||
## H. ComfyUI as a platform (written 2026-10-11)
|
||||
|
||||
G made one model at a time work: flux for pictures, TripoSR and SF3D for
|
||||
meshes, each with its own endpoint, its own request shape and its own
|
||||
replica. The user's direction (2026-10-11): llmgw is a generic platform,
|
||||
and should serve whatever ComfyUI can run — pictures, edits, video,
|
||||
audio, meshes, upscaling, segmentation — as data in the catalog rather
|
||||
than code per model. Two decisions shape everything below:
|
||||
|
||||
- **The licence is metadata, not a filter.** llmgw does not leave a
|
||||
model out because of its licence. The licence travels with the model
|
||||
and with every file it makes; what a site will not serve is that
|
||||
site's policy (H6). The game studio's EU constraint is one such policy,
|
||||
not a decision the code makes for everyone.
|
||||
- **The install fits the hardware.** The PoC has one Turing card; other
|
||||
sites will have Ampere, Ada and Hopper. A catalog entry must be
|
||||
writable once and placed sensibly anywhere: fp16 on Turing, bf16 or
|
||||
fp8 where the card has them, flash-attention models only where it
|
||||
runs, CUDA extensions built for the card that runs them (H2, H4).
|
||||
|
||||
The first set of models proves the structure at its corners rather than
|
||||
by licence: Qwen-Image-Edit (several weight files, two pictures in),
|
||||
Wan 2.2 small (video out, a long job), ACE-Step (audio out), SF3D (a
|
||||
package not on PyPI, gated weights), Hunyuan3D (a licence that excludes
|
||||
the EU, which H6 must be able to say no to).
|
||||
|
||||
### H1. A workflow says what it takes and makes (M)
|
||||
- Catalog: `comfyui.inputs` — a list of `{name, type, required, default,
|
||||
min, max, choices, max_mb, doc}`, type one of text, int, float, bool,
|
||||
choice, image, mask, audio, video, file. Each is a placeholder of the
|
||||
same name in the workflow. The template's own placeholders are checked
|
||||
against the list when the catalog is built: one the workflow uses that
|
||||
no input or built-in names is refused there, not at the first request.
|
||||
- `comfyui.output` gains `video`, `audio` and `files` beside `pictures`
|
||||
and `meshes`; `files` takes whatever the workflow saves. `comfy.Image`
|
||||
becomes `comfy.File` and `typeByName` knows mp4, webm, gif, wav, flac,
|
||||
mp3, ogg.
|
||||
- Endpoint: `POST /v1/workflows/runs` with `{model, inputs: {...}}`, a
|
||||
job like G1's (`GET /v1/workflows/runs[/{id}]`, `.../cancel`). A file
|
||||
input is base64, a data URL, or a `file-…` id from `/v1/files` for
|
||||
what is too big to send inline (video, long audio). Files go into the
|
||||
replica's input directory through the gate's `/upload/image`, which
|
||||
ComfyUI uses for any file. `/v1/images/generations` and
|
||||
`/v1/3d/generations` stay as they are, as fixed shapes over the same
|
||||
machinery.
|
||||
- The models list (`/v1/models`) gives each ComfyUI model its inputs and
|
||||
output kind, so a client can build a form without reading the catalog.
|
||||
- Done when: the existing flux, triposr and sf3d entries work unchanged,
|
||||
and one model with a picture and a text input (Qwen-Image-Edit) runs
|
||||
through `/v1/workflows/runs` with a 400 for a missing required input
|
||||
before any job is written.
|
||||
|
||||
### H2. Profiles that fit the hardware (M)
|
||||
- A profile may say `gpu: {min_compute_capability, min_memory_gb}` in
|
||||
place of `gpu_class`. When the reconciler places a model, such a
|
||||
profile stands for every GPU class in the cluster that meets it, at
|
||||
the profile's priority: the catalog function that turns profiles into
|
||||
placement's view takes the cluster's classes (class, compute
|
||||
capability, memory) and returns one concrete profile per class that
|
||||
fits, named `<profile>@<class>`. Placement itself is unchanged: it
|
||||
still sees exact classes, and a tensor-parallel group still never
|
||||
mixes them.
|
||||
- A ComfyUI profile may carry its own `comfyui` block (workflow, nodes,
|
||||
checkpoint) over the model's: the fp8 workflow on compute 8.9 and up,
|
||||
the fp16 one on 7.5. A Turing card is not an error, it is the profile
|
||||
that fits it.
|
||||
- Done when: one catalog entry with an "ampere+" profile (min 8.0) and a
|
||||
"turing" fallback places the fallback on cuda01, and the same entry,
|
||||
in a test with an A100 in the inventory, places the other.
|
||||
|
||||
### H3. Environments: which models can share a replica (S, the catalog side)
|
||||
- `comfyui.environment: <name>` puts a model in a named environment.
|
||||
Models in one environment must agree on every custom node they both
|
||||
name (same repository, same commit) and their requirements must
|
||||
resolve together; the catalog refuses an environment that cannot, with
|
||||
both models' names. Models with no environment are each their own, as
|
||||
today.
|
||||
- `Catalog.Environments()` returns each environment with its models and
|
||||
the union of their nodes: what one replica of it must install.
|
||||
- Placement and eviction of a replica that serves every model of its
|
||||
environment (so flux, SF3D and Qwen-Image-Edit stop handing the card to
|
||||
each other with a 30 s cold start each time) is the other session's
|
||||
work (llmgw-b7), built on this.
|
||||
|
||||
### H4. Custom nodes with packages that are built, not downloaded (S)
|
||||
- `nodes[].build: [subdir, ...]`: directories of the node's repository
|
||||
that are Python packages with C++/CUDA extensions (SF3D's texture_baker
|
||||
and uv_unwrapper). The gate builds each as a wheel with the engine's
|
||||
Python, `TORCH_CUDA_ARCH_LIST` set from the compute capability of the
|
||||
card it runs on, and installs it into the replica's own `site` like
|
||||
any other requirement. Built wheels are kept under the engine's home by
|
||||
node commit, subdirectory and architecture, so the second replica on
|
||||
the same kind of card does not build again.
|
||||
- A node needing nvcc on a node without it refuses the replica with that
|
||||
said, not with a compiler's traceback.
|
||||
- Done when: SF3D runs on cuda01 with nothing installed into the engine's
|
||||
venv by hand (the hand install of 2026-10-10 removed first).
|
||||
|
||||
### H5. Licence on every output (S)
|
||||
- `License` gains `id` (SPDX where there is one, else a lowercase name:
|
||||
stability-community, tencent-hunyuan-community), `commercial` (unset is
|
||||
not said), `excluded_regions` (ISO 3166 codes, `EU` for the union),
|
||||
`revenue_cap_usd` and `attribution` ("Powered by Stability AI"). All
|
||||
optional but the name. Done 2026-10-11 in the catalog, so H6 can match
|
||||
on `id`.
|
||||
- Written into the file where the format has a place for it: glTF
|
||||
`asset.extras.license` (as now), PNG an `iTXt` chunk named `license`
|
||||
(the workflow's own text chunks are still stripped), MP4/WebM and the
|
||||
audio formats in their metadata where ffmpeg is at hand on the router,
|
||||
otherwise beside it in the job's answer only — said in the docs.
|
||||
|
||||
### H6. Licence choices (S) — the other session's (llmgw-b7), at the user's request
|
||||
The user asked (2026-10-11) for the choice to be a person's: on their
|
||||
profile they pick the licences they will not use, and their keys and
|
||||
portal session neither see those models in `/v1/models` nor may call
|
||||
them (403, saying which licence). Matched on `license.id`, and on
|
||||
`commercial` / `excluded_regions` where a choice is put that way. Its
|
||||
store table is migration 0046. A site-wide or team-wide default can sit
|
||||
on top of it later.
|
||||
|
||||
### H7. A job worker per model, not one for all (S)
|
||||
The worker runs one job at a time: a job waiting for the card holds up a
|
||||
job for another model that could run now (seen on the PoC, 2026-10-10).
|
||||
Jobs are claimed and run concurrently up to a limit per model (or per
|
||||
environment, after H3) and a cluster-wide cap, with the lease and
|
||||
leader-only rules of G1 kept.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
Reference in new issue
Block a user