diff --git a/BACKLOG.md b/BACKLOG.md index 9b135b8..59bfd27 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1168,7 +1168,12 @@ files endpoint still serves every file as an octet stream. gateways never both run one job (test with the lease), and the usage row carries the job's price. -### G2. Progress from ComfyUI (S, after G1) +### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`) +Built as written: the gate's websocket under its own client id, the +client_id of every /prompt rewritten, /progress/{prompt_id} on the gate, +the router reading it and /queue into the job (only into a job of the same +key). Left: a node that reports no steps (TripoSR's) shows its name and no +percent; a vanished replica is said by the job failing, not by progress. The gate (`internal/comfy/gate.go`) passes seven paths and no websocket. Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once, with `golang.org/x/net/websocket` (already a dependency), under a client @@ -1183,7 +1188,19 @@ bounded (the last N prompts) and the websocket reconnects. Done when a `ctl jobs` and on the account page, and a job whose replica vanished says so within one poll. -### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) +### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR +Built: output kinds, image upload through the gate, every output list +fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh +and usage.meshes (migration 0045), custom nodes cloned at their commit by +the gate with packages beside the replica (a node's named packages at its +range, the rest at ComfyUI's versions). Not as written: the model. On the +PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3, +its published background remover (RMBG-2.0) is non-commercial, and its +sparse attention and Direct3D-S2's need flash-attention (Ampere and +newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture +in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT, +as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated +encoder, a missing node refusing the replica by name. - Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an output kind; `ServesComfyUI` stays). 3D weight folders in the gate's `modelFolders`; the workflow's placeholders gain `image` (the input @@ -1226,7 +1243,10 @@ says so within one poll. the card is free again after the idle timeout, and the same request against flux-schnell is refused as "makes pictures, not meshes". -### G4. Portal and console (S, after G2) +### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`) +Built: /3d with model-viewer served from the gateway, progress, download +and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs +was taken by the control plane's own work). Left: the console's Jobs view. A 3D page beside `/images` (upload a picture, watch progress, download the .glb, a viewer with `` from the allowed CDNs), `ctl jobs list|get|cancel`, the console's Jobs view. The page sends down the diff --git a/CHANGELOG.md b/CHANGELOG.md index 7e89fbe..0e50dca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands. | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released | | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released | | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released | -| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) | +| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | Until v1.0.0 the minor version is the breaking one: configuration formats and Go @@ -26,6 +26,23 @@ APIs change between minors without a deprecation period. ## Unreleased +## v0.14.3 — 2026-10-10 + +3D models from a picture, as jobs: made by ComfyUI on the cluster's cards, +with their progress, their licence written into the mesh, a price per mesh +and a page to make them. Also: a profile's own weights, a band for bulk +work, and fixes for control planes behind a balancer and agents that +restart. + +**Upgrading.** +- The store gains tables and columns (migrations 0043–0045: notify_sent, + jobs, usage meshes); nothing to do. +- **Nodes** need this release for their engines' output to survive an + agent restart and for ComfyUI's custom nodes and progress; engines + started by an older agent keep its pipe until they are restarted. +- A 3D model needs a model that makes them (see install.md, "A model that + makes them"); none is added. + ### Added - **Jobs: 3D models asked after, not waited for.** `POST @@ -38,9 +55,25 @@ APIs change between minors without a deprecation period. it is charged like any request, tagged `job=`, and a model still loading is waited out for as long as the job has (24 h) rather than refused. A job survives the gateway that took it; results are kept for - `router.batch.retain_days`. Migration 0044. No model makes 3D models - yet: every request is refused at the door until the ComfyUI adapter - learns meshes (BACKLOG G3). + `router.batch.retain_days`. Migration 0044. +- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The + picture is uploaded to the replica's ComfyUI and the workflow's + `{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned + into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards + stronger ones can be added the same way. +- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in + each replica's own `custom_nodes` by the gate, with its packages + installed beside the replica, never into ComfyUI's environment. +- **A job's progress**: the gate listens to ComfyUI's websocket and the job + says queued and how many ahead, then the node running, its step and + percent. +- **A model's `license`** (name, url, notice), in `/v1/models`, in the job, + on the portal and in the GLB's `asset.extras`. +- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count + `meshes`. +- **`/3d` on the portal**, beside `/images`: a picture chosen, the job + followed, the mesh in a viewer with a download and its licence; and + **`llmgw ctl 3d list|get|cancel`** for every owner's jobs. - **A profile's own `source`**: the weights it loads instead of the model's, so an embedding model can run on vLLM from its repository on a card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers diff --git a/README.md b/README.md index ef4302e..da4c3d9 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released | | v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released | | v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released | -| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) | +| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | As of v0.6.0 a cluster runs unattended: agents report their hardware, the diff --git a/docs/install.md b/docs/install.md index cd8251a..b2a8c82 100644 --- a/docs/install.md +++ b/docs/install.md @@ -2092,8 +2092,80 @@ curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorizat files; the job's record goes with it. A project under a `no_store` policy is refused jobs, as it is batches. -No model makes 3D models yet: until the ComfyUI adapter learns meshes, -every request is refused at the door as "does not make 3D models". +A request may also send `seed` and `resolution` (the model's own grid, a +multiple of 32 from 64 to 2048). The picture should show one object on a +plain background: the model makes a mesh of what it sees, background +included. + +### A model that makes them + +A 3D model is made by a ComfyUI model whose `comfyui.output` is `meshes` +(see [Pictures from ComfyUI](#pictures-from-comfyui) for engine `comfyui`). +Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the +uploaded picture, which a `LoadImage` node reads; `{{seed}}`, +`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that +saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB, +vertex colours kept. TripoSR (MIT) on any card of 8 GB: + +```yaml +- name: triposr + source: hf:stabilityai/TripoSR/model.ckpt + comfyui: + workflow: /etc/llmgw/workflows/triposr.json # on every control plane + output: meshes + nodes: + - repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR + commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3 + requirements: [omegaconf, trimesh, scikit-image, imageio, + "transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers] + license: + name: MIT + url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE + notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..." + scaling: {min: 0, max: 1, idle_timeout: 10m} + profiles: + - {name: gpu, engine: comfyui, gpu_class: 6000-24g, tp: 1} +``` + +```json +{ + "1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}}, + "2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}}, + "3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["1", 0], + "geometry_resolution": "{{resolution|256}}", "threshold": 25.0}}, + "4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}} +} +``` + +- **Custom nodes** are named by https repository and full commit (a branch + is refused: it moves). The gate puts each in the replica's own + `custom_nodes` and installs its `requirements` beside the replica, never + into ComfyUI's environment. A package named there is installed at the + range named; every other one at the version ComfyUI already runs. A + package named is first on ComfyUI's path, so its range must suit ComfyUI + as well. TripoSR's checkpoint loads with transformers 4, which ComfyUI's + `>=4.50.3` allows. +- **Progress:** the gate listens to ComfyUI's websocket and the router + writes what it hears into the job: `{"state": "queued", "ahead": 2}`, + then `{"state": "running", "node_class": "KSampler", "step": 12, + "steps": 30, "percent": 40}` for a node that counts its steps, and + `done` at the end. +- **Licence:** a model's `license` is shown in `/v1/models`, in the job and + on the portal, and written into the GLB's `asset.extras.license`, since + the file travels without the API. Choose the model for its licence before + its quality when what it makes is shipped. TRELLIS.2 needs Meta's gated + DINOv3 and, as published, a background remover under a non-commercial + licence; Hunyuan3D's licence excludes the EU, the UK and South Korea. +- **Price:** `per_mesh` in `auth.pricing`, beside `per_image`; each usage + row says how many meshes it made (`meshes`). +- **Cards:** the strongest models today (TRELLIS.2, Direct3D-S2) need + flash-attention, which runs on Ampere and newer. A Turing card such as + the RTX 6000 runs TripoSR. + +People make them at `/3d` on the portal: they choose a picture and watch +the job, and the mesh is shown in a viewer (model-viewer, served from the +gateway) with a download link and the licence. `llmgw ctl 3d list`, `get` +and `cancel` show and stop every owner's jobs. ## Conversations across models diff --git a/internal/router/jobworker.go b/internal/router/jobworker.go index c454e0f..898734e 100644 --- a/internal/router/jobworker.go +++ b/internal/router/jobworker.go @@ -202,6 +202,10 @@ func (r *Router) serveJob(ctx context.Context, j *store.Job) error { if err := r.store.PutFile(ctx, f, w.buf.Bytes()); err != nil { return err } + // Said done while it may still be said: the last word heard was the + // node running, and a finished job saying it is running reads as + // one that stopped halfway. + _ = r.store.SetJobProgress(ctx, j.ID, `{"state":"done","percent":100}`, "") j.Status, j.OutputFileID, j.ContentType, j.FinishedAt = store.JobCompleted, f.ID, ct, done return r.store.UpdateJob(ctx, *j) } diff --git a/internal/router/meshes_test.go b/internal/router/meshes_test.go index 3e2e904..a5673b7 100644 --- a/internal/router/meshes_test.go +++ b/internal/router/meshes_test.go @@ -161,6 +161,9 @@ func TestA3DModelIsMadeFromAPictureWithItsLicenceInside(t *testing.T) { if got["status"] != "completed" || got["content_type"] != "model/gltf-binary" { t.Fatalf("job %v", got) } + if p, _ := got["progress"].(map[string]any); p["state"] != "done" { + t.Errorf("a finished job says %v", got["progress"]) + } if lic, _ := got["license"].(map[string]any); lic["name"] != "MIT" { t.Errorf("the job does not say the licence: %v", got["license"]) }