diff --git a/BACKLOG.md b/BACKLOG.md index b3860089..def30ef4 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1101,6 +1101,93 @@ at a third of its CPUs. 28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the chat model, and a reranker. +## G. Jobs: 3D generation through ComfyUI (2026-10-10) + +A request for a 3D model (Hunyuan3D 2 first; TRELLIS later if its quality +is wanted) takes minutes on the one 24 GB card, and the card has to be +emptied of the chat model first. That is not a request to hold a socket for +— `internal/router/activate.go:46` already says so — it is a job: queued, +answered with an id, asked after, its result downloaded. The job machinery +is built first and tested without a GPU; the ComfyUI generalisation and the +custom-node install come after. Decided 2026-10-10: a `jobs` table of its +own rather than a one-line batch (the JSONL shape would be artificial and +has nowhere for progress), and progress is shown, not just a status. + +### G1. Jobs: store, worker, endpoints (M) +- Store: `jobs` (id, owner key/principal, kind, model, status, input + (JSON), progress (JSON: queue position, percent, step), result file id, + error, created/started/finished, lease owner/until), migrations for + sqlite and postgres beside `internal/store/batches.go`, whose + claim/renew/release lease pattern is copied, not shared: a job is one + unit of work, a batch is many lines. +- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any + gateway on the store claims a queued job, asks for the model with + `activator.Want` and waits as long as it takes (no `ActivationWait` + bound: nobody is holding a connection), runs it, stores the output + through the files store, writes usage and price, marks it done. A + gateway that dies mid-job loses its lease and another one picks it up; + the ComfyUI prompt id is in the job row so the second gateway asks + after the same work rather than queueing it again. +- Endpoints: `POST /v1/3d/generations` validates like any inference + request (key, allowlist, limits, admission, guardrails on the prompt), + writes the job, answers 202 with `{id, status: "queued"}`; + `GET /v1/3d/generations/{id}` returns status, progress and, when done, + `output_file_id` and the usage; `POST .../{id}/cancel`; the file comes + from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results + are kept for the batch files' retention and deleted with them: a + departure from `images.go`'s "kept nowhere", because a mesh is a + download, not a JSON field; the retention is the bound. +- Done when: a job survives the gateway that took it being killed, a + cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two + gateways never both run one job (test with the lease), and the usage + row carries the job's price. + +### G2. Progress from ComfyUI (S, after G1) +The gate (`internal/comfy/gate.go`) passes seven paths and no websocket. +Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once, +with `golang.org/x/net/websocket` (already a dependency), under a client +id of its own, and rewrites the `client_id` of every `/prompt` it +forwards to that one. It keeps the last `executing`, `progress` +(value/max/node) and `execution_error` message per prompt id and serves +them at `GET /progress/{prompt_id}` on the gate; the router polls that +beside `/history` and `/queue` and writes queue position, percent and the +current node's class into the job's progress. The gate's buffer is +bounded (the last N prompts) and the websocket reconnects. Done when a +3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in +`ctl jobs` and on the account page, and a job whose replica vanished +says so within one poll. + +### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) +- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an + output kind; `ServesComfyUI` stays). 3D weight folders in the gate's + `modelFolders`; the workflow's placeholders gain `image` (the input + picture, base64 in the request, written to the replica's `input/` by + the gate through a new `POST /upload/image` the gate allows — or a + base64-loading node, whichever Hunyuan3D's wrapper wants). +- `comfy.Client.fetch` reads every output key, not `images` only, and + fetches each file named by `/view`; `Image` becomes `Artifact` with its + content type (model/gltf-binary, application/octet-stream for .obj). +- Custom nodes: a model (or the engine config) names git repositories + and a pip requirements list; the agent's provision step clones them + into the replica's `custom_nodes/` (the gate's `--base-directory` is + where ComfyUI looks, not the install directory) and installs the + requirements into the engine's venv with the same PyTorch index and + lock discipline as A8. A replica with a missing node refuses to start + with the node's name in the error rather than ComfyUI's traceback. +- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`, + `Outcome.Meshes`, the usage sums and exports (as eaea671 did for + pictures). +- Done when: Hunyuan3D 2 on cuda01 with `scaling: {min: 0, + idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint, + the card is free again after the idle timeout, and the same request + against flux-schnell is refused as "makes pictures, not meshes". + +### G4. Portal and console (S, after G2) +A 3D page beside `/images` (upload a picture, watch progress, download +the .glb, a viewer with `` from the allowed CDNs), `ctl +jobs list|get|cancel`, the console's Jobs view. The page sends down the +same path as the API, like `imagesSend` does. + ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.