Compare commits
38
Commits
adapters-f4
...
v0.14.3
No files matched your search
+193
-6
@@ -386,10 +386,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
|
||||
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
|
||||
verified email, roles and memberships from groups, `no_passwords`) and
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
|
||||
signed with a per-process key, so with two control planes behind a
|
||||
balancer the callback must reach the one that started it (sticky
|
||||
sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
|
||||
from the store (2026-10-10), so a callback may reach either control
|
||||
plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
`internal/web/account.go`; map IdP groups to org/team/admin.
|
||||
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
|
||||
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
|
||||
@@ -405,8 +404,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand; each control plane sends
|
||||
budget events on its own, so two behind a balancer may both send one.
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
|
||||
templates driven by `server init` or the console; llmgw-native webhooks
|
||||
@@ -1065,6 +1065,193 @@ training yet; quality gating of a published adapter is by hand (ctl eval).
|
||||
|
||||
---
|
||||
|
||||
### F5. Large imports (M) — measured 2026-10-09, open
|
||||
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
|
||||
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
|
||||
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
|
||||
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
- Done: a batch of embeddings checked text by text against the context
|
||||
(branch `embeddings-batch`); it was refused as one over-long prompt.
|
||||
- Done: `query_prefix` on an embedding model, before questions only.
|
||||
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
|
||||
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
|
||||
profile, so an embedding model's GPU profile runs vLLM and its CPU
|
||||
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
|
||||
`Retry-After` on a busy refusal, which is what had turned passages away
|
||||
(llmgw-kb retries only what says when to).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||||
chat model, and a reranker.
|
||||
|
||||
## G. Jobs: 3D generation through ComfyUI (2026-10-10)
|
||||
|
||||
A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
|
||||
emptied of the chat model first. That is not a request to hold a socket for
|
||||
— `internal/router/activate.go:46` already says so — it is a job: queued,
|
||||
answered with an id, asked after, its result downloaded. The job machinery
|
||||
is built first and tested without a GPU; the ComfyUI generalisation and the
|
||||
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
|
||||
own rather than a one-line batch (the JSONL shape would be artificial and
|
||||
has nowhere for progress), and progress is shown, not just a status.
|
||||
|
||||
Which models (decided 2026-10-10): the assets go into a game sold on
|
||||
Steam, worldwide, the EU included, so the licence of the output decides
|
||||
before the quality does. Hunyuan3D (every version) is out: Tencent's
|
||||
licence excludes the EU, UK and South Korea, and whether a game shipped
|
||||
there with its outputs is "use in the Territory" is not a question to
|
||||
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
|
||||
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
|
||||
both under NVIDIA's non-commercial source licence, so that stage is not
|
||||
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
|
||||
EU-safe textured path — its ComfyUI node and VRAM need are to be
|
||||
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
|
||||
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
|
||||
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
|
||||
community licence, revenue-capped), SAM 3D Objects (Meta's own licence,
|
||||
to be read before it is considered). Steam's disclosure form (changed
|
||||
2026-01-16) lists pre-generated 3D models as content that must be
|
||||
declared on the store page: the licence notice G3 writes into the GLB is
|
||||
what the studio answers that form from.
|
||||
|
||||
### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
|
||||
Built as written, with two choices: a job found running after its gateway
|
||||
died is run again from the start (the prompt id column is there for G2 to
|
||||
follow the same work instead), and the request is validated at the door
|
||||
only as far as costs nothing — key, model, allowlist, the request's shape
|
||||
— with limits, guardrails and admission applied when the worker replays it,
|
||||
said as the job's error. The result file's `content_type` is the job's; the
|
||||
files endpoint still serves every file as an octet stream.
|
||||
- Store: `jobs` (id, owner key/principal, kind, model, status, input
|
||||
(JSON), progress (JSON: queue position, percent, step), result file id,
|
||||
error, created/started/finished, lease owner/until), migrations for
|
||||
sqlite and postgres beside `internal/store/batches.go`, whose
|
||||
claim/renew/release lease pattern is copied, not shared: a job is one
|
||||
unit of work, a batch is many lines.
|
||||
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
|
||||
gateway on the store claims a queued job, asks for the model with
|
||||
`activator.Want` and waits as long as it takes (no `ActivationWait`
|
||||
bound: nobody is holding a connection), runs it, stores the output
|
||||
through the files store, writes usage and price, marks it done. A
|
||||
gateway that dies mid-job loses its lease and another one picks it up;
|
||||
the ComfyUI prompt id is in the job row so the second gateway asks
|
||||
after the same work rather than queueing it again.
|
||||
- Endpoints: `POST /v1/3d/generations` validates like any inference
|
||||
request (key, allowlist, limits, admission, guardrails on the prompt),
|
||||
writes the job, answers 202 with `{id, status: "queued"}`;
|
||||
`GET /v1/3d/generations/{id}` returns status, progress and, when done,
|
||||
`output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
|
||||
from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
|
||||
are kept for the batch files' retention and deleted with them: a
|
||||
departure from `images.go`'s "kept nowhere", because a mesh is a
|
||||
download, not a JSON field; the retention is the bound.
|
||||
- Done when: a job survives the gateway that took it being killed, a
|
||||
cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
|
||||
gateways never both run one job (test with the lease), and the usage
|
||||
row carries the job's price.
|
||||
|
||||
### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built as written: the gate's websocket under its own client id, the
|
||||
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
|
||||
the router reading it and /queue into the job (only into a job of the same
|
||||
key). Left: a node that reports no steps (TripoSR's) shows its name and no
|
||||
percent; a vanished replica is said by the job failing, not by progress.
|
||||
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
|
||||
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
|
||||
with `golang.org/x/net/websocket` (already a dependency), under a client
|
||||
id of its own, and rewrites the `client_id` of every `/prompt` it
|
||||
forwards to that one. It keeps the last `executing`, `progress`
|
||||
(value/max/node) and `execution_error` message per prompt id and serves
|
||||
them at `GET /progress/{prompt_id}` on the gate; the router polls that
|
||||
beside `/history` and `/queue` and writes queue position, percent and the
|
||||
current node's class into the job's progress. The gate's buffer is
|
||||
bounded (the last N prompts) and the websocket reconnects. Done when a
|
||||
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
|
||||
`ctl jobs` and on the account page, and a job whose replica vanished
|
||||
says so within one poll.
|
||||
|
||||
### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
|
||||
Built: output kinds, image upload through the gate, every output list
|
||||
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
|
||||
and usage.meshes (migration 0045), custom nodes cloned at their commit by
|
||||
the gate with packages beside the replica (a node's named packages at its
|
||||
range, the rest at ComfyUI's versions). Not as written: the model. On the
|
||||
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
|
||||
its published background remover (RMBG-2.0) is non-commercial, and its
|
||||
sparse attention and Direct3D-S2's need flash-attention (Ampere and
|
||||
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
|
||||
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
|
||||
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
|
||||
encoder, a missing node refusing the replica by name.
|
||||
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
|
||||
output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
|
||||
`modelFolders`; the workflow's placeholders gain `image` (the input
|
||||
picture, base64 in the request, written to the replica's `input/` by
|
||||
the gate through a new `POST /upload/image` the gate allows — or a
|
||||
base64-loading node, whichever TRELLIS.2's wrapper wants).
|
||||
- `comfy.Client.fetch` reads every output key, not `images` only, and
|
||||
fetches each file named by `/view`; `Image` becomes `Artifact` with its
|
||||
content type (model/gltf-binary, application/octet-stream for .obj).
|
||||
- Custom nodes: a model (or the engine config) names git repositories
|
||||
and a pip requirements list; the agent's provision step clones them
|
||||
into the replica's `custom_nodes/` (the gate's `--base-directory` is
|
||||
where ComfyUI looks, not the install directory) and installs the
|
||||
requirements into the engine's venv with the same PyTorch index and
|
||||
lock discipline as A8. A replica with a missing node refuses to start
|
||||
with the node's name in the error rather than ComfyUI's traceback.
|
||||
- Licence of the output (decided 2026-10-10: it matters). A catalog model
|
||||
carries `license: {name, url, notice}`; a job's answer, `/v1/models`
|
||||
and the portal page show it beside the result, and the mesh's GLB
|
||||
`asset.extras` carries the notice too, since the file travels without
|
||||
the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
|
||||
its outputs free, but its texture baking depends on nvdiffrast, whose
|
||||
NVIDIA Source Code License is non-commercial — a commercial service
|
||||
must render textures another way or skip that stage. Hunyuan3D 2.0 and
|
||||
2.1 claim no rights in outputs, but the licence excludes the EU, UK and
|
||||
South Korea, needs Tencent's permission above 1 M monthly active users,
|
||||
forbids using outputs to train other models, and asks for "Powered by
|
||||
Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
|
||||
MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
|
||||
some of whose objects are CC-BY-NC: a provenance question no licence
|
||||
text settles, to be said in the notice rather than hidden.
|
||||
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
|
||||
`Outcome.Meshes`, the usage sums and exports (as eaea671 did for
|
||||
pictures).
|
||||
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
|
||||
idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
|
||||
with nvdiffrast and nvdiffrec not installed in its venv (the licence
|
||||
is kept by the install, not by a note), the GLB carries the MIT
|
||||
notice,
|
||||
the card is free again after the idle timeout, and the same request
|
||||
against flux-schnell is refused as "makes pictures, not meshes".
|
||||
|
||||
### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built: /3d with model-viewer served from the gateway, progress, download
|
||||
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
|
||||
was taken by the control plane's own work). Left: the console's Jobs view.
|
||||
A 3D page beside `/images` (upload a picture, watch progress, download
|
||||
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
|
||||
jobs list|get|cancel`, the console's Jobs view. The page sends down the
|
||||
same path as the API, like `imagesSend` does.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+152
-1
@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
@@ -26,6 +26,157 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## v0.14.3 — 2026-10-10
|
||||
|
||||
3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
|
||||
with their progress, their licence written into the mesh, a price per mesh
|
||||
and a page to make them. Also: a profile's own weights, a band for bulk
|
||||
work, and fixes for control planes behind a balancer and agents that
|
||||
restart.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains tables and columns (migrations 0043–0045: notify_sent,
|
||||
jobs, usage meshes); nothing to do.
|
||||
- **Nodes** need this release for their engines' output to survive an
|
||||
agent restart and for ComfyUI's custom nodes and progress; engines
|
||||
started by an older agent keep its pipe until they are restarted.
|
||||
- A 3D model needs a model that makes them (see install.md, "A model that
|
||||
makes them"); none is added.
|
||||
|
||||
### Added
|
||||
|
||||
- **Jobs: 3D models asked after, not waited for.** `POST
|
||||
/v1/3d/generations` answers 202 with a job id; `GET
|
||||
/v1/3d/generations/{id}` says its status, progress and, when done, the
|
||||
file its result is in (`/v1/files/{id}/content`); `POST .../cancel` stops
|
||||
it, in the engine's queue too. The request is replayed by whichever
|
||||
gateway holds the job's lease through the same path a client's request
|
||||
takes — key, allowlist, limits, admission, guardrails, accounting — so
|
||||
it is charged like any request, tagged `job=<id>`, and a model still
|
||||
loading is waited out for as long as the job has (24 h) rather than
|
||||
refused. A job survives the gateway that took it; results are kept for
|
||||
`router.batch.retain_days`. Migration 0044.
|
||||
- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The
|
||||
picture is uploaded to the replica's ComfyUI and the workflow's
|
||||
`{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned
|
||||
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
|
||||
stronger ones can be added the same way.
|
||||
- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in
|
||||
each replica's own `custom_nodes` by the gate, with its packages
|
||||
installed beside the replica, never into ComfyUI's environment.
|
||||
- **A job's progress**: the gate listens to ComfyUI's websocket and the job
|
||||
says queued and how many ahead, then the node running, its step and
|
||||
percent.
|
||||
- **A model's `license`** (name, url, notice), in `/v1/models`, in the job,
|
||||
on the portal and in the GLB's `asset.extras`.
|
||||
- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count
|
||||
`meshes`.
|
||||
- **`/3d` on the portal**, beside `/images`: a picture chosen, the job
|
||||
followed, the mesh in a viewer with a download and its licence; and
|
||||
**`llmgw ctl 3d list|get|cancel`** for every owner's jobs.
|
||||
- **A profile's own `source`**: the weights it loads instead of the
|
||||
model's, so an embedding model can run on vLLM from its repository on a
|
||||
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
|
||||
nodes holding each profile's weights, `ctl model add` checks every
|
||||
source, and `ctl weights` collects none a profile loads.
|
||||
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
|
||||
long its requests may wait for room, instead of the admission's 30 s. A
|
||||
band for bulk work — a knowledge-base import — waits behind people rather
|
||||
than being turned away.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An engine's output no longer breaks when its agent restarts.** Engines
|
||||
wrote to a pipe of the agent's, which an agent that restarted and adopted
|
||||
them left with no reader; an engine's next large write failed, and in a
|
||||
Python engine took the request it was reporting on with it. Each process
|
||||
now writes to its own file in the agent's work directory, and an adopted
|
||||
engine's log keeps what it wrote before the restart.
|
||||
- **Single sign-on behind a balancer.** The cookie that carries a sign-in
|
||||
across the identity provider was signed with a key of each process's own,
|
||||
so with two control planes a callback that reached the other one was
|
||||
refused. The key is now a secret in the store, the same on every control
|
||||
plane.
|
||||
- **Notifications sent once by two control planes.** Each kept its own
|
||||
hour of quiet per event, so a budget spent on a team both served was
|
||||
announced twice. The control plane that claims an event in the store
|
||||
(migration 0043) sends it; with the store unreachable it is sent anyway.
|
||||
- **A request turned away for a busy cluster says when to come back.** A
|
||||
request that waited as long as it may (504) or found the queue full (503)
|
||||
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
|
||||
of sending it again; both now say 10 seconds.
|
||||
- The usage sums and exports count pictures (`images`), as the request
|
||||
records did; pictures made before are added to the sums.
|
||||
|
||||
## v0.14.2 — 2026-10-10
|
||||
|
||||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||||
cluster's own cards. The agents install ComfyUI themselves, the gateway
|
||||
prices pictures by the picture, and the portal has a page to make them.
|
||||
Also: on-demand models that survive a failover, and knowledge-base
|
||||
embeddings fixes.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains a column (`usage.images`); nothing to do.
|
||||
- **ComfyUI is installed only where asked**: `llmgw ctl config set
|
||||
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
|
||||
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
|
||||
or run engine `comfyui`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
|
||||
key, allowlist, limit, admission and guardrail checks as any request. A
|
||||
model with profiles on engine `comfyui` is placed on a GPU node like any
|
||||
other. A ComfyUI server outside the cluster can be an external model with
|
||||
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
|
||||
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
|
||||
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
|
||||
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
|
||||
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
|
||||
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
|
||||
that leaves takes its workflow off the queue. `/v1/models` says `images`
|
||||
for such a model.
|
||||
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
|
||||
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
|
||||
gate listens on the replica's port, demands the replica's key, and lets
|
||||
through only what the router asks. The model's `comfyui:` block names
|
||||
the workflow and, for a source that is a set of files, the checkpoint.
|
||||
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
|
||||
checked against a digest llmgw holds, since GitHub publishes none, and a
|
||||
version without one is refused. Its requirements are resolved as PyPI
|
||||
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
|
||||
CUDA 12.9 one for an older driver, and the result is checked on the card.
|
||||
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
|
||||
agent.yaml.
|
||||
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
|
||||
prompt's tokens. A model priced only per picture is not free, so a spent
|
||||
budget still stops it. Each usage row keeps how many pictures were made,
|
||||
shown by `llmgw ctl request`.
|
||||
- **`/images` on the portal**, beside the chat, for the models that make
|
||||
pictures. The chat no longer offers them. The page offers size, how many,
|
||||
seed, steps and a negative prompt, and shows each picture with its seed
|
||||
and a download link. Nothing is kept but the usage.
|
||||
- **An embedding model's `query_prefix`**, put before the question a
|
||||
knowledge-base search embeds and never before documents, for Qwen3-Embedding
|
||||
and models trained like it.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An awake on-demand model survived no restart or change of leader of the
|
||||
control plane.** A control plane that had heard no requests read every
|
||||
on-demand replica its nodes reported as wanted by nobody, and drained them
|
||||
all. An adopted replica now gets a whole idle timeout.
|
||||
- `ctl models` said `not placed` for an on-demand model that had been woken
|
||||
and was serving; it now says `ready (on demand)`.
|
||||
- A batch of embeddings was refused as one over-long prompt when the
|
||||
passages together exceeded the context, though each one fit. The longest
|
||||
input is what has to fit.
|
||||
- The chat's kept conversation sat in the browser under one key for
|
||||
everybody, so the next person to sign in on the same browser opened the
|
||||
last one's conversation. It is kept per account, and signing out clears
|
||||
what the pages stored.
|
||||
|
||||
## v0.14.1 — 2026-10-09
|
||||
|
||||
Adapters trained on the nodes from a knowledge base and kept private to
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
|
||||
@@ -29,6 +29,9 @@
|
||||
{
|
||||
"name": "capacity"
|
||||
},
|
||||
{
|
||||
"name": "client-jobs"
|
||||
},
|
||||
{
|
||||
"name": "eval"
|
||||
},
|
||||
@@ -280,6 +283,133 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs": {
|
||||
"get": {
|
||||
"operationId": "getClientJobs",
|
||||
"summary": "Jobs clients asked for — 3D models — of every owner, newest first",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "status",
|
||||
"in": "query",
|
||||
"description": "Only jobs in this status: queued, running, cancelling, completed, failed, cancelled, expired.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"in": "query",
|
||||
"description": "The most jobs to answer with; 50 when absent.",
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs/{id}": {
|
||||
"get": {
|
||||
"operationId": "getClientJobsId",
|
||||
"summary": "One job a client asked for",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The job's ID, job_….",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs/{id}/cancel": {
|
||||
"post": {
|
||||
"operationId": "postClientJobsIdCancel",
|
||||
"summary": "Stop a job, in the engine's queue too",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The job's ID, job_….",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/eval/runs": {
|
||||
"get": {
|
||||
"operationId": "getEvalRuns",
|
||||
@@ -6016,6 +6146,52 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"JobView": {
|
||||
"type": "object",
|
||||
"description": "JobView is one job.",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"owner": {
|
||||
"type": "string"
|
||||
},
|
||||
"key_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"endpoint": {
|
||||
"type": "string"
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"progress": {
|
||||
"description": "Any JSON value."
|
||||
},
|
||||
"output_file_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"content_type": {
|
||||
"type": "string"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"started_at": {
|
||||
"type": "string"
|
||||
},
|
||||
"finished_at": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"JudgeAskRequest": {
|
||||
"type": "object",
|
||||
"description": "JudgeAskRequest is a text to have a judge read.",
|
||||
@@ -7595,6 +7771,14 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"images": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"meshes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"cost_micros": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
@@ -8565,6 +8749,60 @@
|
||||
},
|
||||
"embeddings": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"images": {
|
||||
"type": "boolean",
|
||||
"description": "Images is a model that makes pictures, asked for at /v1/images/generations. Implied by an external model whose API is ComfyUI's."
|
||||
},
|
||||
"meshes": {
|
||||
"type": "boolean",
|
||||
"description": "Meshes is a model that makes 3D models from a picture, asked for as a job at /v1/3d/generations. Implied by a ComfyUI model whose output is meshes."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.ComfyUI": {
|
||||
"type": "object",
|
||||
"description": "ComfyUI is a ComfyUI model's workflow and checkpoint.",
|
||||
"properties": {
|
||||
"workflow": {
|
||||
"type": "string",
|
||||
"description": "Workflow is a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane, in ComfyUI's API format."
|
||||
},
|
||||
"checkpoint": {
|
||||
"type": "string",
|
||||
"description": "Checkpoint is the file the workflow's {{checkpoint}} names: the source's own name unless set, which a source that is a set of files must."
|
||||
},
|
||||
"output": {
|
||||
"type": "string",
|
||||
"description": "Output is what the workflow makes: pictures (the default), asked for at /v1/images/generations, or meshes, asked for as a job at /v1/3d/generations from a picture."
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.CustomNode"
|
||||
},
|
||||
"description": "Nodes are custom nodes the workflow needs, put in each replica's own custom_nodes at the commit named."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.CustomNode": {
|
||||
"type": "object",
|
||||
"description": "CustomNode is a ComfyUI node from a git repository.",
|
||||
"properties": {
|
||||
"repo": {
|
||||
"type": "string",
|
||||
"description": "Repo is the repository, https only."
|
||||
},
|
||||
"commit": {
|
||||
"type": "string",
|
||||
"description": "Commit is the full commit the node is used at: a branch moves, and a node that changed under a model is a model that changed unasked."
|
||||
},
|
||||
"requirements": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Requirements are the PyPI packages the node needs, by name and version range only. They are installed beside the replica, never into ComfyUI's environment: a package named here at the range named, and every other one at the version ComfyUI already runs. One named here is first on ComfyUI's path, so its range must suit ComfyUI too."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8605,6 +8843,18 @@
|
||||
"peer": {
|
||||
"type": "string",
|
||||
"description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
|
||||
},
|
||||
"api": {
|
||||
"type": "string",
|
||||
"description": "API is what the provider speaks: OpenAI's, unless this says comfyui — a ComfyUI server, asked for pictures at /v1/images/generations and sent the workflow below with the request's prompt in it. Model is then the checkpoint file the workflow loads, as ComfyUI lists it."
|
||||
},
|
||||
"workflow": {
|
||||
"type": "string",
|
||||
"description": "Workflow is the ComfyUI workflow, in its API format: a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane."
|
||||
},
|
||||
"output": {
|
||||
"type": "string",
|
||||
"description": "Output is what it makes: pictures, or meshes (see ComfyUI.Output)."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8625,6 +8875,23 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.License": {
|
||||
"type": "object",
|
||||
"description": "License is the terms what a model makes comes under: said beside every result, and written into a mesh, which travels without the API.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Name is the licence's own: MIT, Apache-2.0."
|
||||
},
|
||||
"url": {
|
||||
"type": "string"
|
||||
},
|
||||
"notice": {
|
||||
"type": "string",
|
||||
"description": "Notice is what must accompany what was made: attribution, terms on the output, what the weights were trained on."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.Model": {
|
||||
"type": "object",
|
||||
"description": "Model is a catalog entry as clients see it.",
|
||||
@@ -8667,6 +8934,14 @@
|
||||
"$ref": "#/components/schemas/catalog.External",
|
||||
"description": "External is a model served by someone else — an OpenAI-compatible API outside the cluster — rather than by replicas placed here. Such a model has no profiles and nothing is placed for it; the router sends its requests to the provider. Most useful named in another model's fallbacks: used only when the cluster has no capacity left. Its prompts leave the cluster, so which keys may name it is worth an allowlist."
|
||||
},
|
||||
"comfyui": {
|
||||
"$ref": "#/components/schemas/catalog.ComfyUI",
|
||||
"description": "ComfyUI is how a model whose profiles run engine comfyui makes its pictures: the workflow, and the file it loads."
|
||||
},
|
||||
"license": {
|
||||
"$ref": "#/components/schemas/catalog.License",
|
||||
"description": "License is the terms of what the model makes, for a model whose output somebody ships: shown with every result and written into a mesh."
|
||||
},
|
||||
"adapters": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
@@ -8710,6 +8985,10 @@
|
||||
"$ref": "#/components/schemas/catalog.Capabilities",
|
||||
"description": "Capabilities is what this model can do beyond plain completion."
|
||||
},
|
||||
"query_prefix": {
|
||||
"type": "string",
|
||||
"description": "QueryPrefix is put before a question an embedding model embeds for a knowledge-base search; documents are embedded without it. Models trained with an instruction on the query side want it, Qwen3-Embedding \"Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: \" for one."
|
||||
},
|
||||
"role": {
|
||||
"type": "string",
|
||||
"description": "Role \"guard\" is a model that judges the gateway's traffic for the guardrails rather than serving clients: hidden from /v1/models and the chat page, refused to any key that names it, critical by default, and always running. Guard says how its answers are read."
|
||||
@@ -8799,6 +9078,10 @@
|
||||
"gpu_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is the weights this profile loads, when not the model's: an embedding model served by vLLM from its repository on a card and by llama.cpp from a GGUF on CPUs is one model with two sources. Empty loads the model's. Updates and canaries move the model's source; a profile with its own keeps it."
|
||||
},
|
||||
"tp": {
|
||||
"type": "integer"
|
||||
},
|
||||
@@ -10250,6 +10533,14 @@
|
||||
"CostMicros": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"Images": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"Meshes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+2
-1
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
|
||||
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
|
||||
| 403 | The key may not use that model. |
|
||||
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
|
||||
|
||||
## Priority
|
||||
|
||||
|
||||
+259
@@ -1038,6 +1038,27 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
No restart, and it survives one: models added this way are stored, while
|
||||
anything in `server.yaml` is declarative and returns on every start.
|
||||
|
||||
A profile may load weights of its own. An embedding model, for example, can
|
||||
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
|
||||
on CPUs, as one model:
|
||||
|
||||
```yaml
|
||||
name: qwen3-embed-4b
|
||||
source: Qwen/Qwen3-Embedding-4B # what a profile loads unless it says
|
||||
capabilities: { embeddings: true }
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: vllm
|
||||
gpu_class: 6000-24g
|
||||
args: ["--runner", "pooling"]
|
||||
- name: cpu
|
||||
engine: llamacpp
|
||||
source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
Updates and canaries move the model's `source`; a profile with its own keeps
|
||||
it.
|
||||
|
||||
### Models that load when someone asks
|
||||
|
||||
A cluster with more models than GPUs can hold every model in its catalog and
|
||||
@@ -1270,6 +1291,128 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Pictures from ComfyUI
|
||||
|
||||
Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
|
||||
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
|
||||
waits, and sends the pictures back as `b64_json`.
|
||||
|
||||
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
|
||||
GPU node like any other, started when asked for, and stopped when unused.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
|
||||
```
|
||||
llmgw ctl config set engines.comfyui 0.39.2
|
||||
```
|
||||
|
||||
The agent fetches ComfyUI's source from GitHub and checks it against the
|
||||
digest llmgw holds for that version. GitHub publishes none for a tag's archive,
|
||||
so a version llmgw does not know is refused. The agent then builds a Python
|
||||
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
|
||||
dependencies are resolved as PyPI stood the day after the release. torch comes
|
||||
from PyPI (CUDA 13), or from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580. The install is checked on the node's GPU, and takes about 10 GB.
|
||||
`engine_overrides` can keep it off nodes that should not have it.
|
||||
|
||||
Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
|
||||
its environment in `.venv`, readable by the user engines run as
|
||||
(`llmgw-engine`). Name it in that node's `agent.yaml`; a node with one named
|
||||
there is not given the installed one:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
comfyui:
|
||||
dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python
|
||||
```
|
||||
|
||||
Then add the model:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
|
||||
comfyui:
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
scaling: {min: 0, max: 1, idle_timeout: 15m}
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: comfyui
|
||||
gpu_class: 6000-24g
|
||||
tp: 1
|
||||
```
|
||||
|
||||
The source is fetched as for any model. It can be one file, such as the
|
||||
single-file checkpoints the built-in workflows load. It can also be a mirror
|
||||
set (a URL ending in `/`) for a model in several files, such as a diffusion
|
||||
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
|
||||
own name, so a workflow of your own can load each one. A set must name the
|
||||
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
|
||||
is named by its own file name.
|
||||
|
||||
ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
|
||||
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
|
||||
listens on the replica's port, demands the replica's key, and lets through only
|
||||
what the router asks: queueing, waiting, fetching a picture and cancelling.
|
||||
Uploads and node installs are refused. Each replica has its own directory,
|
||||
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
|
||||
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
|
||||
`--disable-smart-memory` frees the card between pictures, at the cost of loading
|
||||
the model again for each one.
|
||||
|
||||
**On a ComfyUI server outside the cluster.** Make it an external model with
|
||||
`api: comfyui`:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: external
|
||||
external:
|
||||
url: http://10.0.0.147:8188 # ComfyUI, started with --listen
|
||||
model: flux1-schnell-fp8.safetensors # the checkpoint, as ComfyUI lists it
|
||||
api: comfyui
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
```
|
||||
|
||||
Like any external model, its requests count as leaving the cluster.
|
||||
|
||||
`flux-schnell` (the default) and `flux-dev` are built in. They suit the
|
||||
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
|
||||
For any other model, export the workflow from ComfyUI with **Save (API)** and
|
||||
name the file. In the file, put placeholders where the request's values go:
|
||||
|
||||
- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
|
||||
- `{{width}}` and `{{height}}` are the size.
|
||||
- `{{n}}` is the batch size.
|
||||
- `{{seed}}` is the seed.
|
||||
- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
|
||||
request leaves them out.
|
||||
- `{{checkpoint}}` is the `model` above.
|
||||
|
||||
A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
|
||||
of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
|
||||
send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
|
||||
back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
|
||||
nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
|
||||
before the picture is sent.
|
||||
|
||||
Requests for pictures pass the same checks as any other request: keys,
|
||||
allowlists, limits, admission, and the guardrails on the prompt. A client that
|
||||
leaves takes its workflow off the ComfyUI queue.
|
||||
|
||||
Each request's usage row records how many pictures it made (`llmgw ctl request
|
||||
ID` shows them). A picture's price is set with `per_image`, in micro-units like
|
||||
the other prices, and is charged on top of the prompt's tokens:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
pricing:
|
||||
flux-schnell: {prompt_per_1k: 0, per_image: 20000}
|
||||
```
|
||||
|
||||
An external ComfyUI on a card that llmgw also places models on should be
|
||||
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
|
||||
the card after each picture, and llmgw sees that memory as somebody else's.
|
||||
|
||||
### Federating clusters
|
||||
|
||||
Another llmgw cluster — another site, another team's GPUs — can be a provider
|
||||
@@ -1550,6 +1693,7 @@ and accounting exactly as an API call does.
|
||||
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper |
|
||||
| `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories |
|
||||
| `/v1/files`, `/v1/batches` | batches; see [Batches](#batches) |
|
||||
| `/v1/3d/generations` | 3D models, as jobs asked after; see [Jobs](#jobs-3d-models) |
|
||||
| `GET /v1/models` | the models this key may use |
|
||||
|
||||
## Watching it
|
||||
@@ -1908,6 +2052,121 @@ Each line of the file is `{"custom_id": …, "method": "POST", "url":
|
||||
or past their retention — a project under a guardrails policy with
|
||||
`no_store` is refused it.
|
||||
|
||||
## Jobs: 3D models
|
||||
|
||||
A 3D model from a picture takes minutes on a card that first has to be
|
||||
emptied of whatever else was on it. No client should hold a connection for
|
||||
that, so it is a job: queued, answered with an id, asked after, its result
|
||||
downloaded.
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/3d/generations -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-d '{"model": "trellis2", "image": "<base64>"}'
|
||||
# → 202 {"id": "job_…", "status": "queued", …}
|
||||
curl http://gateway:8080/v1/3d/generations/job_… -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
# → {"status": "completed", "output_file_id": "file-…", "content_type": "model/gltf-binary", "usage": {…}}
|
||||
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY" -o model.glb
|
||||
curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
- **Status:** `queued`, `running`, `completed`, `failed` (with `error`),
|
||||
`cancelling`, `cancelled`, `expired` (not finished within 24 h).
|
||||
`progress` carries what the engine said of the work so far, once it says
|
||||
anything; `GET /v1/3d/generations` lists the caller's jobs.
|
||||
- **Refused at the door:** no key, a model that is not there or not in the
|
||||
key's allowlist, a request the model cannot serve. Refused when served,
|
||||
and said as the job's error: a spent budget, the guardrails, a model
|
||||
that will not come up.
|
||||
- **How it is served:** the request is kept as it was sent and replayed by
|
||||
whichever gateway holds the job's lease, through the same path a
|
||||
client's request takes — limits, admission in the key's band, guardrails,
|
||||
accounting. It is one usage record under the job's id, tagged
|
||||
`job=<id>`, charged like any request; `GET` shows the usage once it is
|
||||
done. A model still loading, a rate limit or a full queue is waited out
|
||||
for as long as the job has: nobody is holding a connection, which is the
|
||||
point.
|
||||
- **Surviving a restart:** a gateway that stops hands its jobs to another
|
||||
within two minutes; a job it was running is run again from the start.
|
||||
- **What it keeps:** the request as sent, and the result as a file owned
|
||||
by the key's project, kept `router.batch.retain_days` (30) like a batch's
|
||||
files; the job's record goes with it. A project under a `no_store`
|
||||
policy is refused jobs, as it is batches.
|
||||
|
||||
A request may also send `seed` and `resolution` (the model's own grid, a
|
||||
multiple of 32 from 64 to 2048). The picture should show one object on a
|
||||
plain background: the model makes a mesh of what it sees, background
|
||||
included.
|
||||
|
||||
### A model that makes them
|
||||
|
||||
A 3D model is made by a ComfyUI model whose `comfyui.output` is `meshes`
|
||||
(see [Pictures from ComfyUI](#pictures-from-comfyui) for engine `comfyui`).
|
||||
Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the
|
||||
uploaded picture, which a `LoadImage` node reads; `{{seed}}`,
|
||||
`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that
|
||||
saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB,
|
||||
vertex colours kept. TripoSR (MIT) on any card of 8 GB:
|
||||
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: hf:stabilityai/TripoSR/model.ckpt
|
||||
comfyui:
|
||||
workflow: /etc/llmgw/workflows/triposr.json # on every control plane
|
||||
output: meshes
|
||||
nodes:
|
||||
- repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR
|
||||
commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3
|
||||
requirements: [omegaconf, trimesh, scikit-image, imageio,
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers]
|
||||
license:
|
||||
name: MIT
|
||||
url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE
|
||||
notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..."
|
||||
scaling: {min: 0, max: 1, idle_timeout: 10m}
|
||||
profiles:
|
||||
- {name: gpu, engine: comfyui, gpu_class: 6000-24g, tp: 1}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["1", 0],
|
||||
"geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
|
||||
"4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
|
||||
}
|
||||
```
|
||||
|
||||
- **Custom nodes** are named by https repository and full commit (a branch
|
||||
is refused: it moves). The gate puts each in the replica's own
|
||||
`custom_nodes` and installs its `requirements` beside the replica, never
|
||||
into ComfyUI's environment. A package named there is installed at the
|
||||
range named; every other one at the version ComfyUI already runs. A
|
||||
package named is first on ComfyUI's path, so its range must suit ComfyUI
|
||||
as well. TripoSR's checkpoint loads with transformers 4, which ComfyUI's
|
||||
`>=4.50.3` allows.
|
||||
- **Progress:** the gate listens to ComfyUI's websocket and the router
|
||||
writes what it hears into the job: `{"state": "queued", "ahead": 2}`,
|
||||
then `{"state": "running", "node_class": "KSampler", "step": 12,
|
||||
"steps": 30, "percent": 40}` for a node that counts its steps, and
|
||||
`done` at the end.
|
||||
- **Licence:** a model's `license` is shown in `/v1/models`, in the job and
|
||||
on the portal, and written into the GLB's `asset.extras.license`, since
|
||||
the file travels without the API. Choose the model for its licence before
|
||||
its quality when what it makes is shipped. TRELLIS.2 needs Meta's gated
|
||||
DINOv3 and, as published, a background remover under a non-commercial
|
||||
licence; Hunyuan3D's licence excludes the EU, the UK and South Korea.
|
||||
- **Price:** `per_mesh` in `auth.pricing`, beside `per_image`; each usage
|
||||
row says how many meshes it made (`meshes`).
|
||||
- **Cards:** the strongest models today (TRELLIS.2, Direct3D-S2) need
|
||||
flash-attention, which runs on Ampere and newer. A Turing card such as
|
||||
the RTX 6000 runs TripoSR.
|
||||
|
||||
People make them at `/3d` on the portal: they choose a picture and watch
|
||||
the job, and the mesh is shown in a viewer (model-viewer, served from the
|
||||
gateway) with a download link and the licence. `llmgw ctl 3d list`, `get`
|
||||
and `cancel` show and stop every owner's jobs.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
@@ -45,6 +45,15 @@ is available to everyone who may use them.
|
||||
A collection stays with the model it was made with: vectors from two
|
||||
models cannot be compared.
|
||||
|
||||
Some embedding models are trained with an instruction before the
|
||||
question and none before the documents. Give it as the entry's
|
||||
`query_prefix`; llmgw puts it before questions only. For
|
||||
Qwen3-Embedding:
|
||||
|
||||
```yaml
|
||||
query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
|
||||
```
|
||||
|
||||
With pgvector 0.7 or later a collection's index is kept at half
|
||||
precision (`halfvec`), which halves its memory and takes models of up to
|
||||
4000 dimensions; with an older pgvector the limit is 2000. A collection
|
||||
@@ -137,6 +146,24 @@ Rules that apply:
|
||||
- Every document added or removed is in the audit log (`llmgw ctl audit`),
|
||||
by name. The text is never recorded.
|
||||
|
||||
### A large import
|
||||
|
||||
A few thousand documents ask the embedding model for far more than it can
|
||||
answer at once. Give the key that sends them a band of its own, below the
|
||||
people's, that may wait for room for minutes rather than the 30 seconds the
|
||||
others wait:
|
||||
|
||||
```sh
|
||||
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
|
||||
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
### Get it
|
||||
|
||||
@@ -60,6 +60,14 @@ rather than repeated.
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
| `images` | `bool` | [Capabilities](#capabilities) |
|
||||
| `meshes` | `bool` | [Capabilities](#capabilities) |
|
||||
| `workflow` | `string` | [ComfyUI](#comfyui) |
|
||||
| `checkpoint` | `string` | [ComfyUI](#comfyui) |
|
||||
| `output` | `string` | [ComfyUI](#comfyui) |
|
||||
| `nodes` | `[]CustomNode` | [ComfyUI](#comfyui) |
|
||||
| `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `python` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `interval` | `time.Duration` | [Control](#control) |
|
||||
| `base_backoff` | `time.Duration` | [Control](#control) |
|
||||
| `max_backoff` | `time.Duration` | [Control](#control) |
|
||||
@@ -71,11 +79,15 @@ rather than repeated.
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `repo` | `string` | [CustomNode](#customnode) |
|
||||
| `commit` | `string` | [CustomNode](#customnode) |
|
||||
| `requirements` | `[]string` | [CustomNode](#customnode) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
| `url` | `string` | [Endpoint](#endpoint) |
|
||||
| `api_key` | `string` | [Endpoint](#endpoint) |
|
||||
| `models` | `[]string` | [Endpoint](#endpoint) |
|
||||
| `engine` | `string` | [Endpoint](#endpoint) |
|
||||
| `workflow` | `string` | [Endpoint](#endpoint) |
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
@@ -90,6 +102,9 @@ rather than repeated.
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -118,6 +133,9 @@ rather than repeated.
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
||||
| `name` | `string` | [License](#license) |
|
||||
| `url` | `string` | [License](#license) |
|
||||
| `notice` | `string` | [License](#license) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -135,6 +153,8 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `comfyui` | `*ComfyUI` | [Model](#model) |
|
||||
| `license` | `*License` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
@@ -145,6 +165,7 @@ rather than repeated.
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
| `max_output` | `int` | [Model](#model) |
|
||||
| `capabilities` | `Capabilities` | [Model](#model) |
|
||||
| `query_prefix` | `string` | [Model](#model) |
|
||||
| `role` | `string` | [Model](#model) |
|
||||
| `guard` | `*GuardModel` | [Model](#model) |
|
||||
| `origin` | `*Origin` | [Model](#model) |
|
||||
@@ -210,6 +231,7 @@ rather than repeated.
|
||||
| `engine` | `string` | [Profile](#profile) |
|
||||
| `engine_version` | `string` | [Profile](#profile) |
|
||||
| `gpu_class` | `string` | [Profile](#profile) |
|
||||
| `source` | `string` | [Profile](#profile) |
|
||||
| `tp` | `int` | [Profile](#profile) |
|
||||
| `pp` | `int` | [Profile](#profile) |
|
||||
| `gpus_per_node` | `int` | [Profile](#profile) |
|
||||
@@ -342,9 +364,12 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
|
||||
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
|
||||
| `default_band` | `string` | [admission.Config](#admissionconfig) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
|
||||
@@ -374,14 +399,18 @@ rather than repeated.
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -715,6 +744,70 @@ token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
`bool`
|
||||
|
||||
### `images`
|
||||
|
||||
`bool`
|
||||
|
||||
Images is a model that makes pictures, asked for at
|
||||
/v1/images/generations. Implied by an external model whose API is
|
||||
ComfyUI's.
|
||||
|
||||
### `meshes`
|
||||
|
||||
`bool`
|
||||
|
||||
Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
meshes.
|
||||
|
||||
## ComfyUI
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is a built-in's name (flux-schnell, the default, or
|
||||
flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
format.
|
||||
|
||||
### `checkpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
source's own name unless set, which a source that is a set of files
|
||||
must.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what the workflow makes: pictures (the default), asked for
|
||||
at /v1/images/generations, or meshes, asked for as a job at
|
||||
/v1/3d/generations from a picture.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]CustomNode`
|
||||
|
||||
Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
custom_nodes at the commit named.
|
||||
|
||||
## ComfyUIConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
engines run as.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
## Control
|
||||
|
||||
### `interval`
|
||||
@@ -800,6 +893,31 @@ see the claim. Long enough to survive a slow query or a brief network
|
||||
stall, short enough that a standby takes over before anyone has finished
|
||||
reading the alert. Zero takes the default.
|
||||
|
||||
## CustomNode
|
||||
|
||||
### `repo`
|
||||
|
||||
`string`
|
||||
|
||||
Repo is the repository, https only.
|
||||
|
||||
### `commit`
|
||||
|
||||
`string`
|
||||
|
||||
Commit is the full commit the node is used at: a branch moves, and a
|
||||
node that changed under a model is a model that changed unasked.
|
||||
|
||||
### `requirements`
|
||||
|
||||
`[]string`
|
||||
|
||||
Requirements are the PyPI packages the node needs, by name and version
|
||||
range only. They are installed beside the replica, never into
|
||||
ComfyUI's environment: a package named here at the range named, and
|
||||
every other one at the version ComfyUI already runs. One named here
|
||||
is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
|
||||
## Endpoint
|
||||
|
||||
### `id`
|
||||
@@ -828,6 +946,13 @@ with one.
|
||||
Engine names the inference engine, which decides what its metrics are
|
||||
called. Empty means vLLM's names.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
name or its JSON; flux-schnell unless set.
|
||||
|
||||
### `pool`
|
||||
|
||||
`string`
|
||||
@@ -919,6 +1044,29 @@ a model that federation added (`llmgw ctl federation add`) rather
|
||||
than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
goes with the peer's own list, and is sent only while the peer is up.
|
||||
|
||||
### `api`
|
||||
|
||||
`string`
|
||||
|
||||
API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
sent the workflow below with the request's prompt in it. Model is then
|
||||
the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
(flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
plane.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1149,6 +1297,25 @@ llmgw in general — no address, model or key of this installation —
|
||||
which the project's repository publishes anyway. On, a visitor is sent
|
||||
to sign in, and without accounts the portal needs an API key.
|
||||
|
||||
## License
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is the licence's own: MIT, Apache-2.0.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
### `notice`
|
||||
|
||||
`string`
|
||||
|
||||
Notice is what must accompany what was made: attribution, terms on
|
||||
the output, what the weights were trained on.
|
||||
|
||||
## LlamaCppConfig
|
||||
|
||||
### `bin`
|
||||
@@ -1305,6 +1472,25 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`*ComfyUI`
|
||||
|
||||
ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
pictures: the workflow, and the file it loads.
|
||||
|
||||
Takes a [ComfyUI](#comfyui) block.
|
||||
|
||||
### `license`
|
||||
|
||||
`*License`
|
||||
|
||||
License is the terms of what the model makes, for a model whose
|
||||
output somebody ships: shown with every result and written into a
|
||||
mesh.
|
||||
|
||||
Takes a [License](#license) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
@@ -1389,6 +1575,16 @@ Capabilities is what this model can do beyond plain completion.
|
||||
|
||||
Takes a [Capabilities](#capabilities) block.
|
||||
|
||||
### `query_prefix`
|
||||
|
||||
`string`
|
||||
|
||||
QueryPrefix is put before a question an embedding model embeds for a
|
||||
knowledge-base search; documents are embedded without it. Models
|
||||
trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
"Instruct: Given a web search query, retrieve relevant passages that
|
||||
answer the query\nQuery: " for one.
|
||||
|
||||
### `role`
|
||||
|
||||
`string`
|
||||
@@ -1826,6 +2022,16 @@ version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
|
||||
|
||||
`string`
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the weights this profile loads, when not the model's: an
|
||||
embedding model served by vLLM from its repository on a card and by
|
||||
llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
loads the model's. Updates and canaries move the model's source; a
|
||||
profile with its own keeps it.
|
||||
|
||||
### `tp`
|
||||
|
||||
`int`
|
||||
@@ -2889,6 +3095,21 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is what one picture costs, for a model that makes them: a
|
||||
picture's cost is the card's time, which its prompt's few tokens do
|
||||
not measure.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
token count measures.
|
||||
|
||||
## admission.Band
|
||||
|
||||
### `name`
|
||||
@@ -2908,6 +3129,15 @@ Priority orders bands; higher is served first.
|
||||
SaturationCeiling is the pool saturation above which this band waits.
|
||||
1.0 means "never hold this band back".
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL, set, is how long this band's requests may wait instead of
|
||||
the controller's: a band for bulk work — a knowledge-base import, a
|
||||
backfill — whose requests had rather wait ten minutes behind people
|
||||
than be turned away after thirty seconds.
|
||||
|
||||
## admission.Config
|
||||
|
||||
### `bands`
|
||||
@@ -3121,6 +3351,14 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`float64`
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL is how long this band's requests may wait for room; the
|
||||
admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
had rather wait behind people than be turned away.
|
||||
|
||||
## config.Price
|
||||
|
||||
### `prompt_per_1k`
|
||||
@@ -3131,6 +3369,19 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is the cost of one picture, for a model that makes them
|
||||
(/v1/images/generations), on top of its prompt's tokens.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
|
||||
## engines.Config
|
||||
|
||||
### `disabled`
|
||||
@@ -3174,6 +3425,14 @@ Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
Takes a [SimConfig](#simconfig) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`ComfyUIConfig`
|
||||
|
||||
ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,17 +39,27 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
t.Error("a model priced per picture reads as free, and would pass a spent budget")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroPricedModelsStayAvailableWhenTheBudgetIsSpent(t *testing.T) {
|
||||
// This is what makes budget fallback work: a key with nothing left can
|
||||
// still be served something that costs nothing.
|
||||
|
||||
@@ -18,6 +18,13 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
// token count measures.
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Zero reports whether this model spends nothing.
|
||||
@@ -25,14 +32,17 @@ type Price struct {
|
||||
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
func divUp(n, d int64) int64 {
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// ComfyUIConfig locates a ComfyUI installation.
|
||||
type ComfyUIConfig struct {
|
||||
// Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
// engines run as.
|
||||
Dir string `koanf:"dir"`
|
||||
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
Python string `koanf:"python"`
|
||||
}
|
||||
|
||||
// ComfyUI runs ComfyUI as an engine, behind llmgw's gate (llmgw
|
||||
// comfyui-gate): ComfyUI asks for no key, so it listens on the loopback and
|
||||
// the gate, which does, on the replica's port. Its replicas make pictures
|
||||
// for /v1/images/generations.
|
||||
type ComfyUI struct {
|
||||
cfg ComfyUIConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*ComfyUI) Name() string { return "comfyui" }
|
||||
|
||||
// HealthPath is the gate's, which answers once ComfyUI does.
|
||||
func (*ComfyUI) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is empty: ComfyUI publishes no metrics, and the router reads
|
||||
// none from it.
|
||||
func (*ComfyUI) MetricsPath() string { return "" }
|
||||
|
||||
// ServesHTTP is true: a ComfyUI replica is one process on one node.
|
||||
func (*ComfyUI) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the environment the agent installed:
|
||||
// its interpreter, with ComfyUI's code beside its bin, in comfyui/.
|
||||
func (c *ComfyUI) UseInstalled(python string) { c.installed.Store(&python) }
|
||||
|
||||
// Configured reports whether the configuration names a ComfyUI.
|
||||
func (c *ComfyUI) Configured() bool { return c.cfg.Dir != "" }
|
||||
|
||||
func (c *ComfyUI) dir() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return filepath.Join(filepath.Dir(filepath.Dir(*p)), "comfyui")
|
||||
}
|
||||
return c.cfg.Dir
|
||||
}
|
||||
|
||||
func (c *ComfyUI) python() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
if c.cfg.Python != "" {
|
||||
return c.cfg.Python
|
||||
}
|
||||
return filepath.Join(c.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var comfyVersion = regexp.MustCompile(`__version__\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports ComfyUI's version, from the file its build writes it to:
|
||||
// importing ComfyUI to ask would load torch and look for cards.
|
||||
func (c *ComfyUI) Probe(context.Context) (string, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no ComfyUI directory is configured")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "main.py")); err != nil {
|
||||
return "", fmt.Errorf("%s is not ComfyUI: %w", dir, err)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "comfyui_version.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // installed, of a version it does not say
|
||||
}
|
||||
if m := comfyVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. ComfyUI runs a
|
||||
// workflow on one device and has no tensor parallelism to spread it by.
|
||||
func (*ComfyUI) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: ComfyUI runs on one GPU of one node", p.Name)
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is the gate, which starts ComfyUI itself.
|
||||
func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return nil, fmt.Errorf("comfyui: no ComfyUI directory is configured (engines.comfyui.dir)")
|
||||
}
|
||||
bin, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := s.Profile
|
||||
args := []string{"comfyui-gate",
|
||||
"--listen", net.JoinHostPort(s.Host(), strconv.Itoa(s.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
"--weights", p.ModelPath,
|
||||
// Relative to where engines run — their user's home — so it is
|
||||
// theirs to write.
|
||||
"--state", filepath.Join("comfyui", s.ReplicaID),
|
||||
"--python", c.python(),
|
||||
"--comfyui", dir,
|
||||
"--",
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(s)
|
||||
if s.APIKey != "" {
|
||||
env["LLMGW_ENGINE_KEY"] = s.APIKey
|
||||
}
|
||||
// A model's custom nodes, which only the catalog sets — a profile's env
|
||||
// may not name the variable — and so taken before baseEnv drops it.
|
||||
if nodes := p.Env[comfyNodesEnv]; nodes != "" {
|
||||
env[comfyNodesEnv] = nodes
|
||||
}
|
||||
return []Command{{Name: "comfyui", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
|
||||
// comfyNodesEnv is the custom nodes the catalog gives a ComfyUI model, as
|
||||
// JSON; catalog.NodesEnv and comfy.NodesEnv by the same name.
|
||||
const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
@@ -0,0 +1,80 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestComfyUIRunsBehindTheGateWithItsKeyOutOfSight(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/comfyui"}}
|
||||
cmds, err := c.Commands(ShardSpec{
|
||||
ReplicaID: "r-flux-1", HTTPPort: 8100, BindAddr: "10.0.0.147", APIKey: "sekret", GPUIndices: []int{0},
|
||||
Profile: Profile{Name: "p", Engine: "comfyui", ModelPath: "/w/flux.safetensors", ServedModelName: "flux",
|
||||
Args: []string{"--disable-smart-memory"}},
|
||||
})
|
||||
if err != nil || len(cmds) != 1 {
|
||||
t.Fatalf("%v %v", cmds, err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
for _, want := range []string{"comfyui-gate", "--listen 10.0.0.147:8100", "--model flux", "--weights /w/flux.safetensors",
|
||||
"--state comfyui/r-flux-1", "--python /opt/comfyui/.venv/bin/python", "--comfyui /opt/comfyui", "-- --disable-smart-memory"} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("%q not in %q", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "sekret") {
|
||||
t.Error("the key is on the command line, where every user can read it")
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "LLMGW_ENGINE_KEY=sekret") || !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||
t.Error("the key or the card is not in the environment")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIRunsOnOneCard(t *testing.T) {
|
||||
c := &ComfyUI{}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 2}, h100(2)); err == nil {
|
||||
t.Error("a two-card ComfyUI profile was accepted")
|
||||
}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 1}, h100(1)); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if _, err := c.Commands(ShardSpec{Profile: Profile{ModelPath: "/w"}}); err == nil {
|
||||
t.Error("commands without a ComfyUI directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsOfferedByTheVersionItsBuildWrote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: dir}}
|
||||
if _, err := c.Probe(t.Context()); err == nil {
|
||||
t.Error("a directory without main.py passed for ComfyUI")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "main.py"), nil, 0o644)
|
||||
_ = os.WriteFile(filepath.Join(dir, "comfyui_version.py"), []byte("__version__ = \"0.39.2\"\n"), 0o644)
|
||||
if v, err := c.Probe(t.Context()); err != nil || v != "0.39.2" {
|
||||
t.Errorf("%q %v", v, err)
|
||||
}
|
||||
if got := NewRegistry(Config{ComfyUI: ComfyUIConfig{Dir: dir}}).ProbeConfigured(t.Context()); got["comfyui"] != "0.39.2" {
|
||||
t.Errorf("a configured ComfyUI is reported as %v", got)
|
||||
}
|
||||
if err := UnsafeArgs("comfyui", []string{"--enable-manager"}); err == nil {
|
||||
t.Error("a profile may install ComfyUI nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnInstalledComfyUIIsFoundBesideItsInterpreter(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/by-hand"}}
|
||||
c.UseInstalled("/opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python")
|
||||
cmds, err := c.Commands(ShardSpec{ReplicaID: "r", HTTPPort: 1, Profile: Profile{ModelPath: "/w", ServedModelName: "m"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
if !strings.Contains(args, "--python /opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python") ||
|
||||
!strings.Contains(args, "--comfyui /opt/llmgw/engines/comfyui/0.39.2-cu13/comfyui") {
|
||||
t.Errorf("the installed ComfyUI is not what runs: %s", args)
|
||||
}
|
||||
}
|
||||
@@ -214,6 +214,7 @@ func NewRegistry(cfg Config) *Registry {
|
||||
r.Add(&LlamaCpp{cfg: cfg.LlamaCpp})
|
||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||
r.Add(&Sim{cfg: cfg.Sim})
|
||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -262,6 +263,8 @@ type Config struct {
|
||||
// offers it only when its interpreter is named here.
|
||||
TRTLLM TRTLLMConfig `koanf:"trtllm"`
|
||||
Sim SimConfig `koanf:"sim"`
|
||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||
}
|
||||
|
||||
// --- shared helpers ---
|
||||
|
||||
@@ -477,7 +477,7 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
||||
|
||||
func TestRegistry(t *testing.T) {
|
||||
r := NewRegistry(Config{})
|
||||
if got := r.Names(); !slices.Equal(got, []string{"llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
t.Errorf("registry contents: %v", got)
|
||||
}
|
||||
if _, err := r.Get("vllm"); err != nil {
|
||||
|
||||
@@ -119,6 +119,19 @@ var flagsNever = map[string]map[string]string{
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
},
|
||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||
// where its files are.
|
||||
"comfyui": {
|
||||
"--listen": "is set by the agent", "--base-directory": "is set by the agent",
|
||||
"--extra-model-paths-config": "is set by the agent",
|
||||
"--output-directory": "writes files", "--temp-directory": "writes files",
|
||||
"--input-directory": "reads files", "--user-directory": "writes files",
|
||||
"--database-url": "writes a database elsewhere", "--front-end-root": "serves a directory",
|
||||
"--enable-cors-header": "lets other sites' pages call it",
|
||||
"--enable-manager": "installs nodes, which run Python",
|
||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||
"--comfy-api-base": "sends work to another service",
|
||||
},
|
||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||
// hyphenated names do not reach them.
|
||||
"trtllm": {
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// comfyuiSource is a ComfyUI release archive as GitHub makes one, and the
|
||||
// recipe pointed at it.
|
||||
func comfyuiSource(t *testing.T, in *Installer, rel *release, digest string) {
|
||||
t.Helper()
|
||||
archive := tarGz(t, map[string]string{
|
||||
"ComfyUI-0.39.2/main.py": "print('comfy')\n",
|
||||
"ComfyUI-0.39.2/comfyui_version.py": "__version__ = \"0.39.2\"\n",
|
||||
"ComfyUI-0.39.2/requirements.txt": "comfyui-frontend-package==1.53.10\ntorch\ntorchvision\n" +
|
||||
"numpy>=1.25.0\n\n#non essential dependencies:\nkornia>=0.7.1\n",
|
||||
}, nil)
|
||||
rel.assets["v0.39.2.tar.gz"] = archive
|
||||
sum := sha256.Sum256(archive)
|
||||
if digest == "" {
|
||||
digest = hex.EncodeToString(sum[:])
|
||||
}
|
||||
old := comfyuiReleases
|
||||
comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{"0.39.2": {digest, "2026-10-07T21:58:07Z"}}
|
||||
t.Cleanup(func() { comfyuiReleases = old })
|
||||
}
|
||||
|
||||
var turingNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
|
||||
NVIDIADriver: "580.82.07", MinComputeCapability: 7.5}
|
||||
|
||||
func TestComfyUIIsInstalledFromItsSourceCheckedAgainstTheRecipe(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready {
|
||||
t.Fatalf("not ready: %+v", got[0])
|
||||
}
|
||||
dir := filepath.Join(in.Root, "comfyui", "0.39.2-cu13")
|
||||
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "comfyui", "main.py")); err != nil {
|
||||
t.Errorf("ComfyUI's code is not beside the interpreter: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
for _, want := range []string{
|
||||
// Not PyPI's date, which ComfyUI has none of: the recipe's, a day on.
|
||||
"--exclude-newer 2026-10-08T21:58:07Z",
|
||||
"comfyui-frontend-package==1.53.10", "numpy>=1.25.0", "kornia>=0.7.1", "torch",
|
||||
"--require-hashes",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if strings.Contains(all, "/pypi/") {
|
||||
t.Error("PyPI was asked when ComfyUI was published")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIOnAnOlderDriverTakesPyTorchsCUDA12Torch(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
host := turingNode
|
||||
host.NVIDIADriver = "550.163.01"
|
||||
in, rel := pythonInstaller(t, host, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("got %+v", got[0])
|
||||
}
|
||||
var project string
|
||||
for _, c := range sys.commands {
|
||||
if strings.HasPrefix(c, "pyproject: ") {
|
||||
project = c
|
||||
}
|
||||
}
|
||||
// torch from PyTorch's index alone, named once, as the build names it.
|
||||
if !strings.Contains(project, `torch = { index = "extra" }`) || strings.Count(project, `"torch"`) != 1 {
|
||||
t.Errorf("torch is not from PyTorch's index alone:\n%s", project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIWhoseSourceIsNotTheOneTheRecipeNamesIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, strings.Repeat("ab", 32))
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match") {
|
||||
t.Fatalf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") {
|
||||
t.Errorf("installed from a source that did not match: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsInstalledOnlyWhereItCanRun(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, ubuntu, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}}); got[0].State != Skipped {
|
||||
t.Errorf("a node with no GPU: %+v", got[0])
|
||||
}
|
||||
in, _ = pythonInstaller(t, turingNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.1.0"}})
|
||||
if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "knows the source digest of ComfyUI 0.39.2") {
|
||||
t.Errorf("a version without a digest: %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequirementsFileIsReadForPackagesAndNothingElse(t *testing.T) {
|
||||
got, err := requirementsIn("torch\ntorchvision>=0.20\nPillow\n# a comment\naiohttp>=3.11 # inline\n", pytorchPackages)
|
||||
if err != nil || strings.Join(got, ",") != "Pillow,aiohttp>=3.11" {
|
||||
t.Errorf("%v %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"-r other.txt", "--index-url https://evil", "x @ https://evil/x.whl", "-e ."} {
|
||||
if _, err := requirementsIn(bad+"\n", nil); err == nil {
|
||||
t.Errorf("%q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -66,12 +66,17 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
|
||||
}
|
||||
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
var asOf string
|
||||
if !b.AsOf.IsZero() {
|
||||
asOf = b.AsOf.UTC().Format(time.RFC3339)
|
||||
} else {
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
asOf = published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
}
|
||||
asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
|
||||
cache := filepath.Join(in.root(), ".uv-cache")
|
||||
env := []string{
|
||||
@@ -96,7 +101,14 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
reqs := b.Requirements
|
||||
reqs := slices.Clone(b.Requirements)
|
||||
if b.Source != nil {
|
||||
more, err := in.unpackSource(ctx, w, b, tmp, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reqs = append(reqs, more...)
|
||||
}
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
@@ -320,3 +332,68 @@ func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Buil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackSource fetches an engine's source archive, checks it against the
|
||||
// digest the recipe holds, puts its code in the environment, and returns the
|
||||
// requirements its requirements file names — less the packages a second
|
||||
// index gives, which the build names itself.
|
||||
func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env string, st *Status) ([]string, error) {
|
||||
src := b.Source
|
||||
url := src.URL
|
||||
if strings.HasPrefix(url, "/") {
|
||||
url = strings.TrimSuffix(in.downloadBase(), "/") + url
|
||||
}
|
||||
scratch := env + ".source"
|
||||
_ = os.RemoveAll(scratch)
|
||||
if err := os.MkdirAll(scratch, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(scratch) //nolint:errcheck // scratch
|
||||
archive := filepath.Join(scratch, "source.tar.gz")
|
||||
name := w.Engine + " " + w.Version + "'s source"
|
||||
if err := in.download(ctx, url, name, archive, src.SHA256, st); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
in.progress(st, "unpacking %s", name)
|
||||
if err := untarGz(archive, scratch); err != nil {
|
||||
return nil, fmt.Errorf("unpacking %s: %w", name, err)
|
||||
}
|
||||
dst := filepath.Join(env, src.Dir)
|
||||
if err := os.Rename(filepath.Join(scratch, src.Top), dst); err != nil {
|
||||
return nil, fmt.Errorf("%s has no %s at its top: %w", name, src.Top, err)
|
||||
}
|
||||
if src.Requirements == "" {
|
||||
return nil, nil
|
||||
}
|
||||
f, err := os.ReadFile(filepath.Join(dst, src.Requirements))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return requirementsIn(string(f), b.IndexPackages)
|
||||
}
|
||||
|
||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||
// or install what the lock never saw.
|
||||
func requirementsIn(text string, skip []string) ([]string, error) {
|
||||
var out []string
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line, _, _ = strings.Cut(line, "#")
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "-") || strings.Contains(line, "://") || strings.Contains(line, " @ ") {
|
||||
return nil, fmt.Errorf("the requirements name %q, which is not a package from PyPI", line)
|
||||
}
|
||||
name := strings.ToLower(line)
|
||||
if i := strings.IndexAny(name, "<>=!~;[ "); i >= 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
if slices.Contains(skip, name) {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -53,6 +53,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return []byte("error: Hash mismatch for torch\n"), errExit
|
||||
}
|
||||
venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
|
||||
_ = os.WriteFile(filepath.Join(venv, "bin", "python"), []byte("#!/bin/sh\n"), 0o755)
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
@@ -68,12 +69,31 @@ type Build struct {
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// Source, for a Python engine that is not on PyPI, is its own code: an
|
||||
// archive pinned by its digest, unpacked into the environment, whose
|
||||
// requirements file is installed with Requirements. AsOf is then the
|
||||
// date its dependencies are resolved as of — the day after its version
|
||||
// was published, which PyPI cannot be asked for.
|
||||
Source *Source
|
||||
AsOf time.Time
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Source is an engine's code as an archive.
|
||||
type Source struct {
|
||||
// URL is fetched as is; SHA256 is what it must hash to, written into
|
||||
// the recipe because a project's source archives come with no published
|
||||
// digest of their own.
|
||||
URL, SHA256 string
|
||||
// Top is the archive's one top directory, put in the environment as Dir.
|
||||
Top, Dir string
|
||||
// Requirements is the requirements file, in Dir.
|
||||
Requirements string
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
@@ -104,6 +124,7 @@ var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
@@ -556,3 +577,78 @@ func orUnknown(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// --- ComfyUI ---
|
||||
|
||||
// comfyuiReleases are the ComfyUI versions this agent installs: each source
|
||||
// archive's digest, taken when it was added here and confirmed from a second
|
||||
// machine, and when the version was published. GitHub publishes no digest
|
||||
// for a tag's archive, so a version not listed is refused rather than
|
||||
// installed unchecked.
|
||||
var comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{
|
||||
"0.39.2": {"1d655e22cf546700fc1d6d2536e27701695c5832a51f63bb16c41866ac09be0b", "2026-10-07T21:58:07Z"},
|
||||
}
|
||||
|
||||
// ComfyUIRepo is where ComfyUI's source is published.
|
||||
const ComfyUIRepo = "Comfy-Org/ComfyUI"
|
||||
|
||||
// comfyuiDir is where ComfyUI's code is put in its environment: main.py is
|
||||
// <environment>/comfyui/main.py, beside bin/python.
|
||||
const comfyuiDir = "comfyui"
|
||||
|
||||
// comfyuiCandidates are the ComfyUI builds for a machine: its source, in a
|
||||
// Python environment with the torch for the node's driver — PyPI's, built
|
||||
// for CUDA 13, or PyTorch's CUDA 12.9 build for a driver CUDA 13 does not
|
||||
// run on.
|
||||
func comfyuiCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("ComfyUI", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rel, ok := comfyuiReleases[version]
|
||||
if !ok {
|
||||
known := make([]string, 0, len(comfyuiReleases))
|
||||
for v := range comfyuiReleases {
|
||||
known = append(known, v)
|
||||
}
|
||||
slices.Sort(known)
|
||||
return nil, fmt.Errorf("this agent knows the source digest of ComfyUI %s, not of %s; "+
|
||||
"install that one by hand and name it in agent.yaml (engines.comfyui.dir)",
|
||||
strings.Join(known, ", "), version)
|
||||
}
|
||||
published, err := time.Parse(time.RFC3339, rel.published)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
build := func(variant string) Build {
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Source: &Source{
|
||||
URL: "/" + ComfyUIRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||
SHA256: rel.sha256, Top: "ComfyUI-" + version, Dir: comfyuiDir,
|
||||
Requirements: "requirements.txt",
|
||||
},
|
||||
AsOf: published.Add(24 * time.Hour),
|
||||
// ComfyUI is started as a script, so what the agent is given is
|
||||
// the environment's interpreter; the proof is importing what it
|
||||
// imports first, from where its code is.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", `import os, sys, torch, aiohttp, safetensors
|
||||
d = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(sys.executable))), "` + comfyuiDir + `")
|
||||
assert os.path.isfile(os.path.join(d, "main.py")), d + " has no main.py"
|
||||
print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 15,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := build("cu129")
|
||||
cu129.ExtraIndex, cu129.IndexPackages = pytorchCU129, pytorchPackages
|
||||
return pick("ComfyUI "+version, variant, []candidate{
|
||||
{build("cu13"), gpuNeeds{minDriverForCUDA13, 7.5}, "ComfyUI " + version + " on CUDA 13's torch"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, 7.0}, "ComfyUI " + version + " on CUDA 12.9's torch"},
|
||||
}, h)
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// An engine's output goes to a file of its own, not to a pipe of the
|
||||
// agent's. A pipe has one reader, and that reader is the agent that
|
||||
// started the engine: when that agent restarts, the engine it left running
|
||||
// — and adopts a moment later — writes to a pipe nobody reads, and its next
|
||||
// write past the pipe's buffer fails with EPIPE. A Python engine's progress
|
||||
// bar then raises, and takes the request it was reporting on with it. A
|
||||
// file has no reader to lose: the engine appends to it, every agent reads
|
||||
// it, and the output from before a restart is there for the one after.
|
||||
|
||||
// maxOutputBytes bounds one process's file: past it the file is emptied and
|
||||
// the process goes on appending to it. A progress bar rewriting itself for
|
||||
// an hour would otherwise fill the disk.
|
||||
const maxOutputBytes = 64 << 20
|
||||
|
||||
// outputPath is where a shard's process writes.
|
||||
func (s *Supervisor) outputPath(shardID, name string) string {
|
||||
if name != "" {
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+"."+name+".log")
|
||||
}
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".log")
|
||||
}
|
||||
|
||||
// openOutput is the file a process is started with: new, for a process
|
||||
// that is new.
|
||||
func (s *Supervisor) openOutput(shardID, name string) (*os.File, error) {
|
||||
return os.OpenFile(s.outputPath(shardID, name), os.O_CREATE|os.O_WRONLY|os.O_APPEND|os.O_TRUNC, 0o600) //nolint:gosec // the agent's own directory
|
||||
}
|
||||
|
||||
// follow reads a process's file into the shard's log as it is written,
|
||||
// until the shard's output is stopped and the file has been read to its
|
||||
// end. A file an earlier agent began is read from the start, so the output
|
||||
// from before a restart is in the log too.
|
||||
func (s *Supervisor) follow(sh *Shard, path, name string) {
|
||||
defer sh.outDone.Done()
|
||||
f, err := os.Open(path) //nolint:gosec // the agent's own directory
|
||||
if err != nil {
|
||||
sh.logs.add("[" + name + "] output unavailable: " + err.Error())
|
||||
return
|
||||
}
|
||||
defer f.Close()
|
||||
var offset int64
|
||||
var rest []byte
|
||||
buf := make([]byte, 64<<10)
|
||||
read := func() {
|
||||
for {
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if st.Size() < offset {
|
||||
// Emptied, by this agent or another: begin again.
|
||||
offset, rest = 0, nil
|
||||
}
|
||||
if st.Size() > maxOutputBytes {
|
||||
_ = os.Truncate(path, 0)
|
||||
offset, rest = 0, nil
|
||||
continue
|
||||
}
|
||||
n, err := f.ReadAt(buf, offset)
|
||||
if n > 0 {
|
||||
offset += int64(n)
|
||||
rest = append(rest, buf[:n]...)
|
||||
for {
|
||||
i := bytes.IndexByte(rest, '\n')
|
||||
if i < 0 {
|
||||
break
|
||||
}
|
||||
s.addLine(sh, name, rest[:i])
|
||||
rest = rest[i+1:]
|
||||
}
|
||||
// A progress bar is a line rewritten with \r and never ended:
|
||||
// its latest state is what is wanted, not a line a megabyte
|
||||
// long. Kept, as llama.cpp and tqdm write them.
|
||||
if len(rest) > 1<<20 {
|
||||
s.addLine(sh, name, rest)
|
||||
rest = nil
|
||||
}
|
||||
}
|
||||
if err != nil || n == 0 {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
t := time.NewTicker(200 * time.Millisecond)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-sh.outStop:
|
||||
read()
|
||||
if len(rest) > 0 {
|
||||
s.addLine(sh, name, rest)
|
||||
}
|
||||
return
|
||||
case <-t.C:
|
||||
read()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Supervisor) addLine(sh *Shard, name string, b []byte) {
|
||||
// The last state of a line rewritten with \r, as a terminal would show.
|
||||
if i := bytes.LastIndexByte(b, '\r'); i >= 0 {
|
||||
b = b[i+1:]
|
||||
}
|
||||
line := string(b)
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
|
||||
// removeOutput drops a shard's files once nothing writes to them.
|
||||
func (s *Supervisor) removeOutput(shardID string) {
|
||||
matches, _ := filepath.Glob(filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".*log"))
|
||||
for _, m := range matches {
|
||||
_ = os.Remove(m)
|
||||
}
|
||||
_ = os.Remove(s.outputPath(shardID, ""))
|
||||
}
|
||||
|
||||
// stopOutput has every follower read what is left and stop. Called once
|
||||
// the process has ended, before the shard is done, so what it wrote last
|
||||
// is in the log by the time anyone reads the log for why it ended.
|
||||
func (s *Supervisor) stopOutput(sh *Shard) {
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
@@ -104,6 +102,12 @@ type Shard struct {
|
||||
ID string
|
||||
Spec engines.ShardSpec
|
||||
|
||||
// outStop ends the followers of the processes' output files, and
|
||||
// outDone is their having read to the end.
|
||||
outStop chan struct{}
|
||||
outOnce sync.Once
|
||||
outDone sync.WaitGroup
|
||||
|
||||
mu sync.Mutex
|
||||
state State
|
||||
pid int
|
||||
@@ -248,6 +252,7 @@ func (s *Supervisor) Start(ctx context.Context, id string, spec engines.ShardSpe
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: time.Now(),
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
}
|
||||
s.shards[id] = sh
|
||||
s.mu.Unlock()
|
||||
@@ -315,14 +320,15 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
// CommandContext kills only the leader, so replace that with a group kill.
|
||||
c.Cancel = func() error { return killGroup(c.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := c.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, "")
|
||||
if err != nil {
|
||||
cancel()
|
||||
return err
|
||||
}
|
||||
c.Stderr = c.Stdout.(io.Writer) // interleave, as the engine writes them
|
||||
|
||||
if err := c.Start(); err != nil {
|
||||
c.Stdout, c.Stderr = out, out // interleaved, as the engine writes them
|
||||
err = c.Start()
|
||||
out.Close() // the process holds its own
|
||||
if err != nil {
|
||||
cancel()
|
||||
return fmt.Errorf("starting %s: %w", cmd.Path, err)
|
||||
}
|
||||
@@ -339,7 +345,8 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
"shard", sh.ID, "err", err)
|
||||
}
|
||||
|
||||
go s.pump(sh, stdout, cmd.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), cmd.Name)
|
||||
go s.wait(sh)
|
||||
return nil
|
||||
}
|
||||
@@ -353,16 +360,18 @@ func (s *Supervisor) spawn(ctx context.Context, sh *Shard, c engines.Command) (*
|
||||
s.dropTo(proc)
|
||||
proc.Cancel = func() error { return killGroup(proc.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := proc.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, c.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
proc.Stderr = proc.Stdout.(io.Writer)
|
||||
|
||||
if err := proc.Start(); err != nil {
|
||||
proc.Stdout, proc.Stderr = out, out
|
||||
err = proc.Start()
|
||||
out.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("starting %s: %w", c.Path, err)
|
||||
}
|
||||
go s.pump(sh, stdout, c.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, c.Name), c.Name)
|
||||
// Reaped so it does not become a zombie; its exit does not end the shard.
|
||||
go func() { _ = proc.Wait() }()
|
||||
return proc, nil
|
||||
@@ -395,24 +404,6 @@ func waitForPort(ctx context.Context, addr string, timeout time.Duration) (err e
|
||||
}
|
||||
|
||||
// pump copies a process's output into the ring buffer and the log.
|
||||
func (s *Supervisor) pump(sh *Shard, r io.Reader, name string) {
|
||||
sc := bufio.NewScanner(r)
|
||||
// Engine tracebacks and progress bars produce long lines; the default
|
||||
// 64 KiB limit truncates them exactly when they matter.
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
}
|
||||
|
||||
// wait reaps the process and records why it ended.
|
||||
func (s *Supervisor) wait(sh *Shard) {
|
||||
defer close(sh.done)
|
||||
@@ -440,6 +431,7 @@ func (s *Supervisor) wait(sh *Shard) {
|
||||
final, code := sh.state, sh.exitCode
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
if final == Failed {
|
||||
s.log.Error("shard failed", "shard", sh.ID, "exit_code", code, "err", err,
|
||||
@@ -760,6 +752,7 @@ func (s *Supervisor) writeState(sh *Shard) error {
|
||||
|
||||
func (s *Supervisor) removeState(sh *Shard) {
|
||||
_ = os.Remove(s.statePath(sh.ID))
|
||||
s.removeOutput(sh.ID)
|
||||
}
|
||||
|
||||
// Orphan is an engine process left behind by a previous agent.
|
||||
@@ -838,15 +831,18 @@ func (s *Supervisor) Adopt(o Orphan) (*Shard, error) {
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: o.StartedAt,
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
adopted: true,
|
||||
cmdline: o.Cmdline,
|
||||
}
|
||||
s.shards[o.ShardID] = sh
|
||||
s.mu.Unlock()
|
||||
|
||||
// Output is gone with the previous agent; say so rather than presenting an
|
||||
// empty log as if the engine had been silent.
|
||||
sh.logs.add("[adopted from a previous agent; earlier output is not available]")
|
||||
// Its output file is read from the start: what it wrote under the
|
||||
// previous agent is there, and what it writes now goes on arriving.
|
||||
sh.logs.add("[adopted from a previous agent]")
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), "")
|
||||
|
||||
go s.poll(sh)
|
||||
return sh, nil
|
||||
@@ -874,6 +870,7 @@ func (s *Supervisor) poll(sh *Shard) {
|
||||
}
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
s.log.Info("adopted shard exited", "shard", sh.ID, "expected", expected)
|
||||
return
|
||||
@@ -900,6 +897,7 @@ func (s *Supervisor) KillOrphan(o Orphan) error {
|
||||
}
|
||||
// An already-missing state file is success: the previous agent may have
|
||||
// reaped the process itself between the scan and this call.
|
||||
s.removeOutput(o.ShardID)
|
||||
if err := os.Remove(s.statePath(o.ShardID)); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -905,3 +906,78 @@ func TestAnAdoptedShardsComputeIsKeptWithoutCrashing(t *testing.T) {
|
||||
t.Fatalf("lost: %+v", again)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdoptedEngineGoesOnWritingAndItsEarlierOutputIsKept(t *testing.T) {
|
||||
// An engine's output went to a pipe of the agent's. The agent that
|
||||
// restarted and adopted the engine left that pipe with no reader, and
|
||||
// the engine's next write past the pipe's buffer failed with EPIPE —
|
||||
// which, in a Python engine, raised inside whatever was being reported
|
||||
// on and failed the request. Output goes to a file now: a reader is not
|
||||
// something a file can lose, and the lines from before the restart are
|
||||
// there for the agent after it.
|
||||
s, _ := newSupervisor(t)
|
||||
// 256 KiB at once outgrows a pipe's buffer; a reader gone means EPIPE.
|
||||
sh, err := s.Start(t.Context(), "a", engines.ShardSpec{ReplicaID: "r"},
|
||||
shellCommand(`echo before-restart; sleep 1; head -c 262144 /dev/zero | tr '\0' x; echo; echo after-restart-done; sleep 30`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 3*time.Second, "the first line was never captured", func() bool {
|
||||
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "before-restart")
|
||||
})
|
||||
// The agent goes away without stopping the engine: nothing reads the
|
||||
// output between here and the adoption.
|
||||
pid := sh.Status().PID
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
|
||||
s2, err := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 3 * time.Second, LogLines: 64},
|
||||
&scriptedHealth{}, slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
orphans, _ := s2.FindOrphans()
|
||||
adopted, err := s2.Adopt(orphans[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 5*time.Second, "the engine's output after the restart never arrived: "+strings.Join(adopted.Logs(5), " | "), func() bool {
|
||||
return strings.Contains(strings.Join(adopted.Logs(64), "\n"), "after-restart-done")
|
||||
})
|
||||
if !strings.Contains(strings.Join(adopted.Logs(64), "\n"), "before-restart") {
|
||||
t.Error("the output from before the restart is not in the adopted log")
|
||||
}
|
||||
if !processAlive(pid) {
|
||||
t.Fatal("the engine died writing output nobody was reading")
|
||||
}
|
||||
if err := s2.Stop(t.Context(), "a"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(s2.outputPath("a", "")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("the output file was not removed with the shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
|
||||
s, _ := newSupervisor(t)
|
||||
path := s.outputPath("a", "")
|
||||
// A file already past the bound, as a long-running engine leaves one.
|
||||
big, _ := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
_, _ = big.Write(bytes.Repeat([]byte("x"), maxOutputBytes+1))
|
||||
big.Close()
|
||||
sh := &Shard{ID: "a", logs: newRingBuffer(8), done: make(chan struct{}), outStop: make(chan struct{})}
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, path, "")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
f, _ := os.OpenFile(path, os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
_, _ = f.WriteString("still here\n")
|
||||
f.Close()
|
||||
s.stopOutput(sh)
|
||||
st, _ := os.Stat(path)
|
||||
if st.Size() > 1024 {
|
||||
t.Errorf("the file was not emptied: %d bytes", st.Size())
|
||||
}
|
||||
if !strings.Contains(strings.Join(sh.Logs(8), "\n"), "still here") {
|
||||
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
)
|
||||
|
||||
// comfyKeyEnv carries the replica's key to the gate: the environment, which
|
||||
// only the engine's user and root can read, rather than the command line,
|
||||
// which every user on the node can.
|
||||
const comfyKeyEnv = "LLMGW_ENGINE_KEY"
|
||||
|
||||
func newComfyUIGateCmd() *cobra.Command {
|
||||
var cfg comfy.GateConfig
|
||||
cmd := &cobra.Command{
|
||||
Use: "comfyui-gate [flags] [-- main.py flags]",
|
||||
Short: "Run ComfyUI as an engine: behind a key, on the loopback (started by the agent)",
|
||||
Hidden: true,
|
||||
Long: "Started by the node agent for a replica whose engine is comfyui. ComfyUI asks\n" +
|
||||
"nobody for a key, so it listens on the loopback only and this gate on the\n" +
|
||||
"replica's port, letting through, with the replica's key, only what the router\n" +
|
||||
"asks of it. It also lays the fetched weights out where ComfyUI looks for them.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
cfg.Key = os.Getenv(comfyKeyEnv)
|
||||
_ = os.Unsetenv(comfyKeyEnv)
|
||||
cfg.Args = args
|
||||
cfg.Log = logger()
|
||||
return comfy.RunGate(ctx, cfg)
|
||||
},
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&cfg.Listen, "listen", "", "the replica's address")
|
||||
f.StringVar(&cfg.Model, "model", "", "the model's name, for /v1/models")
|
||||
f.StringVar(&cfg.Weights, "weights", "", "the fetched weights: a file or a directory")
|
||||
f.StringVar(&cfg.State, "state", "", "the replica's own directory")
|
||||
f.StringVar(&cfg.Python, "python", "python3", "the interpreter of ComfyUI's environment")
|
||||
f.StringVar(&cfg.Dir, "comfyui", "", "ComfyUI's directory, where main.py is")
|
||||
for _, n := range []string{"listen", "weights", "state", "comfyui"} {
|
||||
_ = cmd.MarkFlagRequired(n)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
@@ -178,6 +178,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newJobsCmd(client, &asJSON),
|
||||
newMeshJobsCmd(client, &asJSON),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(client),
|
||||
@@ -331,6 +332,9 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
return "stopped"
|
||||
case m.Ready >= m.Min && m.Min > 0:
|
||||
return "ready"
|
||||
case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total:
|
||||
// Asked for, and up until it goes unused for long enough.
|
||||
return "ready (on demand)"
|
||||
case m.Total > m.Ready:
|
||||
return "starting"
|
||||
default:
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// newMeshJobsCmd is the 3D models clients asked for, as jobs: every owner's,
|
||||
// what each is doing, and a way to stop one. (ctl jobs is the control
|
||||
// plane's own work, serving and updating models.)
|
||||
func newMeshJobsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "3d",
|
||||
Short: "3D models clients asked for: the jobs, how far along each is, and stopping one",
|
||||
}
|
||||
var status string
|
||||
var limit int
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every owner's 3D jobs, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
q := url.Values{}
|
||||
if status != "" {
|
||||
q.Set("status", status)
|
||||
}
|
||||
if limit > 0 {
|
||||
q.Set("limit", fmt.Sprint(limit))
|
||||
}
|
||||
var jobs []adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs?"+q.Encode(), &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no 3D jobs")
|
||||
return nil
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "JOB\tMODEL\tSTATUS\tPROGRESS\tOWNER\tASKED")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s ago\n", j.ID, j.Model, j.Status, progressText(j),
|
||||
j.Owner, time.Since(j.CreatedAt).Round(time.Second))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
list.Flags().StringVar(&status, "status", "", "only jobs in this status: queued, running, completed, failed…")
|
||||
list.Flags().IntVar(&limit, "limit", 0, "the most jobs to show (50)")
|
||||
get := &cobra.Command{
|
||||
Use: "get JOB",
|
||||
Short: "One 3D job: its status, progress, result or error",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0]), &j); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(j)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "job %s\nmodel %s\nstatus %s\n", j.ID, j.Model, j.Status)
|
||||
if p := progressText(j); p != "" {
|
||||
fmt.Fprintf(out, "progress %s\n", p)
|
||||
}
|
||||
fmt.Fprintf(out, "owner %s (key %s)\nasked %s\n", j.Owner, j.KeyID, j.CreatedAt.UTC().Format(time.RFC3339))
|
||||
if j.FinishedAt != nil && j.StartedAt != nil {
|
||||
fmt.Fprintf(out, "took %s\n", j.FinishedAt.Sub(*j.StartedAt).Round(time.Second))
|
||||
}
|
||||
if j.OutputFileID != "" {
|
||||
fmt.Fprintf(out, "result %s (%s)\n", j.OutputFileID, j.ContentType)
|
||||
}
|
||||
if j.Error != "" {
|
||||
fmt.Fprintf(out, "error %s\n", j.Error)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cancel := &cobra.Command{
|
||||
Use: "cancel JOB",
|
||||
Short: "Stop a 3D job, in the engine's queue too",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().post(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0])+"/cancel", nil, &j); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", j.ID, j.Status)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list, get, cancel)
|
||||
return cmd
|
||||
}
|
||||
|
||||
// progressText is a job's progress as a person reads it.
|
||||
func progressText(j adminapi.JobView) string {
|
||||
if len(j.Progress) == 0 {
|
||||
return ""
|
||||
}
|
||||
var p struct {
|
||||
State string `json:"state"`
|
||||
Ahead int `json:"ahead"`
|
||||
NodeClass string `json:"node_class"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Percent int `json:"percent"`
|
||||
}
|
||||
if json.Unmarshal(j.Progress, &p) != nil {
|
||||
return ""
|
||||
}
|
||||
switch {
|
||||
case p.State == "queued" && p.Ahead > 0:
|
||||
return fmt.Sprintf("queued, %d ahead", p.Ahead)
|
||||
case p.State == "running" && p.Steps > 0:
|
||||
return fmt.Sprintf("running %d%%, %s step %d/%d", p.Percent, p.NodeClass, p.Step, p.Steps)
|
||||
case p.State == "running" && p.NodeClass != "":
|
||||
return "running " + p.NodeClass
|
||||
}
|
||||
return p.State
|
||||
}
|
||||
@@ -56,6 +56,12 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
fmt.Fprintf(out, "status %d\n", v.Status)
|
||||
fmt.Fprintf(out, "time %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
|
||||
fmt.Fprintf(out, "tokens %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
|
||||
if v.Images > 0 {
|
||||
fmt.Fprintf(out, "images %d\n", v.Images)
|
||||
}
|
||||
if v.Meshes > 0 {
|
||||
fmt.Fprintf(out, "meshes %d\n", v.Meshes)
|
||||
}
|
||||
if v.CacheHit {
|
||||
fmt.Fprintf(out, "cache answered from the project's answer cache: nothing generated, nothing charged\n")
|
||||
}
|
||||
|
||||
+11
-8
@@ -80,15 +80,18 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
// resolve is otherwise discovered by a replica failing to start
|
||||
// minutes later on a node — with an error naming the node rather
|
||||
// than the typo.
|
||||
// Every source the model's profiles load, not the model's alone.
|
||||
if !skipCheck {
|
||||
size, err := sourcecheck.Check(cmd.Context(), m.Source)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
|
||||
for _, src := range m.WeightSources() {
|
||||
size, err := sourcecheck.Check(cmd.Context(), src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(src); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(src, size))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -54,6 +54,7 @@ func Root() *cobra.Command {
|
||||
newMirrorCmd(),
|
||||
newBundleCmd(),
|
||||
newSimEngineCmd(),
|
||||
newComfyUIGateCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
|
||||
+14
-10
@@ -210,6 +210,10 @@ func newServerCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if st := rt.Store(); st != nil {
|
||||
// Each control plane raises the same events; one sends each.
|
||||
notifier.SetClaimer(st)
|
||||
}
|
||||
rt.SetNotifier(notifier)
|
||||
if gate != nil && notifier != nil {
|
||||
gate.SetJudgeAlarm(func(model string, err error) {
|
||||
@@ -1169,18 +1173,18 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
|
||||
if llamaVersion == "" {
|
||||
return nil
|
||||
}
|
||||
usesLlama := false
|
||||
// Each source a llama.cpp profile loads: its own, or the model's.
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
usesLlama = true
|
||||
if p.Engine != "llamacpp" {
|
||||
continue
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.SourceOf(p))
|
||||
if !ok || !gguf.IsGGUF(url) {
|
||||
continue
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.Source)
|
||||
if !usesLlama || !ok || !gguf.IsGGUF(url) {
|
||||
return nil
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) {
|
||||
{"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"},
|
||||
{"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"},
|
||||
{"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"},
|
||||
{"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"},
|
||||
{"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := modelStatus(tc.view); got != tc.want {
|
||||
|
||||
@@ -0,0 +1,717 @@
|
||||
// Package comfy has images made by ComfyUI: a workflow filled in with what
|
||||
// was asked for, queued, waited on, and its pictures fetched.
|
||||
//
|
||||
// ComfyUI speaks no OpenAI. A workflow is a graph of nodes that is queued with
|
||||
// POST /prompt, is done when GET /history/{id} lists it, and leaves its
|
||||
// pictures to be read one by one from GET /view. What the graph is depends on
|
||||
// the model — FLUX schnell is not FLUX dev, and neither is SDXL — so the graph
|
||||
// is a template the catalog names, with placeholders where a request's
|
||||
// prompt, size and seed go.
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed workflows/*.json
|
||||
var builtin embed.FS
|
||||
|
||||
// DefaultWorkflow is the template used when the catalog names none.
|
||||
const DefaultWorkflow = "flux-schnell"
|
||||
|
||||
// Builtins are the workflows that come with llmgw, by name.
|
||||
func Builtins() []string {
|
||||
entries, _ := builtin.ReadDir("workflows")
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Template is a workflow in ComfyUI's API format, with placeholders.
|
||||
//
|
||||
// A placeholder is a string value "{{name}}", or "{{name|default}}" for one a
|
||||
// request may leave out. One that is the whole value becomes the value
|
||||
// itself, a number where the node wants a number; one inside a longer string
|
||||
// is written into it. The names are those of Params.
|
||||
type Template struct {
|
||||
graph map[string]any
|
||||
}
|
||||
|
||||
// Load reads a template: a built-in's name, or the workflow's JSON itself.
|
||||
func Load(nameOrJSON string) (*Template, error) {
|
||||
s := strings.TrimSpace(nameOrJSON)
|
||||
if s == "" {
|
||||
s = DefaultWorkflow
|
||||
}
|
||||
raw := []byte(s)
|
||||
if !strings.HasPrefix(s, "{") {
|
||||
b, err := builtin.ReadFile("workflows/" + s + ".json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("no built-in workflow %q (there are %s)", s, strings.Join(Builtins(), ", "))
|
||||
}
|
||||
raw = b
|
||||
}
|
||||
var g map[string]any
|
||||
if err := json.Unmarshal(raw, &g); err != nil {
|
||||
return nil, fmt.Errorf("the workflow is not JSON: %w", err)
|
||||
}
|
||||
if len(g) == 0 {
|
||||
return nil, errors.New("the workflow has no nodes")
|
||||
}
|
||||
for id, n := range g {
|
||||
node, ok := n.(map[string]any)
|
||||
if !ok || node["class_type"] == nil {
|
||||
// The UI's own save format has a "nodes" list instead, and is
|
||||
// not something /prompt accepts.
|
||||
return nil, fmt.Errorf("node %q has no class_type: the workflow must be saved in API format", id)
|
||||
}
|
||||
}
|
||||
return &Template{graph: g}, nil
|
||||
}
|
||||
|
||||
// Params are what a request decides. A zero number is one not given, which
|
||||
// leaves it to the template's default.
|
||||
type Params struct {
|
||||
Checkpoint string
|
||||
Prompt string
|
||||
NegativePrompt string
|
||||
Width, Height int
|
||||
N int
|
||||
Seed int64
|
||||
Steps int
|
||||
Guidance float64
|
||||
// Image is the input picture, by the name Upload gave it in ComfyUI's
|
||||
// input directory: {{image}}.
|
||||
Image string
|
||||
// Extra are a workflow's own placeholders, by name: {{resolution}}.
|
||||
Extra map[string]any
|
||||
}
|
||||
|
||||
func (p Params) value(name string) (any, bool) {
|
||||
switch name {
|
||||
case "checkpoint":
|
||||
return p.Checkpoint, p.Checkpoint != ""
|
||||
case "prompt":
|
||||
return p.Prompt, true
|
||||
case "negative_prompt":
|
||||
return p.NegativePrompt, true
|
||||
case "width":
|
||||
return p.Width, p.Width > 0
|
||||
case "height":
|
||||
return p.Height, p.Height > 0
|
||||
case "n":
|
||||
return max(p.N, 1), true
|
||||
case "seed":
|
||||
return p.Seed, true
|
||||
case "steps":
|
||||
return p.Steps, p.Steps > 0
|
||||
case "guidance":
|
||||
return p.Guidance, p.Guidance > 0
|
||||
case "image":
|
||||
return p.Image, p.Image != ""
|
||||
}
|
||||
v, ok := p.Extra[name]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
// Fill is the workflow for one request.
|
||||
func (t *Template) Fill(p Params) (map[string]any, error) {
|
||||
var missing []string
|
||||
var fill func(v any) any
|
||||
fill = func(v any) any {
|
||||
switch v := v.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for k, x := range v {
|
||||
out[k] = fill(x)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, x := range v {
|
||||
out[i] = fill(x)
|
||||
}
|
||||
return out
|
||||
case string:
|
||||
return fillString(v, p, &missing)
|
||||
}
|
||||
return v
|
||||
}
|
||||
g := fill(t.graph).(map[string]any)
|
||||
if len(missing) > 0 {
|
||||
slices.Sort(missing)
|
||||
return nil, fmt.Errorf("the workflow wants %s, which the request did not give and the workflow has no default for",
|
||||
strings.Join(slices.Compact(missing), ", "))
|
||||
}
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func fillString(s string, p Params, missing *[]string) any {
|
||||
if !strings.Contains(s, "{{") {
|
||||
return s
|
||||
}
|
||||
resolve := func(ph string) (any, bool) {
|
||||
name, def, hasDef := strings.Cut(ph, "|")
|
||||
name = strings.TrimSpace(name)
|
||||
if v, ok := p.value(name); ok {
|
||||
return v, true
|
||||
}
|
||||
if hasDef {
|
||||
def = strings.TrimSpace(def)
|
||||
if f, err := strconv.ParseFloat(def, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
return def, true
|
||||
}
|
||||
*missing = append(*missing, name)
|
||||
return nil, false
|
||||
}
|
||||
// The whole value: what the node gets is the value, not its text.
|
||||
if strings.HasPrefix(s, "{{") && strings.HasSuffix(s, "}}") && strings.Count(s, "{{") == 1 {
|
||||
v, _ := resolve(s[2 : len(s)-2])
|
||||
return v
|
||||
}
|
||||
var b strings.Builder
|
||||
for {
|
||||
i := strings.Index(s, "{{")
|
||||
j := strings.Index(s, "}}")
|
||||
if i < 0 || j < i {
|
||||
b.WriteString(s)
|
||||
return b.String()
|
||||
}
|
||||
b.WriteString(s[:i])
|
||||
if v, ok := resolve(s[i+2 : j]); ok {
|
||||
fmt.Fprint(&b, v)
|
||||
}
|
||||
s = s[j+2:]
|
||||
}
|
||||
}
|
||||
|
||||
// RandomSeed is a seed for a request that named none. ComfyUI's are 64-bit,
|
||||
// but JSON numbers are read as doubles on the way: kept to 2^53.
|
||||
func RandomSeed() int64 {
|
||||
n, err := rand.Int(rand.Reader, big.NewInt(1<<53))
|
||||
if err != nil {
|
||||
return time.Now().UnixNano() & (1<<53 - 1)
|
||||
}
|
||||
return n.Int64()
|
||||
}
|
||||
|
||||
// Image is one file a workflow made: a picture, a mesh.
|
||||
type Image struct {
|
||||
Data []byte
|
||||
ContentType string
|
||||
// Name is the file's own, as ComfyUI saved it.
|
||||
Name string
|
||||
}
|
||||
|
||||
// Error is a failure, said as what it means for the request.
|
||||
type Error struct {
|
||||
Msg string
|
||||
// Retryable is a failure of this server, which another one might not
|
||||
// have: it could not be reached, or it answered 5xx.
|
||||
Retryable bool
|
||||
}
|
||||
|
||||
func (e *Error) Error() string { return e.Msg }
|
||||
|
||||
// Client queues workflows and waits for them.
|
||||
type Client struct {
|
||||
HTTP *http.Client
|
||||
// Poll is how often a queued workflow is asked after.
|
||||
Poll time.Duration
|
||||
}
|
||||
|
||||
// Generate runs a workflow on the ComfyUI at base and returns its pictures,
|
||||
// in the order of the nodes that made them. A context that ends takes the
|
||||
// workflow out of the queue, or interrupts it if it is running, so a client
|
||||
// that went away does not leave the card working for nobody.
|
||||
func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
|
||||
return c.GenerateWatch(ctx, base, apiKey, workflow, nil)
|
||||
}
|
||||
|
||||
// progressEvery is how often a workflow's progress is asked after, beside
|
||||
// its history: a few requests a second is for the end, not for the news.
|
||||
const progressEvery = time.Second
|
||||
|
||||
// GenerateWatch is Generate, saying how far along the workflow is as it
|
||||
// changes: queued and how many ahead, then the node running and its step.
|
||||
func (c *Client) GenerateWatch(ctx context.Context, base, apiKey string, workflow map[string]any,
|
||||
onProgress func(Progress)) ([]Image, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
client := make([]byte, 8)
|
||||
_, _ = rand.Read(client)
|
||||
body, err := json.Marshal(map[string]any{"prompt": workflow, "client_id": "llmgw-" + hex.EncodeToString(client)})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var queued struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Error json.RawMessage `json:"error"`
|
||||
NodeErrors json.RawMessage `json:"node_errors"`
|
||||
}
|
||||
status, err := c.do(ctx, http.MethodPost, base+"/prompt", apiKey, body, &queued)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status != http.StatusOK || queued.PromptID == "" {
|
||||
// A workflow ComfyUI will not run: a checkpoint it does not have, a
|
||||
// node it does not know. The catalog's to fix, not the client's, and
|
||||
// no other replica set up the same way would take it either.
|
||||
return nil, &Error{Msg: "ComfyUI refused the workflow: " + refusal(queued.Error, queued.NodeErrors)}
|
||||
}
|
||||
id := queued.PromptID
|
||||
|
||||
poll := c.Poll
|
||||
if poll <= 0 {
|
||||
poll = 250 * time.Millisecond
|
||||
}
|
||||
t := time.NewTicker(poll)
|
||||
defer t.Stop()
|
||||
var last Progress
|
||||
var asked time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.cancel(context.WithoutCancel(ctx), base, apiKey, id)
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
if onProgress != nil && time.Since(asked) >= progressEvery {
|
||||
asked = time.Now()
|
||||
if p, ok := c.progress(ctx, base, apiKey, id, workflow); ok && p != last {
|
||||
last = p
|
||||
onProgress(p)
|
||||
}
|
||||
}
|
||||
var hist map[string]historyEntry
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
continue // said by the select
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
h, ok := hist[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if h.Status.StatusStr == "error" {
|
||||
return nil, &Error{Msg: "the workflow failed in ComfyUI: " + h.failure()}
|
||||
}
|
||||
if !h.Status.Completed {
|
||||
continue
|
||||
}
|
||||
return c.fetch(ctx, base, apiKey, h)
|
||||
}
|
||||
}
|
||||
|
||||
type historyEntry struct {
|
||||
Status struct {
|
||||
StatusStr string `json:"status_str"`
|
||||
Completed bool `json:"completed"`
|
||||
Messages [][]json.RawMessage `json:"messages"`
|
||||
} `json:"status"`
|
||||
// Outputs are each node's, by kind: "images" for a picture, "3d" for
|
||||
// the mesh a 3D node saves, and whatever else a custom node names.
|
||||
Outputs map[string]map[string]json.RawMessage `json:"outputs"`
|
||||
}
|
||||
|
||||
// fileRef is a file an output names, to be read from /view.
|
||||
type fileRef struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
// failure is what ComfyUI said went wrong, from its execution_error.
|
||||
func (h historyEntry) failure() string {
|
||||
for _, m := range h.Status.Messages {
|
||||
if len(m) < 2 || string(m[0]) != `"execution_error"` {
|
||||
continue
|
||||
}
|
||||
var e struct {
|
||||
NodeType string `json:"node_type"`
|
||||
Message string `json:"exception_message"`
|
||||
}
|
||||
if json.Unmarshal(m[1], &e) == nil && e.Message != "" {
|
||||
return strings.TrimSpace(e.NodeType + ": " + strings.TrimSpace(e.Message))
|
||||
}
|
||||
}
|
||||
return "no reason given"
|
||||
}
|
||||
|
||||
func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry) ([]Image, error) {
|
||||
nodes := make([]string, 0, len(h.Outputs))
|
||||
for id := range h.Outputs {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
// Numerically where they are numbers, which is how ComfyUI names them.
|
||||
slices.SortFunc(nodes, func(a, b string) int {
|
||||
x, ea := strconv.Atoi(a)
|
||||
y, eb := strconv.Atoi(b)
|
||||
if ea == nil && eb == nil {
|
||||
return x - y
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
var out []Image
|
||||
for _, n := range nodes {
|
||||
kinds := make([]string, 0, len(h.Outputs[n]))
|
||||
for k := range h.Outputs[n] {
|
||||
kinds = append(kinds, k)
|
||||
}
|
||||
slices.Sort(kinds)
|
||||
for _, k := range kinds {
|
||||
// Any list of files, whatever the node calls it; text and
|
||||
// numbers a node reports beside them are not files.
|
||||
var refs []fileRef
|
||||
if json.Unmarshal(h.Outputs[n][k], &refs) != nil {
|
||||
continue
|
||||
}
|
||||
for _, f := range refs {
|
||||
if f.Filename == "" {
|
||||
continue
|
||||
}
|
||||
q := url.Values{"filename": {f.Filename}, "subfolder": {f.Subfolder}, "type": {f.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
img.Name = f.Filename
|
||||
if t := typeByName(f.Filename); t != "" && (img.ContentType == "" ||
|
||||
strings.HasPrefix(img.ContentType, "application/octet-stream") || strings.HasPrefix(img.ContentType, "text/plain")) {
|
||||
img.ContentType = t
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, &Error{Msg: "the workflow finished without a file: it has no node that saves one"}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// cancel takes a workflow out of the queue, or stops it if it is the one
|
||||
// running. Best effort: the server may be gone, which also stops it.
|
||||
func (c *Client) cancel(ctx context.Context, base, apiKey, id string) {
|
||||
ctx, done := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer done()
|
||||
body, _ := json.Marshal(map[string]any{"delete": []string{id}})
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/queue", apiKey, body, nil)
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return
|
||||
}
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == strconv.Quote(id) {
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/interrupt", apiKey, []byte("{}"), nil)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
return http.DefaultClient
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method, u, apiKey string, body []byte) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return nil, &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// do sends a request and reads a JSON answer into out. A 5xx is an error;
|
||||
// any other status is returned with whatever body could be read.
|
||||
func (c *Client) do(ctx context.Context, method, u, apiKey string, body []byte, out any) (int, error) {
|
||||
resp, err := c.request(ctx, method, u, apiKey, body)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
|
||||
if err != nil {
|
||||
return 0, &Error{Msg: "reading ComfyUI's answer: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
return resp.StatusCode, &Error{Msg: fmt.Sprintf("ComfyUI answered %d: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
if out != nil && len(b) > 0 {
|
||||
if err := json.Unmarshal(b, out); err != nil && resp.StatusCode == http.StatusOK {
|
||||
return resp.StatusCode, &Error{Msg: "ComfyUI's answer is not JSON: " + snippet(b)}
|
||||
}
|
||||
}
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
|
||||
resp, err := c.request(ctx, http.MethodGet, u, apiKey, nil)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, maxFile))
|
||||
if err != nil {
|
||||
return Image{}, &Error{Msg: "reading a file from ComfyUI: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a file it made", resp.StatusCode)}
|
||||
}
|
||||
ct := resp.Header.Get("Content-Type")
|
||||
if ct == "" || ct == "application/octet-stream" {
|
||||
ct = http.DetectContentType(b)
|
||||
}
|
||||
if ct == "image/png" {
|
||||
b = withoutPNGText(b)
|
||||
}
|
||||
return Image{Data: b, ContentType: ct}, nil
|
||||
}
|
||||
|
||||
// withoutPNGText is a PNG without its text chunks. ComfyUI writes the whole
|
||||
// workflow into every picture it saves — the prompt, and the checkpoint and
|
||||
// nodes behind it — and what a picture says about how it was made is the
|
||||
// operator's to publish, not something to hand every client. A file that is
|
||||
// not a well-formed PNG is returned as it came.
|
||||
func withoutPNGText(b []byte) []byte {
|
||||
const sig = "\x89PNG\r\n\x1a\n"
|
||||
if len(b) < len(sig) || string(b[:len(sig)]) != sig {
|
||||
return b
|
||||
}
|
||||
out := make([]byte, 0, len(b))
|
||||
out = append(out, b[:len(sig)]...)
|
||||
for i := len(sig); i < len(b); {
|
||||
if i+12 > len(b) {
|
||||
return b
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(b[i:]))
|
||||
end := i + 12 + n
|
||||
if n < 0 || end > len(b) {
|
||||
return b
|
||||
}
|
||||
switch string(b[i+4 : i+8]) {
|
||||
case "tEXt", "iTXt", "zTXt":
|
||||
default:
|
||||
out = append(out, b[i:end]...)
|
||||
}
|
||||
i = end
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// refusal is what /prompt said when it would not queue a workflow.
|
||||
func refusal(e, nodes json.RawMessage) string {
|
||||
var top struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
}
|
||||
msg := snippet(e)
|
||||
if json.Unmarshal(e, &top) == nil && top.Message != "" {
|
||||
msg = top.Message
|
||||
if top.Details != "" {
|
||||
msg += ": " + top.Details
|
||||
}
|
||||
}
|
||||
var perNode map[string]struct {
|
||||
ClassType string `json:"class_type"`
|
||||
Errors []struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
} `json:"errors"`
|
||||
}
|
||||
if json.Unmarshal(nodes, &perNode) == nil {
|
||||
var parts []string
|
||||
for _, n := range perNode {
|
||||
for _, er := range n.Errors {
|
||||
parts = append(parts, strings.TrimSpace(n.ClassType+": "+er.Message+" "+er.Details))
|
||||
}
|
||||
}
|
||||
slices.Sort(parts)
|
||||
if len(parts) > 0 {
|
||||
msg += " (" + strings.Join(parts, "; ") + ")"
|
||||
}
|
||||
}
|
||||
if msg == "" {
|
||||
msg = "no reason given"
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
func snippet(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// maxFile bounds one file read from ComfyUI: a mesh with its textures runs
|
||||
// to tens of megabytes, a picture to a few.
|
||||
const maxFile = 256 << 20
|
||||
|
||||
// typeByName is a file's type by its extension, for those a server sends
|
||||
// as octet-stream: the meshes a 3D node saves.
|
||||
func typeByName(name string) string {
|
||||
switch strings.ToLower(path.Ext(name)) {
|
||||
case ".glb":
|
||||
return "model/gltf-binary"
|
||||
case ".gltf":
|
||||
return "model/gltf+json"
|
||||
case ".obj":
|
||||
return "model/obj"
|
||||
case ".stl":
|
||||
return "model/stl"
|
||||
case ".ply":
|
||||
return "application/ply"
|
||||
case ".png":
|
||||
return "image/png"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Upload puts a picture in ComfyUI's input directory, where a LoadImage node
|
||||
// reads it, and returns the name it was given there.
|
||||
func (c *Client) Upload(ctx context.Context, base, apiKey, name string, data []byte) (string, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
var body bytes.Buffer
|
||||
mw := multipart.NewWriter(&body)
|
||||
fw, err := mw.CreateFormFile("image", name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := fw.Write(data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_ = mw.WriteField("type", "input")
|
||||
_ = mw.WriteField("overwrite", "true")
|
||||
if err := mw.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/upload/image", &body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return "", ctx.Err()
|
||||
}
|
||||
return "", &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode >= 500 {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI answered %d to the picture: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
var out struct {
|
||||
Name string `json:"name"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK || json.Unmarshal(b, &out) != nil || out.Name == "" {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI did not take the picture (%d): %s", resp.StatusCode, snippet(b))}
|
||||
}
|
||||
if out.Subfolder != "" {
|
||||
return out.Subfolder + "/" + out.Name, nil
|
||||
}
|
||||
return out.Name, nil
|
||||
}
|
||||
|
||||
// progress is a workflow's: its place in the queue, from ComfyUI, and the
|
||||
// node running and its step, from the gate when there is one — a ComfyUI
|
||||
// outside the cluster has no gate, and says only where it is queued.
|
||||
func (c *Client) progress(ctx context.Context, base, apiKey, id string, workflow map[string]any) (Progress, bool) {
|
||||
p := Progress{PromptID: id, State: "queued"}
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
Pending [][]json.RawMessage `json:"queue_pending"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return p, false
|
||||
}
|
||||
quoted := strconv.Quote(id)
|
||||
queued := false
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == quoted {
|
||||
p.State = "running"
|
||||
}
|
||||
}
|
||||
if p.State != "running" {
|
||||
// Pending in the order ComfyUI will take them: by number.
|
||||
mine := -1.0
|
||||
var numbers []float64
|
||||
for _, r := range q.Pending {
|
||||
if len(r) < 2 {
|
||||
continue
|
||||
}
|
||||
var n float64
|
||||
_ = json.Unmarshal(r[0], &n)
|
||||
if string(r[1]) == quoted {
|
||||
mine, queued = n, true
|
||||
continue
|
||||
}
|
||||
numbers = append(numbers, n)
|
||||
}
|
||||
if queued {
|
||||
p.Ahead = len(q.Running)
|
||||
for _, n := range numbers {
|
||||
if n < mine {
|
||||
p.Ahead++
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
var g Progress
|
||||
if st, err := c.do(ctx, http.MethodGet, base+"/progress/"+url.PathEscape(id), apiKey, nil, &g); err == nil &&
|
||||
st == http.StatusOK && g.State != "" {
|
||||
p = g
|
||||
p.PromptID = id
|
||||
if node, ok := workflow[p.Node].(map[string]any); ok {
|
||||
p.NodeClass, _ = node["class_type"].(string)
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
for _, name := range Builtins() {
|
||||
tpl, err := Load(name)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "flux.safetensors", Prompt: `a "red" fox`, Width: 768, Height: 512, N: 2, Seed: 42})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
b, _ := json.Marshal(g)
|
||||
s := string(b)
|
||||
for _, want := range []string{`"ckpt_name":"flux.safetensors"`, `"text":"a \"red\" fox"`, `"width":768`,
|
||||
`"height":512`, `"batch_size":2`, `"seed":42`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("%s: %s not in %s", name, want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "{{") {
|
||||
t.Errorf("%s: a placeholder was left: %s", name, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
|
||||
tpl, _ := Load("flux-schnell")
|
||||
g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 4.0 {
|
||||
t.Errorf("default steps = %v, want 4", steps)
|
||||
}
|
||||
g, _ = tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512, Steps: 8})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 8 {
|
||||
t.Errorf("steps = %v, want 8", steps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPlaceholderWithoutAValueOrADefaultIsSaid(t *testing.T) {
|
||||
tpl, err := Load(`{"1":{"class_type":"X","inputs":{"a":"{{checkpoint}}","b":"size {{width}}x{{height}}"}}}`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tpl.Fill(Params{Width: 5}); err == nil || !strings.Contains(err.Error(), "checkpoint, height") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "c", Width: 5, Height: 6})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b := g["1"].(map[string]any)["inputs"].(map[string]any)["b"]; b != "size 5x6" {
|
||||
t.Errorf("b = %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowInTheUIsFormatIsRefused(t *testing.T) {
|
||||
if _, err := Load(`{"nodes":[{"id":1}],"links":[]}`); err == nil || !strings.Contains(err.Error(), "API format") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
if _, err := Load("sdxl-turbo"); err == nil {
|
||||
t.Fatal("an unknown built-in was loaded")
|
||||
}
|
||||
}
|
||||
|
||||
// fakeComfy is just enough of ComfyUI: /prompt queues, /history answers
|
||||
// after a few asks, /view serves a picture.
|
||||
type fakeComfy struct {
|
||||
mu sync.Mutex
|
||||
asked int
|
||||
fail string
|
||||
refuse bool
|
||||
deleted []string
|
||||
queued map[string]any
|
||||
readyAt int
|
||||
srv *httptest.Server
|
||||
interrupted bool
|
||||
}
|
||||
|
||||
func newFakeComfy(t *testing.T) *fakeComfy {
|
||||
f := &fakeComfy{readyAt: 2}
|
||||
f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
switch {
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/prompt":
|
||||
if f.refuse {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
fmt.Fprint(w, `{"error":{"message":"Prompt outputs failed validation","details":""},"node_errors":{"1":{"class_type":"CheckpointLoaderSimple","errors":[{"message":"Value not in list","details":"ckpt_name: 'x' not in []"}]}}}`)
|
||||
return
|
||||
}
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.queued = b.Prompt
|
||||
fmt.Fprint(w, `{"prompt_id":"p1","number":1,"node_errors":{}}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
f.asked++
|
||||
if f.asked < f.readyAt {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
if f.fail != "" {
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"error","completed":false,"messages":[["execution_start",{}],["execution_error",{"node_type":"KSampler","exception_message":%q}]]},"outputs":{}}}`, f.fail)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true,"messages":[]},"outputs":{"7":{"images":[{"filename":"a.png","subfolder":"","type":"temp"},{"filename":"b.png","subfolder":"","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "PNG:"+r.URL.Query().Get("filename"))
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost:
|
||||
var b struct {
|
||||
Delete []string `json:"delete"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.deleted = append(f.deleted, b.Delete...)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"p1",{},{},[]]],"queue_pending":[]}`)
|
||||
case r.URL.Path == "/interrupt":
|
||||
f.interrupted = true
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(f.srv.Close)
|
||||
return f
|
||||
}
|
||||
|
||||
func TestGenerateWaitsForTheWorkflowAndFetchesItsPictures(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
images, err := c.Generate(t.Context(), f.srv.URL, "", map[string]any{"1": map[string]any{"class_type": "X"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(images) != 2 || string(images[0].Data) != "PNG:a.png" || images[1].ContentType != "image/png" {
|
||||
t.Fatalf("images = %+v", images)
|
||||
}
|
||||
if f.queued["1"] == nil {
|
||||
t.Error("the workflow was not what was queued")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowThatFailsSaysWhy(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.fail = "CUDA out of memory"
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if err == nil || !strings.Contains(err.Error(), "KSampler: CUDA out of memory") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowComfyUIRefusesIsNotRetried(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.refuse = true
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
ce, ok := err.(*Error)
|
||||
if !ok || ce.Retryable || !strings.Contains(ce.Msg, "Value not in list") {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIThatCannotBeReachedIsRetryable(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.srv.Close()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if ce, ok := err.(*Error); !ok || !ce.Retryable {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientThatLeavesTakesItsWorkflowOffTheCard(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.readyAt = 1 << 30
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(ctx, f.srv.URL, "", map[string]any{})
|
||||
if err == nil {
|
||||
t.Fatal("no error")
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if len(f.deleted) != 1 || f.deleted[0] != "p1" || !f.interrupted {
|
||||
t.Errorf("deleted %v, interrupted %v", f.deleted, f.interrupted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
|
||||
chunk := func(kind, data string) []byte {
|
||||
b := make([]byte, 0, 12+len(data))
|
||||
b = binary.BigEndian.AppendUint32(b, uint32(len(data)))
|
||||
b = append(b, kind...)
|
||||
b = append(b, data...)
|
||||
return append(b, 0, 0, 0, 0) // a CRC nobody here checks
|
||||
}
|
||||
png := []byte("\x89PNG\r\n\x1a\n")
|
||||
png = append(png, chunk("IHDR", "0123456789abc")...)
|
||||
png = append(png, chunk("tEXt", "prompt\x00{\"1\":...}")...)
|
||||
png = append(png, chunk("IDAT", "pixels")...)
|
||||
png = append(png, chunk("iTXt", "workflow\x00...")...)
|
||||
png = append(png, chunk("IEND", "")...)
|
||||
got := string(withoutPNGText(png))
|
||||
if strings.Contains(got, "prompt") || strings.Contains(got, "workflow") {
|
||||
t.Errorf("text left in: %q", got)
|
||||
}
|
||||
for _, kind := range []string{"IHDR", "IDAT", "pixels", "IEND"} {
|
||||
if !strings.Contains(got, kind) {
|
||||
t.Errorf("%s gone: %q", kind, got)
|
||||
}
|
||||
}
|
||||
if broken := png[:30]; string(withoutPNGText(broken)) != string(broken) {
|
||||
t.Error("a truncated PNG was changed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The gate is how a node runs ComfyUI as one of llmgw's engines.
|
||||
//
|
||||
// ComfyUI asks nobody for a key: whoever reaches its port can queue work,
|
||||
// upload files and install nodes. Every other engine here demands the key
|
||||
// the control plane made for its replica, so ComfyUI is put behind a gate
|
||||
// that does: ComfyUI listens on the loopback only, and the gate on the
|
||||
// replica's port, letting through only the requests the router makes. The
|
||||
// gate also answers what the agent asks every engine — /health once
|
||||
// ComfyUI is up, and /v1/models naming the model — and lays the weights out
|
||||
// where ComfyUI looks for them.
|
||||
//
|
||||
// ComfyUI runs as the gate's child, in its process group, so stopping the
|
||||
// replica stops both, the card memory ComfyUI holds is counted as the
|
||||
// replica's, and either one exiting ends the other.
|
||||
|
||||
// GateConfig is one replica's gate.
|
||||
type GateConfig struct {
|
||||
// Listen is the replica's address, the one the router is given.
|
||||
Listen string
|
||||
// Model is the name the agent asks /v1/models for.
|
||||
Model string
|
||||
// Key is what every request must carry as a bearer token; empty lets
|
||||
// every request through, as an engine started without one does.
|
||||
Key string
|
||||
// Weights is the model as the agent fetched it: a file, or a directory
|
||||
// of files.
|
||||
Weights string
|
||||
// State is the replica's own directory: ComfyUI's base directory, with
|
||||
// its output, temp and user directories, and the models it is shown.
|
||||
State string
|
||||
// Python runs ComfyUI's main.py, found in Dir.
|
||||
Python, Dir string
|
||||
// Args are passed on to main.py, after the gate's own.
|
||||
Args []string
|
||||
// Output is where ComfyUI's output goes.
|
||||
Output io.Writer
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// The requests the router makes of ComfyUI: queueing, waiting, fetching and
|
||||
// cancelling. Nothing that uploads, installs or changes ComfyUI.
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info",
|
||||
// The input picture of a 3D model: into ComfyUI's input directory,
|
||||
// which is the replica's own.
|
||||
"/upload/image"}
|
||||
|
||||
// modelFolders are the kinds of file ComfyUI's loaders look for. Each is
|
||||
// shown the same directory: a workflow names its files, so a file being
|
||||
// listed where it is not wanted costs nothing.
|
||||
var modelFolders = []string{"checkpoints", "diffusion_models", "text_encoders", "vae", "loras",
|
||||
"clip_vision", "controlnet", "upscale_models", "embeddings", "style_models"}
|
||||
|
||||
// RunGate lays out the replica, starts ComfyUI and serves the gate until
|
||||
// ctx ends or ComfyUI exits.
|
||||
func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
log := cfg.Log
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
state, err := filepath.Abs(cfg.State)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What ComfyUI expects to find in its base directory, and dies
|
||||
// without: custom_nodes is listed before anything else is done.
|
||||
for _, d := range []string{"custom_nodes", "input", "output", "temp", "user"} {
|
||||
if err := os.MkdirAll(filepath.Join(state, d), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
}
|
||||
nodes, err := parseNodes(os.Getenv(NodesEnv))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
site, err := installNodes(ctx, state, cfg.Python, nodes, findUV(), func(s string) { fmt.Fprintln(out, "[gate] "+s) })
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
return err
|
||||
}
|
||||
extra := filepath.Join(state, "extra_model_paths.yaml")
|
||||
if err := os.WriteFile(extra, []byte(extraModelPaths(models)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return err
|
||||
}
|
||||
|
||||
inner, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
args := append([]string{filepath.Join(cfg.Dir, "main.py"),
|
||||
"--listen", "127.0.0.1", "--port", strconv.Itoa(inner),
|
||||
"--base-directory", state, "--extra-model-paths-config", extra,
|
||||
"--disable-auto-launch"}, cfg.Args...)
|
||||
child := exec.Command(cfg.Python, args...) //nolint:gosec // the agent's own configuration
|
||||
child.Dir = cfg.Dir
|
||||
child.Env = os.Environ()
|
||||
if site != "" {
|
||||
pp := site
|
||||
if old := os.Getenv("PYTHONPATH"); old != "" {
|
||||
pp += string(os.PathListSeparator) + old
|
||||
}
|
||||
child.Env = append(child.Env, "PYTHONPATH="+pp)
|
||||
}
|
||||
// ComfyUI writes to a pipe of the gate's, never to what the gate was
|
||||
// given. That is the agent's, and an agent that restarts and adopts the
|
||||
// replica leaves it with no reader: ComfyUI's next progress bar would
|
||||
// fail with a broken pipe, and take the workflow it belongs to with it.
|
||||
// The gate reads the pipe for as long as ComfyUI writes, and passes on
|
||||
// what it can.
|
||||
signal.Ignore(syscall.SIGPIPE)
|
||||
pr, err := child.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
child.Stderr = child.Stdout
|
||||
if err := child.Start(); err != nil {
|
||||
return fmt.Errorf("starting ComfyUI: %w", err)
|
||||
}
|
||||
relayed := make(chan struct{})
|
||||
go func() { relay(pr, out); close(relayed) }()
|
||||
exited := make(chan error, 1)
|
||||
go func() {
|
||||
// Wait closes the pipe; what is in it is read first.
|
||||
<-relayed
|
||||
exited <- child.Wait()
|
||||
}()
|
||||
|
||||
upstream := &url.URL{Scheme: "http", Host: net.JoinHostPort("127.0.0.1", strconv.Itoa(inner))}
|
||||
// The workflows' progress, heard on ComfyUI's websocket under an id of
|
||||
// the gate's own.
|
||||
book := newProgressBook()
|
||||
cid := make([]byte, 8)
|
||||
_, _ = rand.Read(cid)
|
||||
clientID := "llmgw-gate-" + hex.EncodeToString(cid)
|
||||
go book.watch(ctx, upstream, clientID, log)
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: newGate(upstream, cfg.Key, cfg.Model, book, clientID),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
go func() { served <- srv.ListenAndServe() }()
|
||||
log.Info("ComfyUI gate up", "listen", cfg.Listen, "comfyui", upstream.Host, "weights", cfg.Weights)
|
||||
|
||||
stop := func() {
|
||||
_ = child.Process.Signal(syscall.SIGTERM)
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(20 * time.Second):
|
||||
_ = child.Process.Kill()
|
||||
<-exited
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdown(srv)
|
||||
stop()
|
||||
return nil
|
||||
case err := <-exited:
|
||||
shutdown(srv)
|
||||
if err == nil {
|
||||
err = errors.New("exited")
|
||||
}
|
||||
return fmt.Errorf("ComfyUI %w", err)
|
||||
case err := <-served:
|
||||
stop()
|
||||
return fmt.Errorf("the gate stopped serving: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
func shutdown(srv *http.Server) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = srv.Shutdown(ctx)
|
||||
}
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
return newGate(upstream, key, model, newProgressBook(), "")
|
||||
}
|
||||
|
||||
// newGate is the gate with the book of its workflows' progress, kept under
|
||||
// clientID; an empty one leaves the workflows' own client ids alone.
|
||||
func newGate(upstream *url.URL, key, model string, book *progressBook, clientID string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
base(r)
|
||||
r.Header.Del("Authorization")
|
||||
// ComfyUI refuses a request whose Origin is not its own host; the
|
||||
// router sends none, and nothing else gets this far.
|
||||
r.Header.Del("Origin")
|
||||
r.Host = upstream.Host
|
||||
}
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
up := func(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, upstream.String()+"/system_stats", nil)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("ComfyUI answered %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if key != "" {
|
||||
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
|
||||
if !ok || subtle.ConstantTimeCompare([]byte(got), []byte(key)) != 1 {
|
||||
http.Error(w, "this engine wants its key", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
}
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
case "/v1/models":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
return
|
||||
}
|
||||
if id, ok := strings.CutPrefix(r.URL.Path, "/progress/"); ok && r.Method == http.MethodGet {
|
||||
p, found := book.get(id)
|
||||
if !found {
|
||||
http.Error(w, "no word of that workflow", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(p)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/prompt" && r.Method == http.MethodPost && clientID != "" {
|
||||
book.promptThroughGate(proxy, clientID)(w, r)
|
||||
return
|
||||
}
|
||||
for _, p := range gatePaths {
|
||||
if r.URL.Path == p || strings.HasSuffix(p, "/") && strings.HasPrefix(r.URL.Path, p) {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// cachedName is a file the agent fetched by itself: its cache names it by
|
||||
// the source's hash, then the name it had.
|
||||
var cachedName = regexp.MustCompile(`^[0-9a-f]{16}-`)
|
||||
|
||||
// FileName is what a weights file is called where ComfyUI sees it: its own
|
||||
// name, without the prefix the agent's cache gives it.
|
||||
func FileName(path string) string {
|
||||
return cachedName.ReplaceAllString(filepath.Base(path), "")
|
||||
}
|
||||
|
||||
// LinkWeights shows ComfyUI the weights: a link in dir to the file, or to
|
||||
// every file of the directory. Links made before are taken away first, so
|
||||
// a replica started again on other weights sees only those.
|
||||
func LinkWeights(weights, dir string) error {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
old, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range old {
|
||||
if e.Type()&os.ModeSymlink != 0 {
|
||||
_ = os.Remove(filepath.Join(dir, e.Name()))
|
||||
}
|
||||
}
|
||||
weights, err = filepath.Abs(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
info, err := os.Stat(weights)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the weights: %w", err)
|
||||
}
|
||||
files := []string{weights}
|
||||
if info.IsDir() {
|
||||
entries, err := os.ReadDir(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files = files[:0]
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && !strings.HasPrefix(e.Name(), ".") {
|
||||
files = append(files, filepath.Join(weights, e.Name()))
|
||||
}
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return fmt.Errorf("the weights directory %s has no files", weights)
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if err := os.Symlink(f, filepath.Join(dir, FileName(f))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extraModelPaths(models string) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "llmgw:\n base_path: %q\n is_default: true\n", models)
|
||||
for _, f := range modelFolders {
|
||||
fmt.Fprintf(&b, " %s: .\n", f)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// relay copies r to w for as long as r has anything, and goes on reading
|
||||
// after w fails: a writer that stops reading must not stop the one writing.
|
||||
func relay(r io.Reader, w io.Writer) {
|
||||
buf := make([]byte, 32<<10)
|
||||
broken := false
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
if n > 0 && !broken {
|
||||
if _, werr := w.Write(buf[:n]); werr != nil {
|
||||
broken = true
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheGateWantsTheKeyAndLetsThroughOnlyTheRoutersRequests(t *testing.T) {
|
||||
var got []string
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = append(got, r.Method+" "+r.URL.Path+" auth="+r.Header.Get("Authorization"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
gate := httptest.NewServer(NewGate(u, "k", "flux"))
|
||||
defer gate.Close()
|
||||
|
||||
do := func(method, path, key string) (int, string) {
|
||||
req, _ := http.NewRequest(method, gate.URL+path, strings.NewReader("{}"))
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", ""); code != http.StatusUnauthorized {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", "wrong"); code != http.StatusUnauthorized {
|
||||
t.Errorf("wrong key: %d", code)
|
||||
}
|
||||
for _, p := range []string{"/upload/mask", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusNotFound {
|
||||
t.Errorf("%s: %d, want it refused", p, code)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt", "/upload/image"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusOK {
|
||||
t.Errorf("%s: %d", p, code)
|
||||
}
|
||||
}
|
||||
if code, body := do("GET", "/v1/models", "k"); code != 200 || !strings.Contains(body, `"id":"flux"`) {
|
||||
t.Errorf("/v1/models: %d %s", code, body)
|
||||
}
|
||||
if code, _ := do("GET", "/health", "k"); code != 200 {
|
||||
t.Errorf("/health: %d", code)
|
||||
}
|
||||
for _, g := range got {
|
||||
if strings.Contains(g, "auth=Bearer") {
|
||||
t.Errorf("the key reached ComfyUI: %s", g)
|
||||
}
|
||||
}
|
||||
comfy.Close()
|
||||
if code, _ := do("GET", "/health", "k"); code != http.StatusServiceUnavailable {
|
||||
t.Errorf("/health with ComfyUI down: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheWeightsAreShownByTheirOwnNames(t *testing.T) {
|
||||
cache := t.TempDir()
|
||||
one := filepath.Join(cache, "0123456789abcdef-flux1-schnell-fp8.safetensors")
|
||||
_ = os.WriteFile(one, []byte("w"), 0o644)
|
||||
models := filepath.Join(t.TempDir(), "models")
|
||||
if err := LinkWeights(one, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(models, "flux1-schnell-fp8.safetensors")); err != nil || target != one {
|
||||
t.Fatalf("%q %v", target, err)
|
||||
}
|
||||
|
||||
set := t.TempDir()
|
||||
for _, n := range []string{"flux1-dev.safetensors", "ae.safetensors", "t5xxl_fp8.safetensors", ".complete"} {
|
||||
_ = os.WriteFile(filepath.Join(set, n), []byte("w"), 0o644)
|
||||
}
|
||||
if err := LinkWeights(set, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(models)
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
if strings.Join(names, ",") != "ae.safetensors,flux1-dev.safetensors,t5xxl_fp8.safetensors" {
|
||||
t.Errorf("shown %v: the old weights' link should be gone and the marker not shown", names)
|
||||
}
|
||||
if !strings.Contains(extraModelPaths(models), "checkpoints: .") {
|
||||
t.Error(extraModelPaths(models))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMain lets the test binary stand in for ComfyUI: started by the gate
|
||||
// as `python main.py --listen … --port …`, it answers /system_stats and
|
||||
// says what it was given.
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI") == "1" {
|
||||
fakeComfyUI()
|
||||
return
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func fakeComfyUI() {
|
||||
var port, base string
|
||||
for i, a := range os.Args {
|
||||
switch a {
|
||||
case "--port":
|
||||
port = os.Args[i+1]
|
||||
case "--base-directory":
|
||||
base = os.Args[i+1]
|
||||
}
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(base, "args"), []byte(strings.Join(os.Args[1:], " ")+"\nkey="+os.Getenv("LLMGW_ENGINE_KEY")), 0o644)
|
||||
http.HandleFunc("/system_stats", func(w http.ResponseWriter, _ *http.Request) {
|
||||
// A progress bar's worth of output with every answer: a write that
|
||||
// failed would end this process, as it fails ComfyUI's workflow.
|
||||
if _, err := os.Stdout.Write([]byte(strings.Repeat("progress ", 20000) + "\n")); err != nil {
|
||||
os.Exit(4)
|
||||
}
|
||||
_, _ = io.WriteString(w, "{}")
|
||||
})
|
||||
http.HandleFunc("/quit", func(w http.ResponseWriter, _ *http.Request) { os.Exit(3) })
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI_EXIT") == "1" {
|
||||
time.AfterFunc(200*time.Millisecond, func() { os.Exit(3) })
|
||||
}
|
||||
_ = http.ListenAndServe("127.0.0.1:"+port, nil)
|
||||
}
|
||||
|
||||
func TestTheGateRunsComfyUIAndEndsWithIt(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "flux.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
state := t.TempDir()
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Model: "flux", Weights: weights, State: state,
|
||||
Python: os.Args[0], Dir: t.TempDir(), Args: []string{"--disable-smart-memory"}, Output: io.Discard})
|
||||
}()
|
||||
get := func(path string) int {
|
||||
resp, err := http.Get("http://" + listen + path)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for get("/health") != http.StatusOK {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the gate never said ComfyUI was up")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "args"))
|
||||
for _, want := range []string{"main.py --listen 127.0.0.1 --port", "--base-directory " + state,
|
||||
"--extra-model-paths-config " + filepath.Join(state, "extra_model_paths.yaml"), "--disable-smart-memory"} {
|
||||
if !strings.Contains(string(b), want) {
|
||||
t.Errorf("%q not in what ComfyUI was given: %s", want, b)
|
||||
}
|
||||
}
|
||||
if _, err := os.Lstat(filepath.Join(state, "models", "flux.safetensors")); err != nil {
|
||||
t.Error("the weights were not shown to ComfyUI:", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(state, "custom_nodes")); err != nil {
|
||||
t.Error("ComfyUI lists custom_nodes in its base directory first thing, and it is not there:", err)
|
||||
}
|
||||
if get("/quit") != http.StatusNotFound {
|
||||
t.Error("a path the router never asks for was let through")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Errorf("stopped, the gate said %v", err)
|
||||
}
|
||||
case <-time.After(25 * time.Second):
|
||||
t.Fatal("the gate did not stop")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIExitingEndsTheGate(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI_EXIT", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
err := RunGate(t.Context(), GateConfig{Listen: "127.0.0.1:0", Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: io.Discard})
|
||||
if err == nil || !strings.Contains(err.Error(), "exit status 3") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type brokenWriter struct{}
|
||||
|
||||
func (brokenWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") }
|
||||
|
||||
func TestComfyUIKeepsWritingWhenWhatTheGateWritesToIsGone(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: brokenWriter{}})
|
||||
}()
|
||||
ok := 0
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for ok < 20 && time.Now().Before(deadline) {
|
||||
resp, err := http.Get("http://" + listen + "/health")
|
||||
if err == nil {
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
ok++
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Fatalf("the gate ended: %v", err)
|
||||
default:
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if ok < 20 {
|
||||
t.Fatalf("ComfyUI answered %d times; output it could not hand on stopped it", ok)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Custom nodes, put in a replica's own base directory at the commit the
|
||||
// catalog names, their packages installed beside the replica rather than
|
||||
// into ComfyUI's environment: one model's node cannot break another
|
||||
// model's replicas, or the pictures every replica makes.
|
||||
|
||||
// Node is a custom node as the catalog gives it.
|
||||
type Node struct {
|
||||
Repo string `json:"repo"`
|
||||
Commit string `json:"commit"`
|
||||
Requirements []string `json:"requirements,omitempty"`
|
||||
}
|
||||
|
||||
// NodesEnv is where the agent hands the gate a model's custom nodes.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var nodeDir = regexp.MustCompile(`[^A-Za-z0-9._-]`)
|
||||
|
||||
// installNodes puts each node in state/custom_nodes at its commit and its
|
||||
// packages in state/site, and returns the PYTHONPATH ComfyUI is started
|
||||
// with: empty when no node wants a package. Run as the engines' user, in
|
||||
// the replica's own directory, before ComfyUI starts.
|
||||
func installNodes(ctx context.Context, state, python string, nodes []Node, uv string, out func(string)) (string, error) {
|
||||
if len(nodes) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
var reqs []string
|
||||
for _, n := range nodes {
|
||||
name := nodeDir.ReplaceAllString(strings.TrimSuffix(path.Base(n.Repo), ".git"), "_")
|
||||
dir := filepath.Join(state, "custom_nodes", name)
|
||||
if head, err := gitOutput(ctx, dir, "rev-parse", "HEAD"); err == nil && head == n.Commit {
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
continue
|
||||
}
|
||||
out(fmt.Sprintf("custom node %s at %s", n.Repo, n.Commit[:12]))
|
||||
_ = os.RemoveAll(dir)
|
||||
for _, args := range [][]string{
|
||||
{"init", "-q", dir},
|
||||
{"-C", dir, "remote", "add", "origin", n.Repo},
|
||||
{"-C", dir, "fetch", "-q", "--depth", "1", "origin", n.Commit},
|
||||
{"-C", dir, "checkout", "-q", "FETCH_HEAD"},
|
||||
} {
|
||||
if b, err := exec.CommandContext(ctx, "git", args...).CombinedOutput(); err != nil { //nolint:gosec // the catalog's repository, checked as https
|
||||
return "", fmt.Errorf("custom node %s at %s: git %s: %v: %s", n.Repo, n.Commit, args[len(args)-2], err, lastLines(b))
|
||||
}
|
||||
}
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
}
|
||||
if len(reqs) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
if uv == "" {
|
||||
return "", fmt.Errorf("custom nodes need %s, and no uv is installed to install them with", strings.Join(reqs, ", "))
|
||||
}
|
||||
site := filepath.Join(state, "site")
|
||||
marker := filepath.Join(site, ".llmgw-requirements")
|
||||
want := strings.Join(reqs, "\n")
|
||||
if b, err := os.ReadFile(marker); err == nil && string(b) == want {
|
||||
return site, nil
|
||||
}
|
||||
out("installing " + strings.Join(reqs, " ") + " for the custom nodes")
|
||||
// What ComfyUI's environment has, as constraints: a package both want is
|
||||
// installed at the version ComfyUI runs, so the replica's copy and
|
||||
// ComfyUI's never disagree.
|
||||
freeze, err := exec.CommandContext(ctx, uv, "pip", "freeze", "--python", python).Output() //nolint:gosec // the agent's own uv
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading ComfyUI's packages: %w", err)
|
||||
}
|
||||
constraints := filepath.Join(state, "constraints.txt")
|
||||
if err := os.WriteFile(constraints, withoutNamed(freeze, reqs), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
_ = os.RemoveAll(site)
|
||||
args := append([]string{"pip", "install", "--python", python, "--target", site, "-c", constraints, "--quiet"}, reqs...)
|
||||
cmd := exec.CommandContext(ctx, uv, args...) //nolint:gosec // requirements checked by the catalog as package specs
|
||||
cmd.Env = append(os.Environ(), "UV_NO_CONFIG=1", "UV_CACHE_DIR="+filepath.Join(state, ".uv-cache"))
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("installing the custom nodes' packages (%s): %v: %s", strings.Join(reqs, " "), err, lastLines(b))
|
||||
}
|
||||
// The replica's copies come first on ComfyUI's path, but of anything
|
||||
// both have they are the same version, so which is found is the same.
|
||||
if err := os.WriteFile(marker, []byte(want), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
return site, nil
|
||||
}
|
||||
|
||||
// parseNodes reads the nodes the agent handed over.
|
||||
func parseNodes(s string) ([]Node, error) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var nodes []Node
|
||||
if err := json.Unmarshal([]byte(s), &nodes); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", NodesEnv, err)
|
||||
}
|
||||
return nodes, nil
|
||||
}
|
||||
|
||||
// findUV is the uv the agent installed engines with.
|
||||
func findUV() string {
|
||||
if m, _ := filepath.Glob("/opt/llmgw/engines/uv/*/*/uv"); len(m) > 0 {
|
||||
return m[len(m)-1]
|
||||
}
|
||||
if p, err := exec.LookPath("uv"); err == nil {
|
||||
return p
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func gitOutput(ctx context.Context, dir string, args ...string) (string, error) {
|
||||
b, err := exec.CommandContext(ctx, "git", append([]string{"-C", dir}, args...)...).Output() //nolint:gosec // fixed arguments
|
||||
return strings.TrimSpace(string(b)), err
|
||||
}
|
||||
|
||||
func lastLines(b []byte) string {
|
||||
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
if len(lines) > 4 {
|
||||
lines = lines[len(lines)-4:]
|
||||
}
|
||||
return strings.Join(lines, " | ")
|
||||
}
|
||||
|
||||
// withoutNamed is ComfyUI's packages less those a node names itself: of a
|
||||
// package a node asks for by name, the node's range decides — TripoSR's
|
||||
// checkpoint loads with transformers 4 and not 5 — and of everything else,
|
||||
// what ComfyUI runs. A node that names a package must name a range ComfyUI
|
||||
// can run with too: the copy it gets is first on ComfyUI's path.
|
||||
func withoutNamed(freeze []byte, reqs []string) []byte {
|
||||
named := map[string]bool{}
|
||||
for _, r := range reqs {
|
||||
named[packageName(r)] = true
|
||||
}
|
||||
var out []string
|
||||
for _, line := range strings.Split(string(freeze), "\n") {
|
||||
if strings.TrimSpace(line) == "" || named[packageName(line)] {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return []byte(strings.Join(out, "\n") + "\n")
|
||||
}
|
||||
|
||||
// packageName is a requirement's package, as PyPI compares them.
|
||||
func packageName(spec string) string {
|
||||
s := strings.TrimSpace(spec)
|
||||
if i := strings.IndexAny(s, "<>=!~;[ @"); i >= 0 {
|
||||
s = s[:i]
|
||||
}
|
||||
return strings.NewReplacer("_", "-", ".", "-").Replace(strings.ToLower(s))
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/websocket"
|
||||
)
|
||||
|
||||
// How far along a workflow is, which ComfyUI says only on its websocket and
|
||||
// only to the client that queued the workflow. The gate keeps one websocket
|
||||
// open to its ComfyUI, under a client id of its own, puts that id on every
|
||||
// workflow it lets through, and keeps the last word on each: the node
|
||||
// running and its step out of how many. The router asks at
|
||||
// /progress/{prompt_id} while it waits, and a job says it.
|
||||
|
||||
// Progress is a workflow's state as the gate last heard it.
|
||||
type Progress struct {
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
// State is queued, running, done or failed.
|
||||
State string `json:"state"`
|
||||
// Ahead is how many workflows wait before it, while it is queued.
|
||||
Ahead int `json:"ahead,omitempty"`
|
||||
// Node is the workflow's node running, by its id; NodeClass its kind.
|
||||
Node string `json:"node,omitempty"`
|
||||
NodeClass string `json:"node_class,omitempty"`
|
||||
// Step of Steps is the running node's own count: a sampler's steps.
|
||||
Step int `json:"step,omitempty"`
|
||||
Steps int `json:"steps,omitempty"`
|
||||
// Percent is Step of Steps, where a node counts.
|
||||
Percent int `json:"percent,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// progressBook is the last word on each of the most recent workflows.
|
||||
type progressBook struct {
|
||||
mu sync.Mutex
|
||||
by map[string]*Progress
|
||||
order []string
|
||||
}
|
||||
|
||||
// keptPrompts bounds the book: a gate serves one replica, which runs one
|
||||
// workflow at a time and queues a few.
|
||||
const keptPrompts = 256
|
||||
|
||||
func newProgressBook() *progressBook { return &progressBook{by: map[string]*Progress{}} }
|
||||
|
||||
func (b *progressBook) update(id string, f func(*Progress)) {
|
||||
if id == "" {
|
||||
return
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
p = &Progress{PromptID: id, State: "queued"}
|
||||
b.by[id] = p
|
||||
b.order = append(b.order, id)
|
||||
for len(b.order) > keptPrompts {
|
||||
delete(b.by, b.order[0])
|
||||
b.order = b.order[1:]
|
||||
}
|
||||
}
|
||||
f(p)
|
||||
}
|
||||
|
||||
func (b *progressBook) get(id string) (Progress, bool) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
return Progress{}, false
|
||||
}
|
||||
return *p, true
|
||||
}
|
||||
|
||||
// read takes one message off ComfyUI's websocket into the book.
|
||||
func (b *progressBook) read(msg []byte) {
|
||||
var m struct {
|
||||
Type string `json:"type"`
|
||||
Data struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Node json.RawMessage `json:"node"`
|
||||
Value int `json:"value"`
|
||||
Max int `json:"max"`
|
||||
Message string `json:"exception_message"`
|
||||
NodeType string `json:"node_type"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(msg, &m) != nil {
|
||||
return
|
||||
}
|
||||
d := m.Data
|
||||
var node string
|
||||
_ = json.Unmarshal(d.Node, &node)
|
||||
switch m.Type {
|
||||
case "execution_start":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Ahead = "running", 0 })
|
||||
case "executing":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
if node == "" {
|
||||
// No node: the workflow is through.
|
||||
p.State, p.Node, p.Step, p.Steps, p.Percent = "done", "", 0, 0, 100
|
||||
return
|
||||
}
|
||||
p.State, p.Ahead, p.Node, p.Step, p.Steps, p.Percent = "running", 0, node, 0, 0, 0
|
||||
})
|
||||
case "progress":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State, p.Step, p.Steps = "running", d.Value, d.Max
|
||||
if node != "" {
|
||||
p.Node = node
|
||||
}
|
||||
if d.Max > 0 {
|
||||
p.Percent = min(100, d.Value*100/d.Max)
|
||||
}
|
||||
})
|
||||
case "execution_success":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Percent = "done", 100 })
|
||||
case "execution_error", "execution_interrupted":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State = "failed"
|
||||
p.Error = strings.TrimSpace(d.NodeType + ": " + d.Message)
|
||||
if m.Type == "execution_interrupted" {
|
||||
p.Error = "interrupted"
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// watch keeps a websocket open to the ComfyUI at upstream under clientID,
|
||||
// reading into the book, until ctx ends: opened again when it closes, as it
|
||||
// does whenever ComfyUI restarts.
|
||||
func (b *progressBook) watch(ctx context.Context, upstream *url.URL, clientID string, log *slog.Logger) {
|
||||
ws := "ws://" + upstream.Host + "/ws?clientId=" + url.QueryEscape(clientID)
|
||||
wait := time.Second
|
||||
for ctx.Err() == nil {
|
||||
conn, err := websocket.Dial(ws, "", "http://"+upstream.Host)
|
||||
if err == nil {
|
||||
wait = time.Second
|
||||
go func() { <-ctx.Done(); conn.Close() }()
|
||||
for {
|
||||
var msg []byte
|
||||
if err := websocket.Message.Receive(conn, &msg); err != nil {
|
||||
break
|
||||
}
|
||||
// Binary frames are previews of pictures; only text is news.
|
||||
if len(msg) > 0 && msg[0] == '{' {
|
||||
b.read(msg)
|
||||
}
|
||||
}
|
||||
conn.Close()
|
||||
} else if log != nil {
|
||||
log.Debug("ComfyUI's websocket is not open yet", "err", err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(wait):
|
||||
}
|
||||
wait = min(wait*2, 30*time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
// withClient is a /prompt body with the gate's client id in it, so the
|
||||
// progress of the workflow it queues comes to the gate's websocket.
|
||||
func withClient(body []byte, clientID string) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if json.Unmarshal(body, &m) != nil {
|
||||
return body
|
||||
}
|
||||
id, _ := json.Marshal(clientID)
|
||||
m["client_id"] = id
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// promptThroughGate forwards a /prompt with the gate's client id, and books
|
||||
// the workflow it queued as queued.
|
||||
func (b *progressBook) promptThroughGate(proxy http.Handler, clientID string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, 64<<20))
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
body = withClient(body, clientID)
|
||||
r.Body = io.NopCloser(bytes.NewReader(body))
|
||||
r.ContentLength = int64(len(body))
|
||||
rec := &captured{ResponseWriter: w}
|
||||
proxy.ServeHTTP(rec, r)
|
||||
var out struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
}
|
||||
if json.Unmarshal(rec.body.Bytes(), &out) == nil && out.PromptID != "" {
|
||||
b.update(out.PromptID, func(*Progress) {})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// captured passes a response on and keeps a copy of the first of it.
|
||||
type captured struct {
|
||||
http.ResponseWriter
|
||||
body bytes.Buffer
|
||||
}
|
||||
|
||||
func (c *captured) Write(p []byte) (int, error) {
|
||||
if c.body.Len() < 64<<10 {
|
||||
c.body.Write(p)
|
||||
}
|
||||
return c.ResponseWriter.Write(p)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBookReadsComfyUIsWebsocket(t *testing.T) {
|
||||
b := newProgressBook()
|
||||
for _, m := range []string{
|
||||
`{"type":"status","data":{"status":{"exec_info":{"queue_remaining":1}}}}`,
|
||||
`{"type":"execution_start","data":{"prompt_id":"p1"}}`,
|
||||
`{"type":"executing","data":{"node":"5","prompt_id":"p1"}}`,
|
||||
`{"type":"progress","data":{"value":12,"max":30,"prompt_id":"p1","node":"5"}}`,
|
||||
} {
|
||||
b.read([]byte(m))
|
||||
}
|
||||
p, _ := b.get("p1")
|
||||
if p.State != "running" || p.Node != "5" || p.Step != 12 || p.Steps != 30 || p.Percent != 40 {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"executing","data":{"node":null,"prompt_id":"p1"}}`))
|
||||
if p, _ := b.get("p1"); p.State != "done" || p.Percent != 100 {
|
||||
t.Errorf("finished: %+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"execution_error","data":{"prompt_id":"p2","node_type":"KSampler","exception_message":"CUDA out of memory"}}`))
|
||||
if p, _ := b.get("p2"); p.State != "failed" || p.Error != "KSampler: CUDA out of memory" {
|
||||
t.Errorf("failed: %+v", p)
|
||||
}
|
||||
for i := range keptPrompts + 10 {
|
||||
b.update(fmt.Sprint("x", i), func(*Progress) {})
|
||||
}
|
||||
if _, ok := b.get("p1"); ok || len(b.by) != keptPrompts {
|
||||
t.Errorf("the book kept %d", len(b.by))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGateHearsTheWorkflowsItLetsThrough(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var sent map[string]any
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/prompt" {
|
||||
mu.Lock()
|
||||
_ = json.NewDecoder(r.Body).Decode(&sent)
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p9"}`)
|
||||
}
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
book := newProgressBook()
|
||||
gate := httptest.NewServer(newGate(u, "", "m", book, "llmgw-gate-1"))
|
||||
defer gate.Close()
|
||||
resp, err := http.Post(gate.URL+"/prompt", "application/json", strings.NewReader(`{"prompt":{},"client_id":"router"}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if !strings.Contains(string(b), "p9") {
|
||||
t.Fatalf("the answer did not come through: %s", b)
|
||||
}
|
||||
mu.Lock()
|
||||
if sent["client_id"] != "llmgw-gate-1" {
|
||||
t.Errorf("ComfyUI was sent client %v: its progress would go to nobody listening", sent["client_id"])
|
||||
}
|
||||
mu.Unlock()
|
||||
book.read([]byte(`{"type":"progress","data":{"value":3,"max":4,"prompt_id":"p9","node":"2"}}`))
|
||||
resp, _ = http.Get(gate.URL + "/progress/p9")
|
||||
var p Progress
|
||||
_ = json.NewDecoder(resp.Body).Decode(&p)
|
||||
resp.Body.Close()
|
||||
if p.State != "running" || p.Percent != 75 {
|
||||
t.Errorf("/progress: %+v", p)
|
||||
}
|
||||
if resp, _ := http.Get(gate.URL + "/progress/unknown"); resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("an unknown workflow: %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitingWorkflowSaysHowManyAreAheadAndThenHowFarAlong(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
stage := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
s := stage
|
||||
mu.Unlock()
|
||||
switch r.URL.Path {
|
||||
case "/prompt":
|
||||
fmt.Fprint(w, `{"prompt_id":"me"}`)
|
||||
case "/queue":
|
||||
if s == 0 {
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"a"]],"queue_pending":[[3,"me"],[2,"b"],[4,"c"]]}`)
|
||||
} else {
|
||||
fmt.Fprint(w, `{"queue_running":[[3,"me"]],"queue_pending":[]}`)
|
||||
}
|
||||
case "/progress/me":
|
||||
fmt.Fprint(w, `{"state":"running","node":"5","step":6,"steps":12,"percent":50}`)
|
||||
case "/history/me":
|
||||
if s < 2 {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"me":{"status":{"status_str":"success","completed":true},"outputs":{"9":{"images":[{"filename":"a.png","type":"temp"}]}}}}`)
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png")
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
var seen []Progress
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := c.GenerateWatch(t.Context(), srv.URL, "", map[string]any{"5": map[string]any{"class_type": "KSampler"}},
|
||||
func(p Progress) {
|
||||
mu.Lock()
|
||||
seen = append(seen, p)
|
||||
stage++
|
||||
mu.Unlock()
|
||||
})
|
||||
done <- err
|
||||
}()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(seen) < 2 || seen[0].State != "queued" || seen[0].Ahead != 2 {
|
||||
t.Fatalf("%+v: queued behind the one running and the one numbered before it, wanted first", seen)
|
||||
}
|
||||
if seen[1].State != "running" || seen[1].Percent != 50 || seen[1].NodeClass != "KSampler" {
|
||||
t.Errorf("then %+v", seen[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsPutAtItsCommit(t *testing.T) {
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("no git")
|
||||
}
|
||||
repo := t.TempDir()
|
||||
run := func(args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = repo
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t")
|
||||
b, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v %s", args, err, b)
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
run("init", "-q")
|
||||
run("config", "uploadpack.allowReachableSHA1InWant", "true")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v1\n"), 0o644)
|
||||
run("add", ".")
|
||||
run("commit", "-qm", "one")
|
||||
first := run("rev-parse", "HEAD")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v2\n"), 0o644)
|
||||
run("commit", "-qam", "two")
|
||||
|
||||
state := t.TempDir()
|
||||
site, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first}}, "", func(string) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "custom_nodes", filepath.Base(repo), "__init__.py"))
|
||||
if string(b) != "v1\n" || site != "" {
|
||||
t.Errorf("the node is %q, site %q: wanted the commit named, not the branch's head", b, site)
|
||||
}
|
||||
if _, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first,
|
||||
Requirements: []string{"trimesh"}}}, "", func(string) {}); err == nil || !strings.Contains(err.Error(), "no uv") {
|
||||
t.Errorf("packages without a uv: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOwnRangeDecidesThePackagesItNames(t *testing.T) {
|
||||
freeze := []byte("numpy==2.3.1\ntransformers==5.19.0\nhuggingface-hub==1.33.0\nPillow==12.3.0\n")
|
||||
got := string(withoutNamed(freeze, []string{"transformers>=4.50.3,<5", "Huggingface_Hub<1", "trimesh"}))
|
||||
if strings.Contains(got, "transformers") || strings.Contains(got, "huggingface") ||
|
||||
!strings.Contains(got, "numpy==2.3.1") || !strings.Contains(got, "Pillow==12.3.0") {
|
||||
t.Errorf("constraints %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"8": {"class_type": "FluxGuidance", "inputs": {"guidance": "{{guidance|3.5}}", "conditioning": ["2", 0]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|20}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["8", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|4}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -493,6 +493,11 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Control configures the control plane.
|
||||
@@ -686,6 +691,10 @@ type Band struct {
|
||||
Name string `koanf:"name"`
|
||||
Priority int `koanf:"priority"`
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL is how long this band's requests may wait for room; the
|
||||
// admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
// had rather wait behind people than be turned away.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Timeouts bounds each phase of an upstream request.
|
||||
@@ -723,7 +732,10 @@ type Endpoint struct {
|
||||
Models []string `koanf:"models"`
|
||||
// Engine names the inference engine, which decides what its metrics are
|
||||
// called. Empty means vLLM's names.
|
||||
Engine string `koanf:"engine"`
|
||||
Engine string `koanf:"engine"`
|
||||
// Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
// name or its JSON; flux-schnell unless set.
|
||||
Workflow string `koanf:"workflow"`
|
||||
Pool string `koanf:"pool"`
|
||||
Weight float64 `koanf:"weight"`
|
||||
Priority int `koanf:"priority"`
|
||||
|
||||
@@ -203,6 +203,9 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/origin", a.setOrigin)
|
||||
mux.HandleFunc("GET /api/v1/plan", a.planModel)
|
||||
mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs", a.listJobs)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs/{id}", a.getJob)
|
||||
mux.HandleFunc("POST /api/v1/client-jobs/{id}/cancel", a.cancelJob)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}/body", a.requestBody)
|
||||
mux.HandleFunc("POST /api/v1/requests/{id}/replay", a.replayRequest)
|
||||
@@ -1611,6 +1614,8 @@ type RequestView struct {
|
||||
TTFTMs int64 `json:"ttft_ms"`
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Images int64 `json:"images,omitempty"`
|
||||
Meshes int64 `json:"meshes,omitempty"`
|
||||
CostMicros int64 `json:"cost_micros"`
|
||||
// What the guardrails did, by name.
|
||||
GuardAction string `json:"guard_action,omitempty"`
|
||||
@@ -1644,7 +1649,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
|
||||
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, Meshes: u.Meshes, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
|
||||
@@ -75,11 +75,11 @@ func (a *API) usageExport(w http.ResponseWriter, r *http.Request) {
|
||||
func WriteExportCSV(w io.Writer, rows []store.ExportRow) {
|
||||
c := csv.NewWriter(w)
|
||||
_ = c.Write([]string{"period_start", "org", "team", "project", "user", "key", "model",
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost"})
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost", "images", "meshes"})
|
||||
for _, r := range rows {
|
||||
_ = c.Write([]string{r.Period.Format(time.RFC3339), safe(r.Org), safe(r.Team), safe(r.Project), safe(r.User),
|
||||
r.Key, safe(r.Model), i(r.Requests), i(r.Errors), i(r.Limited), i(r.Prompt), i(r.Completion),
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64)})
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64), i(r.Images), i(r.Meshes)})
|
||||
}
|
||||
c.Flush()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Jobs — 3D models asked for and asked after — as an operator sees them:
|
||||
// every owner's, which a client sees only of its own.
|
||||
|
||||
// jobStore is what jobs need of the store; asked of the usage source by
|
||||
// type assertion, as requestFinder is.
|
||||
type jobStore interface {
|
||||
AllJobs(ctx context.Context, status string, limit int) ([]store.Job, error)
|
||||
JobByID(ctx context.Context, id string) (store.Job, error)
|
||||
CancelJob(ctx context.Context, id string, now time.Time) (store.Job, error)
|
||||
}
|
||||
|
||||
// JobView is one job.
|
||||
type JobView struct {
|
||||
ID string `json:"id"`
|
||||
Owner string `json:"owner"`
|
||||
KeyID string `json:"key_id"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Status string `json:"status"`
|
||||
Progress json.RawMessage `json:"progress,omitempty"`
|
||||
OutputFileID string `json:"output_file_id,omitempty"`
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
|
||||
func jobView(j store.Job) JobView {
|
||||
v := JobView{ID: j.ID, Owner: j.Owner, KeyID: j.KeyID, Endpoint: j.Endpoint, Model: j.Model, Status: j.Status,
|
||||
OutputFileID: j.OutputFileID, ContentType: j.ContentType, Error: j.Error, CreatedAt: j.CreatedAt}
|
||||
if j.Progress != "" && json.Valid([]byte(j.Progress)) {
|
||||
v.Progress = json.RawMessage(j.Progress)
|
||||
}
|
||||
if !j.StartedAt.IsZero() {
|
||||
v.StartedAt = &j.StartedAt
|
||||
}
|
||||
if !j.FinishedAt.IsZero() {
|
||||
v.FinishedAt = &j.FinishedAt
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (a *API) jobs() (jobStore, bool) {
|
||||
s, ok := a.usage.(jobStore)
|
||||
return s, ok
|
||||
}
|
||||
|
||||
func (a *API) listJobs(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
js, err := s.AllJobs(r.Context(), r.URL.Query().Get("status"), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]JobView, 0, len(js))
|
||||
for _, j := range js {
|
||||
out = append(out, jobView(j))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) getJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
j, err := s.JobByID(r.Context(), r.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
|
||||
func (a *API) cancelJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if _, err := s.JobByID(r.Context(), r.PathValue("id")); errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
j, err := s.CancelJob(r.Context(), r.PathValue("id"), time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
@@ -499,7 +499,11 @@ func (a *API) kbSearch(w http.ResponseWriter, r *http.Request) {
|
||||
// Said, not swallowed: a search that quietly lost its meaning half
|
||||
// looks like a collection that does not know the answer.
|
||||
if a.kb.Embed != nil {
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, in.Query)
|
||||
question := in.Query
|
||||
if m, ok := a.ctl.Catalog().Resolve(c.EmbedModel); ok {
|
||||
question = m.QueryPrefix + question
|
||||
}
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, question)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, c.EmbedModel+" did not embed the question: "+err.Error())
|
||||
return
|
||||
|
||||
@@ -62,6 +62,7 @@ var PathParams = map[string]string{
|
||||
"jobs/{id}": "The job's ID, as starting it answered.",
|
||||
"runs/{id}": "The quality run's ID, as starting it answered.",
|
||||
"requests/{id}": "The request's ID: the x-request-id a client was answered with.",
|
||||
"client-jobs/{id}": "The job's ID, job_….",
|
||||
"fabrics/{id}": "The physical fabric's ID, as GET /api/v1/fabrics lists it.",
|
||||
"quotas/{scope}": "The scope: org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT, or their IDs after the colon.",
|
||||
"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
|
||||
@@ -397,6 +398,14 @@ var Routes = []Route{
|
||||
{Name: "format", Description: "json for JSON; CSV otherwise."},
|
||||
},
|
||||
Response: []store.ExportRow{}, CSV: true},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs", Summary: "Jobs clients asked for — 3D models — of every owner, newest first",
|
||||
Query: []Param{
|
||||
{Name: "status", Description: "Only jobs in this status: queued, running, cancelling, completed, failed, cancelled, expired."},
|
||||
{Name: "limit", Type: "integer", Description: "The most jobs to answer with; 50 when absent."},
|
||||
},
|
||||
Response: []JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs/{id}", Summary: "One job a client asked for", Response: JobView{}},
|
||||
{Method: "POST", Path: "/api/v1/client-jobs/{id}/cancel", Summary: "Stop a job, in the engine's queue too", Response: JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}", Summary: "What one request went through",
|
||||
Response: RequestView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}/body", Summary: "A kept request and its answer",
|
||||
|
||||
@@ -30,7 +30,7 @@ type OrphanView struct {
|
||||
func (a *API) Orphans() []OrphanView {
|
||||
var wanted []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
wanted = append(wanted, m.Source)
|
||||
wanted = append(wanted, m.WeightSources()...)
|
||||
// A model's adapters are its weights too; collecting them would
|
||||
// leave replicas that cannot restart.
|
||||
for _, ad := range m.Adapters {
|
||||
|
||||
@@ -7,13 +7,17 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/url"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
@@ -172,6 +176,12 @@ type Profile struct {
|
||||
// 0.10.2. Empty takes whatever version the node has.
|
||||
EngineVersion string `json:"engine_version,omitempty" koanf:"engine_version"`
|
||||
GPUClass string `json:"gpu_class,omitempty" koanf:"gpu_class"`
|
||||
// Source is the weights this profile loads, when not the model's: an
|
||||
// embedding model served by vLLM from its repository on a card and by
|
||||
// llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
// loads the model's. Updates and canaries move the model's source; a
|
||||
// profile with its own keeps it.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
|
||||
TP int `json:"tp,omitempty" koanf:"tp"`
|
||||
PP int `json:"pp,omitempty" koanf:"pp"`
|
||||
@@ -321,6 +331,13 @@ type Model struct {
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
// ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
// pictures: the workflow, and the file it loads.
|
||||
ComfyUI *ComfyUI `json:"comfyui,omitempty" koanf:"comfyui"`
|
||||
// License is the terms of what the model makes, for a model whose
|
||||
// output somebody ships: shown with every result and written into a
|
||||
// mesh.
|
||||
License *License `json:"license,omitempty" koanf:"license"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it — unless
|
||||
@@ -363,6 +380,13 @@ type Model struct {
|
||||
// Capabilities is what this model can do beyond plain completion.
|
||||
Capabilities Capabilities `json:"capabilities,omitempty" koanf:"capabilities"`
|
||||
|
||||
// QueryPrefix is put before a question an embedding model embeds for a
|
||||
// knowledge-base search; documents are embedded without it. Models
|
||||
// trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
// "Instruct: Given a web search query, retrieve relevant passages that
|
||||
// answer the query\nQuery: " for one.
|
||||
QueryPrefix string `json:"query_prefix,omitempty" koanf:"query_prefix"`
|
||||
|
||||
// Role "guard" is a model that judges the gateway's traffic for the
|
||||
// guardrails rather than serving clients: hidden from /v1/models and the
|
||||
// chat page, refused to any key that names it, critical by default, and
|
||||
@@ -450,6 +474,202 @@ type Capabilities struct {
|
||||
Tools bool `json:"tools,omitempty" koanf:"tools"`
|
||||
Vision bool `json:"vision,omitempty" koanf:"vision"`
|
||||
Embeddings bool `json:"embeddings,omitempty" koanf:"embeddings"`
|
||||
// Images is a model that makes pictures, asked for at
|
||||
// /v1/images/generations. Implied by an external model whose API is
|
||||
// ComfyUI's.
|
||||
Images bool `json:"images,omitempty" koanf:"images"`
|
||||
// Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
// job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
// meshes.
|
||||
Meshes bool `json:"meshes,omitempty" koanf:"meshes"`
|
||||
}
|
||||
|
||||
// The kinds of thing a ComfyUI model makes.
|
||||
const (
|
||||
OutputPictures = "pictures"
|
||||
OutputMeshes = "meshes"
|
||||
)
|
||||
|
||||
// comfyOutput is what a ComfyUI model makes: pictures unless it says.
|
||||
func (m Model) comfyOutput() string {
|
||||
out := ""
|
||||
switch {
|
||||
case m.External != nil:
|
||||
out = m.External.Output
|
||||
case m.ComfyUI != nil:
|
||||
out = m.ComfyUI.Output
|
||||
}
|
||||
if out == "" {
|
||||
return OutputPictures
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MakesImages reports whether a model answers /v1/images/generations.
|
||||
func (m Model) MakesImages() bool {
|
||||
return m.Capabilities.Images || m.ServesComfyUI() && m.comfyOutput() == OutputPictures
|
||||
}
|
||||
|
||||
// MakesMeshes reports whether a model answers /v1/3d/generations.
|
||||
func (m Model) MakesMeshes() bool {
|
||||
return m.Capabilities.Meshes || m.ServesComfyUI() && m.comfyOutput() == OutputMeshes
|
||||
}
|
||||
|
||||
// License is the terms what a model makes comes under: said beside every
|
||||
// result, and written into a mesh, which travels without the API.
|
||||
type License struct {
|
||||
// Name is the licence's own: MIT, Apache-2.0.
|
||||
Name string `json:"name" koanf:"name"`
|
||||
URL string `json:"url,omitempty" koanf:"url"`
|
||||
// Notice is what must accompany what was made: attribution, terms on
|
||||
// the output, what the weights were trained on.
|
||||
Notice string `json:"notice,omitempty" koanf:"notice"`
|
||||
}
|
||||
|
||||
// EngineComfyUI is the engine that runs ComfyUI on a node.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// ServesComfyUI reports whether a model is made by ComfyUI, on the
|
||||
// cluster's own nodes or on a server outside it.
|
||||
func (m Model) ServesComfyUI() bool {
|
||||
if m.External != nil {
|
||||
return m.External.API == APIComfyUI
|
||||
}
|
||||
return len(m.Profiles) > 0 && m.Profiles[0].Engine == EngineComfyUI
|
||||
}
|
||||
|
||||
// ComfyUI is a ComfyUI model's workflow and checkpoint.
|
||||
type ComfyUI struct {
|
||||
// Workflow is a built-in's name (flux-schnell, the default, or
|
||||
// flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
// format.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
// source's own name unless set, which a source that is a set of files
|
||||
// must.
|
||||
Checkpoint string `json:"checkpoint,omitempty" koanf:"checkpoint"`
|
||||
// Output is what the workflow makes: pictures (the default), asked for
|
||||
// at /v1/images/generations, or meshes, asked for as a job at
|
||||
// /v1/3d/generations from a picture.
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
// Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
// custom_nodes at the commit named.
|
||||
Nodes []CustomNode `json:"nodes,omitempty" koanf:"nodes"`
|
||||
}
|
||||
|
||||
// CustomNode is a ComfyUI node from a git repository.
|
||||
type CustomNode struct {
|
||||
// Repo is the repository, https only.
|
||||
Repo string `json:"repo" koanf:"repo"`
|
||||
// Commit is the full commit the node is used at: a branch moves, and a
|
||||
// node that changed under a model is a model that changed unasked.
|
||||
Commit string `json:"commit" koanf:"commit"`
|
||||
// Requirements are the PyPI packages the node needs, by name and version
|
||||
// range only. They are installed beside the replica, never into
|
||||
// ComfyUI's environment: a package named here at the range named, and
|
||||
// every other one at the version ComfyUI already runs. One named here
|
||||
// is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
Requirements []string `json:"requirements,omitempty" koanf:"requirements"`
|
||||
}
|
||||
|
||||
// NodesEnv carries a ComfyUI model's custom nodes to the agent, which hands
|
||||
// them to the gate.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var (
|
||||
fullCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
|
||||
packageSpec = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*(\[[A-Za-z0-9,._-]+\])?\s*([<>=!~]=?\s*[A-Za-z0-9.*+!_-]+\s*,?\s*)*$`)
|
||||
)
|
||||
|
||||
func (n CustomNode) validate(model string) error {
|
||||
u, err := url.Parse(n.Repo)
|
||||
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil {
|
||||
return fmt.Errorf("model %q: custom node %q is not an https repository", model, n.Repo)
|
||||
}
|
||||
if !fullCommit.MatchString(n.Commit) {
|
||||
return fmt.Errorf("model %q: custom node %s needs its full commit (40 hex), not %q: a branch moves",
|
||||
model, n.Repo, n.Commit)
|
||||
}
|
||||
for _, r := range n.Requirements {
|
||||
// A package and a version range: an option or a URL would install
|
||||
// from somewhere other than PyPI.
|
||||
if !packageSpec.MatchString(strings.TrimSpace(r)) {
|
||||
return fmt.Errorf("model %q: custom node %s requirement %q is not a package with a version range", model, n.Repo, r)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ComfyWorkflow is what a ComfyUI model's replicas are sent: the workflow
|
||||
// as the catalog names it, and the checkpoint it loads.
|
||||
func (m Model) ComfyWorkflow() (workflow, checkpoint string) {
|
||||
if m.External != nil {
|
||||
return m.External.Workflow, m.External.Model
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
workflow, checkpoint = m.ComfyUI.Workflow, m.ComfyUI.Checkpoint
|
||||
}
|
||||
if checkpoint == "" && !strings.HasSuffix(m.Source, "/") {
|
||||
checkpoint = path.Base(strings.TrimPrefix(m.Source, "hf:"))
|
||||
}
|
||||
return workflow, checkpoint
|
||||
}
|
||||
|
||||
// checkWorkflow refuses a workflow that is neither a built-in nor a file.
|
||||
func checkWorkflow(model, w string) error {
|
||||
// A file is read where the router runs; a name is checked here.
|
||||
if w != "" && !strings.ContainsAny(w, "/.") && !slices.Contains(comfy.Builtins(), w) {
|
||||
return fmt.Errorf("model %q: no built-in workflow %q (there are %s; or name a JSON file)",
|
||||
model, w, strings.Join(comfy.Builtins(), ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) validateComfyUI() error {
|
||||
comfyProfiles := 0
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == EngineComfyUI {
|
||||
comfyProfiles++
|
||||
}
|
||||
}
|
||||
if comfyProfiles == 0 {
|
||||
if m.ComfyUI != nil {
|
||||
return fmt.Errorf("model %q: comfyui: is for a model whose profiles run engine comfyui", m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if comfyProfiles != len(m.Profiles) {
|
||||
return fmt.Errorf("model %q: every profile runs engine comfyui or none does: "+
|
||||
"a request for pictures cannot be answered by an engine that writes text", m.Name)
|
||||
}
|
||||
if len(m.Adapters) > 0 {
|
||||
return fmt.Errorf("model %q: ComfyUI serves no adapters", m.Name)
|
||||
}
|
||||
var w string
|
||||
if m.ComfyUI != nil {
|
||||
w = m.ComfyUI.Workflow
|
||||
}
|
||||
if err := checkWorkflow(m.Name, w); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
for _, n := range m.ComfyUI.Nodes {
|
||||
if err := n.validate(m.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if out := m.comfyOutput(); out != OutputPictures && out != OutputMeshes {
|
||||
return fmt.Errorf("model %q: comfyui.output %q is not pictures or meshes", m.Name, out)
|
||||
}
|
||||
if m.comfyOutput() == OutputMeshes && w == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", m.Name)
|
||||
}
|
||||
if _, ckpt := m.ComfyWorkflow(); ckpt == "" {
|
||||
return fmt.Errorf("model %q: its source is a set of files, so comfyui.checkpoint must name the one the workflow loads",
|
||||
m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Catalog is the set of models the cluster should serve.
|
||||
@@ -658,9 +878,15 @@ func (m Model) normalize() Model {
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
if repo, ok := repositoryPage(p.Source); ok && p.Engine != "llamacpp" {
|
||||
p.Source = repo
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -671,15 +897,26 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a source
|
||||
// llama.cpp loads is left here, which needs a file.
|
||||
if repo, ok := repositoryPage(m.Source); ok && (len(m.Profiles) == 0 ||
|
||||
slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" })) {
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if repo, ok := repositoryPage(p.Source); ok {
|
||||
return fmt.Errorf("model %q profile %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, p.Name, p.Source, repo)
|
||||
}
|
||||
if p.Source != "" && m.External != nil {
|
||||
return fmt.Errorf("model %q profile %q: an external model has no profiles to give a source", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -690,6 +927,9 @@ func (m Model) validate() error {
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if err := m.validateComfyUI(); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
@@ -817,7 +1057,7 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: trtllm runs a replica on one node; "+
|
||||
"use vllm or sglang for a %d-node replica", m.Name, p.Name, p.Nodes)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.Source, "/")), ".gguf") {
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
|
||||
return fmt.Errorf("model %q profile %q: trtllm loads a Hugging Face checkpoint, not a .gguf; "+
|
||||
"serve a GGUF with llamacpp", m.Name, p.Name)
|
||||
}
|
||||
@@ -903,20 +1143,53 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
CPUCores: p.CPUCores,
|
||||
PowerWatts: p.PowerWatts,
|
||||
KeepWithin: facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// The profile's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
Source: m.SourceOf(p),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SourceOf is the weights a profile loads: its own, or the model's.
|
||||
func (m Model) SourceOf(p Profile) string {
|
||||
if p.Source != "" {
|
||||
return p.Source
|
||||
}
|
||||
return m.Source
|
||||
}
|
||||
|
||||
// WeightSources are every source a model's replicas may load: the model's
|
||||
// and each profile's of its own, once each. What a node must not have
|
||||
// collected from under it.
|
||||
func (m Model) WeightSources() []string {
|
||||
out := []string{}
|
||||
if m.Source != "" {
|
||||
out = append(out, m.Source)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Source != "" && !slices.Contains(out, p.Source) {
|
||||
out = append(out, p.Source)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EngineProfile converts a catalog profile into the engine adapter's view.
|
||||
func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
for _, p := range m.Profiles {
|
||||
if p.Name != name {
|
||||
continue
|
||||
}
|
||||
env := p.Env
|
||||
if p.Engine == EngineComfyUI && m.ComfyUI != nil && len(m.ComfyUI.Nodes) > 0 {
|
||||
nodes, _ := json.Marshal(m.ComfyUI.Nodes)
|
||||
env = maps.Clone(p.Env)
|
||||
if env == nil {
|
||||
env = map[string]string{}
|
||||
}
|
||||
env[NodesEnv] = string(nodes)
|
||||
}
|
||||
return engines.Profile{
|
||||
Name: p.Name,
|
||||
Engine: p.Engine,
|
||||
@@ -926,8 +1199,8 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
GPUsPerNode: p.GPUsPerNode,
|
||||
Nodes: p.Nodes,
|
||||
Args: slices.Clone(p.Args),
|
||||
Env: p.Env,
|
||||
ModelPath: m.Source,
|
||||
Env: env,
|
||||
ModelPath: m.SourceOf(p),
|
||||
ServedModelName: m.Name,
|
||||
Adapters: m.engineAdapters(),
|
||||
KVEvents: p.KVEvents,
|
||||
@@ -1094,8 +1367,22 @@ type External struct {
|
||||
// than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
// goes with the peer's own list, and is sent only while the peer is up.
|
||||
Peer string `json:"peer,omitempty" koanf:"peer"`
|
||||
// API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
// a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
// sent the workflow below with the request's prompt in it. Model is then
|
||||
// the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
API string `json:"api,omitempty" koanf:"api"`
|
||||
// Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
// (flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
// plane.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
}
|
||||
|
||||
// APIComfyUI is External.API for a ComfyUI server.
|
||||
const APIComfyUI = "comfyui"
|
||||
|
||||
// IsPeer reports whether a model is offered by a federated cluster.
|
||||
func (m Model) IsPeer() bool { return m.External != nil && m.External.Peer != "" }
|
||||
|
||||
@@ -1110,6 +1397,27 @@ func (e *External) validate(model string, profiles int) error {
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
switch e.API {
|
||||
case "", "openai":
|
||||
if e.Workflow != "" || e.Output != "" {
|
||||
return fmt.Errorf("model %q: a workflow and an output are for api: comfyui", model)
|
||||
}
|
||||
case APIComfyUI:
|
||||
if e.Peer != "" {
|
||||
return fmt.Errorf("model %q: a federated model speaks OpenAI's API, not ComfyUI's", model)
|
||||
}
|
||||
if err := checkWorkflow(model, e.Workflow); err != nil {
|
||||
return err
|
||||
}
|
||||
if e.Output != "" && e.Output != OutputPictures && e.Output != OutputMeshes {
|
||||
return fmt.Errorf("model %q: external output %q is not pictures or meshes", model, e.Output)
|
||||
}
|
||||
if e.Output == OutputMeshes && e.Workflow == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", model)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func comfyModel(source string, c *ComfyUI, profiles ...Profile) Model {
|
||||
if len(profiles) == 0 {
|
||||
profiles = []Profile{{Name: "gpu", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1}}
|
||||
}
|
||||
return Model{Name: "flux", Source: source, ComfyUI: c, Profiles: profiles}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelMakesPicturesFromItsSourcesFile(t *testing.T) {
|
||||
cat, err := New([]Model{comfyModel("hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors", nil)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
if !m.MakesImages() || !m.ServesComfyUI() {
|
||||
t.Error("a ComfyUI model does not say it makes pictures")
|
||||
}
|
||||
if w, ckpt := m.ComfyWorkflow(); w != "" || ckpt != "flux1-schnell-fp8.safetensors" {
|
||||
t.Errorf("workflow %q, checkpoint %q", w, ckpt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsRefusedWhatItCannotDo(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
m Model
|
||||
want string
|
||||
}{
|
||||
"a set without a checkpoint": {comfyModel("https://mirror/flux-dev/v1/", nil), "comfyui.checkpoint"},
|
||||
"an unknown workflow": {comfyModel("/w/f.safetensors", &ComfyUI{Workflow: "sdxl"}), "no built-in workflow"},
|
||||
"mixed engines": {comfyModel("/w/f.safetensors", nil,
|
||||
Profile{Name: "a", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1},
|
||||
Profile{Name: "b", Engine: "vllm", GPUClass: "6000-24g", TP: 1}), "every profile"},
|
||||
"comfyui settings without the engine": {Model{Name: "x", Source: "/w/x", ComfyUI: &ComfyUI{},
|
||||
Profiles: []Profile{{Name: "a", Engine: "vllm", GPUClass: "6000-24g", TP: 1}}}, "engine comfyui"},
|
||||
"no card": {comfyModel("/w/f.safetensors", nil, Profile{Name: "a", Engine: EngineComfyUI, TP: 1}), "gpu_class"},
|
||||
} {
|
||||
if _, err := New([]Model{c.m}); err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: %v", name, err)
|
||||
}
|
||||
}
|
||||
ok := comfyModel("https://mirror/flux-dev/v1/", &ComfyUI{Workflow: "flux-dev", Checkpoint: "flux1-dev-fp8.safetensors"})
|
||||
if _, err := New([]Model{ok}); err != nil {
|
||||
t.Errorf("a set with its checkpoint named was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACustomNodeIsPinnedAndItsPackagesArePackages(t *testing.T) {
|
||||
node := func(n CustomNode) Model {
|
||||
return comfyModel("hf:stabilityai/TripoSR/model.ckpt", &ComfyUI{Workflow: "w.json", Output: OutputMeshes,
|
||||
Nodes: []CustomNode{n}})
|
||||
}
|
||||
ok := CustomNode{Repo: "https://github.com/flowtyone/ComfyUI-Flowty-TripoSR",
|
||||
Commit: "0123456789abcdef0123456789abcdef01234567", Requirements: []string{"omegaconf", "trimesh>=4,<5", "scikit-image"}}
|
||||
cat, err := New([]Model{node(ok)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
p, _ := m.EngineProfile("gpu")
|
||||
if !strings.Contains(p.Env[NodesEnv], "ComfyUI-Flowty-TripoSR") || !m.MakesMeshes() || m.MakesImages() {
|
||||
t.Errorf("env %v meshes %v pictures %v", p.Env, m.MakesMeshes(), m.MakesImages())
|
||||
}
|
||||
for name, n := range map[string]CustomNode{
|
||||
"a branch": {Repo: ok.Repo, Commit: "main"},
|
||||
"http": {Repo: "http://example.com/n", Commit: ok.Commit},
|
||||
"an option": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"--index-url=https://evil"}},
|
||||
"a url": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"x @ https://evil/x.whl"}},
|
||||
} {
|
||||
if _, err := New([]Model{node(n)}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
noWorkflow := comfyModel("hf:a/b/c.ckpt", &ComfyUI{Output: OutputMeshes})
|
||||
if _, err := New([]Model{noWorkflow}); err == nil || !strings.Contains(err.Error(), "names its workflow") {
|
||||
t.Errorf("a mesh model without a workflow: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// embedModel is an embedding model the way it was wanted on the PoC: vLLM
|
||||
// on the card from the repository, llama.cpp on CPUs from a GGUF.
|
||||
func embedModel() Model {
|
||||
return Model{Name: "qwen3-embed-4b", Source: "Qwen/Qwen3-Embedding-4B",
|
||||
Profiles: []Profile{
|
||||
{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1},
|
||||
{Name: "cpu", Engine: "llamacpp", Source: "hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestEachProfileLoadsItsOwnSource(t *testing.T) {
|
||||
// The catalog had one source a model, so serving an embedding model by
|
||||
// vLLM on a card and by llama.cpp on CPUs meant swapping the entry by
|
||||
// hand halfway through an import.
|
||||
cat, err := New([]Model{embedModel()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
gpu, _ := m.EngineProfile("gpu")
|
||||
cpu, _ := m.EngineProfile("cpu")
|
||||
if gpu.ModelPath != "Qwen/Qwen3-Embedding-4B" || !strings.HasSuffix(cpu.ModelPath, "Q8_0.gguf") {
|
||||
t.Errorf("gpu loads %q, cpu %q", gpu.ModelPath, cpu.ModelPath)
|
||||
}
|
||||
for _, p := range m.PlacementProfiles() {
|
||||
if want := map[string]string{"gpu": gpu.ModelPath, "cpu": cpu.ModelPath}[p.Name]; p.Source != want {
|
||||
t.Errorf("placement for %s looks for %q, want %q", p.Name, p.Source, want)
|
||||
}
|
||||
}
|
||||
if got := m.WeightSources(); len(got) != 2 || !slices.Contains(got, cpu.ModelPath) {
|
||||
t.Errorf("weight sources %v: a profile's own must not be collected", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileWithoutASourceKeepsItsFingerprint(t *testing.T) {
|
||||
// Replicas are compared with their profile by fingerprint; a field added
|
||||
// to every profile must not make every running replica look stale.
|
||||
p := Profile{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1}
|
||||
before := p.Fingerprint()
|
||||
if strings.Contains(mustJSON(t, p), "source") {
|
||||
t.Error("an empty source is written into the profile")
|
||||
}
|
||||
p.Source = "Qwen/Other"
|
||||
if p.Fingerprint() == before {
|
||||
t.Error("a profile whose source changed reads as unchanged")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileSourceIsCheckedAsTheModelsIs(t *testing.T) {
|
||||
page := embedModel()
|
||||
page.Profiles[1].Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF"
|
||||
if _, err := New([]Model{page}); err == nil || !strings.Contains(err.Error(), `profile "cpu"`) {
|
||||
t.Errorf("a repository page for llama.cpp: %v", err)
|
||||
}
|
||||
// The model's own source may be a page vLLM reads as a repository, now
|
||||
// that the llama.cpp profile loads something else.
|
||||
repo := embedModel()
|
||||
repo.Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B"
|
||||
cat, err := New([]Model{repo})
|
||||
if err != nil {
|
||||
t.Fatalf("a repository page for the vLLM profile beside a llama.cpp one with its own GGUF: %v", err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
if m.Source != "Qwen/Qwen3-Embedding-4B" {
|
||||
t.Errorf("the page was not read as the repository: %q", m.Source)
|
||||
}
|
||||
trt := embedModel()
|
||||
trt.Profiles[0] = Profile{Name: "gpu", Engine: "trtllm", GPUClass: "6000-24g", TP: 1, Source: "hf:a/b/m.gguf"}
|
||||
if _, err := New([]Model{trt}); err == nil || !strings.Contains(err.Error(), "not a .gguf") {
|
||||
t.Errorf("trtllm given a GGUF by its profile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(t *testing.T, v any) string {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -360,7 +361,7 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
names := append([]string{m.Name}, m.Aliases...)
|
||||
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
out = append(out, endpoint.Info{
|
||||
info := endpoint.Info{
|
||||
ID: rep.ID,
|
||||
URL: strings.TrimRight(rep.Endpoint, "/"),
|
||||
Models: names,
|
||||
@@ -369,7 +370,18 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Pool: rep.Pool,
|
||||
Weight: rep.Weight,
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
if prof.Engine == catalog.EngineComfyUI {
|
||||
workflow, ckpt := m.ComfyWorkflow()
|
||||
w, err := c.workflow(workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; its replica is not offered",
|
||||
"model", m.Name, "replica", rep.ID, "err", err)
|
||||
continue
|
||||
}
|
||||
info.Workflow, info.UpstreamModel = w, ckpt
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
@@ -412,23 +424,45 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
if m.IsPeer() {
|
||||
id = "peer:" + m.Name
|
||||
}
|
||||
engine, workflow := endpoint.EngineExternal, ""
|
||||
if m.External.API == catalog.APIComfyUI {
|
||||
w, err := c.workflow(m.External.Workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
engine, workflow = endpoint.EngineComfyUI, w
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: id, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
Engine: engine, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
Peer: m.External.Peer, PeerCluster: cluster, PeerTags: tags,
|
||||
Workflow: workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// workflow is a ComfyUI workflow as the router is sent it: a built-in's
|
||||
// name as it is, a file read here, on the control plane the catalog says it
|
||||
// is on, and sent as the workflow itself.
|
||||
func (c *Controller) workflow(w string) (string, error) {
|
||||
if !strings.ContainsAny(w, "/.") {
|
||||
return w, nil
|
||||
}
|
||||
b, err := os.ReadFile(w)
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// fingerprint summarizes a list so an unchanged one can be skipped.
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)))
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)),
|
||||
sha256.Sum256([]byte(i.Workflow)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -87,8 +87,8 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
}
|
||||
out := map[string][]catalog.Model{}
|
||||
for _, m := range c.Models() {
|
||||
if m.Source != "" {
|
||||
out[m.Source] = append(out[m.Source], m)
|
||||
for _, src := range m.WeightSources() {
|
||||
out[src] = append(out[src], m)
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
out[ad.Source] = append(out[ad.Source], m)
|
||||
|
||||
@@ -111,7 +111,14 @@ func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []Ad
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
// An on-demand model that is up was asked for, by somebody this
|
||||
// control plane has not heard of: it started before a restart or a
|
||||
// change of leader. Without this it reads as wanted by nobody and is
|
||||
// drained on the next pass — every awake on-demand model, at every
|
||||
// failover. It gets a whole idle timeout from now instead.
|
||||
if _, asked := r.wanted[m.Name]; !asked && m.Scaling.OnDemand() {
|
||||
r.wanted[m.Name] = r.now()
|
||||
}
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
@@ -1409,6 +1409,29 @@ func TestUseKeepsAnOnDemandModelAlive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAwakeOnDemandModelSurvivesARestartOfTheControlPlane(t *testing.T) {
|
||||
// A control plane that has just started, or just become leader, knows
|
||||
// nobody asked for anything. Taking that to mean nobody wants the
|
||||
// on-demand models a node reports running drained every one of them at
|
||||
// every failover, under whoever was using them.
|
||||
h := newHarness(t, onDemandModel(10*time.Minute), gpuNode("1", 8))
|
||||
h.r.Adopt("1", []AdoptedShard{adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(5 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
if reps := h.r.Replicas(); len(reps) != 1 || !reps[0].State.Counts() {
|
||||
t.Fatalf("the adopted replica of an on-demand model was taken down: %+v", reps)
|
||||
}
|
||||
// Still an on-demand model: unused for a whole idle timeout, it goes.
|
||||
h.advance(6 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State.Counts() {
|
||||
t.Errorf("an adopted replica nobody used outlived its idle timeout: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelWithMaxZeroStaysOff(t *testing.T) {
|
||||
// min 0 max 0 is configured and deliberately not running, which is a
|
||||
// different and equally useful thing from "load it when wanted".
|
||||
|
||||
@@ -64,6 +64,18 @@ var Reference = []Block{
|
||||
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
|
||||
{Key: "meshes", Type: "bool", Doc: "Meshes is a model that makes 3D models from a picture, asked for as a\njob at /v1/3d/generations. Implied by a ComfyUI model whose output is\nmeshes.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ComfyUI", Title: "ComfyUI", Doc: "", Fields: []Field{
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is a built-in's name (flux-schnell, the default, or\nflux-dev) or a JSON file on the control plane, in ComfyUI's API\nformat.", Nested: ""},
|
||||
{Key: "checkpoint", Type: "string", Doc: "Checkpoint is the file the workflow's {{checkpoint}} names: the\nsource's own name unless set, which a source that is a set of files\nmust.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what the workflow makes: pictures (the default), asked for\nat /v1/images/generations, or meshes, asked for as a job at\n/v1/3d/generations from a picture.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]CustomNode", Doc: "Nodes are custom nodes the workflow needs, put in each replica's own\ncustom_nodes at the commit named.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "ComfyUIConfig", Title: "ComfyUIConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is ComfyUI's directory, where main.py is. Readable by the user\nengines run as.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
|
||||
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
|
||||
@@ -78,12 +90,18 @@ var Reference = []Block{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
|
||||
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "CustomNode", Title: "CustomNode", Doc: "", Fields: []Field{
|
||||
{Key: "repo", Type: "string", Doc: "Repo is the repository, https only.", Nested: ""},
|
||||
{Key: "commit", Type: "string", Doc: "Commit is the full commit the node is used at: a branch moves, and a\nnode that changed under a model is a model that changed unasked.", Nested: ""},
|
||||
{Key: "requirements", Type: "[]string", Doc: "Requirements are the PyPI packages the node needs, by name and version\nrange only. They are installed beside the replica, never into\nComfyUI's environment: a package named here at the range named, and\nevery other one at the version ComfyUI already runs. One named here\nis first on ComfyUI's path, so its range must suit ComfyUI too.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
|
||||
{Key: "id", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
|
||||
{Key: "models", Type: "[]string", Doc: "", Nested: ""},
|
||||
{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow for engine: comfyui, a built-in's\nname or its JSON; flux-schnell unless set.", Nested: ""},
|
||||
{Key: "pool", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
@@ -104,6 +122,9 @@ var Reference = []Block{
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
@@ -149,6 +170,11 @@ var Reference = []Block{
|
||||
{Key: "language", Type: "string", Doc: "Language is what the web portal speaks to someone who has not chosen\na language with the switcher at the foot of every page and whose\nbrowser asks for none it speaks: \"en\" or \"tr\". Empty is English.", Nested: ""},
|
||||
{Key: "docs_sign_in", Type: "bool", Doc: "DocsSignIn serves the docs portal at /docs only to someone signed in.\nOff, the portal is public like this page: its client guide is for the\nperson who has no account yet, and the operator documents describe\nllmgw in general — no address, model or key of this installation —\nwhich the project's repository publishes anyway. On, a visitor is sent\nto sign in, and without accounts the portal needs an API key.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "License", Title: "License", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "Name is the licence's own: MIT, Apache-2.0.", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "notice", Type: "string", Doc: "Notice is what must accompany what was made: attribution, terms on\nthe output, what the weights were trained on.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
|
||||
{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
|
||||
@@ -171,6 +197,8 @@ var Reference = []Block{
|
||||
{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
|
||||
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
|
||||
{Key: "comfyui", Type: "*ComfyUI", Doc: "ComfyUI is how a model whose profiles run engine comfyui makes its\npictures: the workflow, and the file it loads.", Nested: "catalog.ComfyUI"},
|
||||
{Key: "license", Type: "*License", Doc: "License is the terms of what the model makes, for a model whose\noutput somebody ships: shown with every result and written into a\nmesh.", Nested: "catalog.License"},
|
||||
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it — unless\nMaxLoadedAdapters is set.", Nested: ""},
|
||||
{Key: "max_loaded_adapters", Type: "int", Doc: "MaxLoadedAdapters, above zero, has each replica hold at most this many\nadapters at once, loaded when a request first asks for one and unloaded\nleast recently used first to make room. The router sends a request for\nan adapter to a replica that already holds it while that one has room\nto take it. vLLM profiles only: llama-server loads adapters at start\nand cannot load one afterwards. Zero loads every adapter on every\nreplica at start.", Nested: ""},
|
||||
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
|
||||
@@ -181,6 +209,7 @@ var Reference = []Block{
|
||||
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
|
||||
{Key: "max_output", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
|
||||
{Key: "query_prefix", Type: "string", Doc: "QueryPrefix is put before a question an embedding model embeds for a\nknowledge-base search; documents are embedded without it. Models\ntrained with an instruction on the query side want it, Qwen3-Embedding\n\"Instruct: Given a web search query, retrieve relevant passages that\nanswer the query\\nQuery: \" for one.", Nested: ""},
|
||||
{Key: "role", Type: "string", Doc: "Role \"guard\" is a model that judges the gateway's traffic for the\nguardrails rather than serving clients: hidden from /v1/models and the\nchat page, refused to any key that names it, critical by default, and\nalways running. Guard says how its answers are read.", Nested: ""},
|
||||
{Key: "guard", Type: "*GuardModel", Doc: "", Nested: "catalog.GuardModel"},
|
||||
{Key: "origin", Type: "*Origin", Doc: "Origin is the registry repository the weights came from, at which\ncommit, and each file's checksum there. Written when a model is served\nfrom the registry; for one that was not, worked out from the mirror's\nrecord of where it fetched the file. It is what lets the control plane\nsay that the repository has published different weights since, and\nmove the model to them.", Nested: "catalog.Origin"},
|
||||
@@ -270,6 +299,7 @@ var Reference = []Block{
|
||||
{Key: "engine", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "engine_version", Type: "string", Doc: "EngineVersion restricts the profile to nodes whose installed engine's\nversion starts with it: \"0.11\" takes vLLM 0.11.0 and 0.11.1, not\n0.10.2. Empty takes whatever version the node has.", Nested: ""},
|
||||
{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the weights this profile loads, when not the model's: an\nembedding model served by vLLM from its repository on a card and by\nllama.cpp from a GGUF on CPUs is one model with two sources. Empty\nloads the model's. Updates and canaries move the model's source; a\nprofile with its own keeps it.", Nested: ""},
|
||||
{Key: "tp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "pp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
|
||||
@@ -444,11 +474,14 @@ var Reference = []Block{
|
||||
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is what one 3D model costs: minutes of a card, which no\ntoken count measures.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL, set, is how long this band's requests may wait instead of\nthe controller's: a band for bulk work — a knowledge-base import, a\nbackfill — whose requests had rather wait ten minutes behind people\nthan be turned away after thirty seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
|
||||
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
|
||||
@@ -484,10 +517,13 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL is how long this band's requests may wait for room; the\nadmission's queue_ttl unless set. Long for a band of bulk work, which\nhad rather wait behind people than be turned away.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is the cost of one 3D model (/v1/3d/generations).", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
|
||||
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
|
||||
@@ -496,6 +532,7 @@ var Reference = []Block{
|
||||
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
|
||||
{Key: "trtllm", Type: "TRTLLMConfig", Doc: "TRTLLM is TensorRT-LLM, which the agent does not install: a node\noffers it only when its interpreter is named here.", Nested: "engines.TRTLLMConfig"},
|
||||
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
|
||||
{Key: "comfyui", Type: "ComfyUIConfig", Doc: "ComfyUI makes pictures; offered by a node whose ComfyUI is named here.", Nested: "engines.ComfyUIConfig"},
|
||||
}},
|
||||
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
|
||||
|
||||
@@ -22,6 +22,7 @@ var Sources = []string{
|
||||
"internal/agent/engines/sglang.go",
|
||||
"internal/agent/engines/trtllm.go",
|
||||
"internal/agent/engines/sim.go",
|
||||
"internal/agent/engines/comfyui.go",
|
||||
"internal/agent/supervisor/supervisor.go",
|
||||
"internal/tlsconf/tlsconf.go",
|
||||
"internal/control/catalog/catalog.go",
|
||||
|
||||
@@ -110,6 +110,23 @@ type Notifier struct {
|
||||
mu sync.Mutex
|
||||
sent map[string]time.Time
|
||||
wg sync.WaitGroup
|
||||
|
||||
claim Claimer
|
||||
}
|
||||
|
||||
// Claimer is where control planes agree which of them sends an event: each
|
||||
// raises the same ones — a budget is spent on every control plane serving
|
||||
// its requests — and only the one that claims an event sends it.
|
||||
type Claimer interface {
|
||||
ClaimNotice(ctx context.Context, key string, at time.Time, window time.Duration) (bool, error)
|
||||
}
|
||||
|
||||
// SetClaimer has events claimed before they are sent. Set before the first
|
||||
// event; nil leaves each process to send its own.
|
||||
func (n *Notifier) SetClaimer(c Claimer) {
|
||||
if n != nil {
|
||||
n.claim = c
|
||||
}
|
||||
}
|
||||
|
||||
type hook struct {
|
||||
@@ -199,6 +216,9 @@ func (n *Notifier) run(ctx context.Context) {
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !n.claimed(ctx, e) {
|
||||
continue
|
||||
}
|
||||
for _, h := range n.hooks {
|
||||
if len(h.Events) > 0 && !slices.Contains(h.Events, e.Kind) {
|
||||
continue
|
||||
@@ -295,3 +315,19 @@ func (n *Notifier) send(ctx context.Context, h hook, e Event, attempts int) erro
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// claimed reports whether this process is the one to send an event. A store
|
||||
// that cannot be asked sends it anyway: told twice beats not told.
|
||||
func (n *Notifier) claimed(ctx context.Context, e Event) bool {
|
||||
if n.claim == nil {
|
||||
return true
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
ok, err := n.claim.ClaimNotice(cctx, e.Kind+"\x00"+e.Subject, e.At, dedupe)
|
||||
if err != nil {
|
||||
n.log.Warn("could not ask the store whether a notification was sent; sending it", "kind", e.Kind, "err", err)
|
||||
return true
|
||||
}
|
||||
return ok
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -167,3 +168,57 @@ func TestTestSaysHowEachAnsweredWithoutTheSecretPath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sharedClaims is a store two control planes share.
|
||||
type sharedClaims struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *sharedClaims) ClaimNotice(_ context.Context, key string, at time.Time, window time.Duration) (bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.err != nil {
|
||||
return false, s.err
|
||||
}
|
||||
if prev, ok := s.last[key]; ok && at.Sub(prev) < window {
|
||||
return false, nil
|
||||
}
|
||||
s.last[key] = at
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func TestTwoControlPlanesSendAnEventOnce(t *testing.T) {
|
||||
// Each control plane serving a team's requests sees its budget spent,
|
||||
// and each kept its own hour of quiet: two behind a balancer sent every
|
||||
// budget event twice.
|
||||
var g got
|
||||
srv := httptest.NewServer(g.handler(0))
|
||||
defer srv.Close()
|
||||
claims := &sharedClaims{last: map[string]time.Time{}}
|
||||
a := newNotifier(t, Config{Host: "ctl1", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
b := newNotifier(t, Config{Host: "ctl2", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
a.SetClaimer(claims)
|
||||
b.SetClaimer(claims)
|
||||
a.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t2", "spent", nil)
|
||||
a.Wait()
|
||||
b.Wait()
|
||||
if len(g.calls) != 2 {
|
||||
t.Errorf("%d sent, want one for each team", len(g.calls))
|
||||
}
|
||||
|
||||
// A store that cannot be asked: sent, rather than lost.
|
||||
var g2 got
|
||||
srv2 := httptest.NewServer(g2.handler(0))
|
||||
defer srv2.Close()
|
||||
c := newNotifier(t, Config{Webhooks: []Webhook{{URL: srv2.URL}}})
|
||||
c.SetClaimer(&sharedClaims{err: errors.New("store down")})
|
||||
c.Notify(NodeLost, "node3", "lost", nil)
|
||||
c.Wait()
|
||||
if len(g2.calls) != 1 {
|
||||
t.Errorf("with the store down, %d sent, want 1", len(g2.calls))
|
||||
}
|
||||
}
|
||||
@@ -511,3 +511,35 @@ func TestSaturationWithNoUsableEndpoints(t *testing.T) {
|
||||
t.Errorf("an empty pool should report full saturation, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABandForBulkWorkWaitsLongerThanTheRest(t *testing.T) {
|
||||
// A knowledge-base import sends far more than the cluster can take at
|
||||
// once; turned away after the 30 s every band shared, its requests had
|
||||
// to be sent again. A band of its own may wait as long as it likes.
|
||||
s := &fixedSat{}
|
||||
s.set(1.0)
|
||||
cfg := testConfig()
|
||||
cfg.QueueTTL = 100 * time.Millisecond
|
||||
for i := range cfg.Bands {
|
||||
if cfg.Bands[i].Name == "bronze" {
|
||||
cfg.Bands[i].QueueTTL = 600 * time.Millisecond
|
||||
}
|
||||
}
|
||||
c := newController(t, cfg, s.fn)
|
||||
|
||||
start := time.Now()
|
||||
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueTimeout) {
|
||||
t.Fatalf("gold: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d > 400*time.Millisecond {
|
||||
t.Errorf("gold waited %v, past the controller's queue_ttl", d)
|
||||
}
|
||||
start = time.Now()
|
||||
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if !errors.Is(err, ErrQueueTimeout) || !strings.Contains(err.Error(), "600ms") {
|
||||
t.Fatalf("bronze: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d < 500*time.Millisecond {
|
||||
t.Errorf("bronze gave up after %v, before its own queue_ttl", d)
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,11 @@ type Band struct {
|
||||
// SaturationCeiling is the pool saturation above which this band waits.
|
||||
// 1.0 means "never hold this band back".
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL, set, is how long this band's requests may wait instead of
|
||||
// the controller's: a band for bulk work — a knowledge-base import, a
|
||||
// backfill — whose requests had rather wait ten minutes behind people
|
||||
// than be turned away after thirty seconds.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Config configures the controller.
|
||||
@@ -86,6 +91,9 @@ func (c *Config) normalize() error {
|
||||
if b.SaturationCeiling <= 0 {
|
||||
b.SaturationCeiling = 1
|
||||
}
|
||||
if b.QueueTTL < 0 {
|
||||
return fmt.Errorf("band %q: queue_ttl %v is negative", b.Name, b.QueueTTL)
|
||||
}
|
||||
}
|
||||
if c.DefaultBand == "" {
|
||||
c.DefaultBand = c.Bands[len(c.Bands)-1].Name
|
||||
@@ -279,12 +287,16 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
ErrQueueFull, c.cfg.QueueCapacityTotal)
|
||||
}
|
||||
|
||||
ttl := c.cfg.QueueTTL
|
||||
if band.QueueTTL > 0 {
|
||||
ttl = band.QueueTTL
|
||||
}
|
||||
w := &waiter{
|
||||
tenant: tenantKey(r),
|
||||
model: r.Model,
|
||||
ready: make(chan struct{}),
|
||||
enqueued: time.Now(),
|
||||
deadline: time.Now().Add(c.cfg.QueueTTL),
|
||||
deadline: time.Now().Add(ttl),
|
||||
}
|
||||
c.queues[idx].push(w)
|
||||
c.mu.Unlock()
|
||||
@@ -312,7 +324,7 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
|
||||
case <-time.After(time.Until(w.deadline)):
|
||||
c.remove(idx, w, "ttl")
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, c.cfg.QueueTTL)
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, ttl)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ func (r *Router) serveLines(ctx context.Context, b *store.Batch, lines []batchLi
|
||||
for _, d := range done {
|
||||
served[d.Line] = true
|
||||
}
|
||||
p, perr := r.batchPrincipalOf(ctx, b.KeyID)
|
||||
p, perr := r.principalOfKey(ctx, b.KeyID, r.batchBand())
|
||||
|
||||
var mu sync.Mutex
|
||||
stopWith := ""
|
||||
@@ -221,9 +221,10 @@ func (r *Router) serveLines(ctx context.Context, b *store.Batch, lines []batchLi
|
||||
return stopWith, nil
|
||||
}
|
||||
|
||||
// batchPrincipalOf is the key a batch was made with, as it stands now: a key
|
||||
// revoked since serves no more of it.
|
||||
func (r *Router) batchPrincipalOf(ctx context.Context, keyID string) (auth.Principal, error) {
|
||||
// principalOfKey is the key a batch or a job was made with, as it stands now:
|
||||
// a key revoked since serves no more of it. band is the admission band the
|
||||
// work goes in; empty is the key's own.
|
||||
func (r *Router) principalOfKey(ctx context.Context, keyID, band string) (auth.Principal, error) {
|
||||
k, err := r.store.KeyByID(ctx, keyID)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
@@ -238,10 +239,13 @@ func (r *Router) batchPrincipalOf(ctx context.Context, keyID string) (auth.Princ
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
if band == "" {
|
||||
band = eff.Band
|
||||
}
|
||||
return auth.Principal{
|
||||
KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID,
|
||||
TeamID: eff.TeamID, OrgID: eff.OrgID,
|
||||
Band: r.batchBand(), RoutingProfile: eff.RoutingProfile, Limits: eff,
|
||||
Band: band, RoutingProfile: eff.RoutingProfile, Limits: eff,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -42,7 +42,7 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
|
||||
// answer than saying the prompt is too large, and it would be done
|
||||
// silently, to a request that was already being quietly moved.
|
||||
canonical, cand := r.resolve(name)
|
||||
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
|
||||
if why := checkContextFits(cand, req.contextTokens(), req.MaxTokens); why != "" {
|
||||
continue
|
||||
}
|
||||
res, err := r.acct.Check(p, canonical, req.estimatedTokens())
|
||||
|
||||
+10
-2
@@ -56,6 +56,8 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
func(w http.ResponseWriter, req *http.Request, _ store.User) {
|
||||
web.ReplyRenderer(r.cfg.MaxBodyBytes)(w, req)
|
||||
}))
|
||||
r.registerImagesPage(mux, pages)
|
||||
r.registerMeshesPage(mux, pages)
|
||||
}
|
||||
|
||||
// chatSend dispatches one turn on behalf of a signed-in person.
|
||||
@@ -256,6 +258,12 @@ func (r *Router) principalFor(req *http.Request, user store.User, asked string)
|
||||
// by name, "big-model" came first and every visitor who pressed Send without
|
||||
// touching the picker got placeholder text.
|
||||
func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
return r.modelsOf(req, user, func(m catalog.Model) bool { return !m.MakesImages() && !m.MakesMeshes() })
|
||||
}
|
||||
|
||||
// modelsOf is the models a person may use from a page: those that make
|
||||
// pictures, or those that write text.
|
||||
func (r *Router) modelsOf(req *http.Request, user store.User, want func(catalog.Model) bool) []web.ChatModel {
|
||||
principal, err := r.principalFor(req, user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -283,7 +291,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if want(entry) && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
@@ -299,7 +307,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if want(entry) && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{
|
||||
Name: name, Simulated: entry.Simulated(), OnDemand: true,
|
||||
})
|
||||
|
||||
@@ -48,12 +48,23 @@ type Info struct {
|
||||
Peer string
|
||||
PeerCluster string
|
||||
PeerTags bool
|
||||
// Workflow is the ComfyUI workflow sent for an endpoint whose engine is
|
||||
// EngineComfyUI: a built-in's name or the workflow's JSON.
|
||||
Workflow string
|
||||
}
|
||||
|
||||
// EngineExternal marks an endpoint that is a provider outside the cluster.
|
||||
// It publishes no metrics, so it is never scraped and never reads as stale.
|
||||
const EngineExternal = "external"
|
||||
|
||||
// EngineComfyUI marks a ComfyUI server. The router speaks ComfyUI's API to
|
||||
// it, not OpenAI's, and it publishes no metrics either.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// Scraped reports whether an engine publishes metrics the router reads. One
|
||||
// that does not is never read as stale for having none.
|
||||
func Scraped(engine string) bool { return engine != EngineExternal && engine != EngineComfyUI }
|
||||
|
||||
// Serves reports whether the replica serves the given model.
|
||||
func (i Info) Serves(model string) bool {
|
||||
return slices.Contains(i.Models, model)
|
||||
@@ -257,7 +268,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
||||
continue
|
||||
}
|
||||
var age time.Duration
|
||||
stale := l.info.Engine != EngineExternal
|
||||
stale := Scraped(l.info.Engine)
|
||||
if !l.scrapedAt.IsZero() {
|
||||
age = now.Sub(l.scrapedAt)
|
||||
stale = age > r.cfg.StaleAfter
|
||||
|
||||
@@ -84,10 +84,12 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
}{
|
||||
{"queue full", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
|
||||
}, 503, ""},
|
||||
}, 503, "10"},
|
||||
// Busy, not broken: a client told when to come back comes back,
|
||||
// where one told nothing — llmgw-kb among them — gives up.
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
}, 504, "10"},
|
||||
{"displaced for a higher band", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
|
||||
}, 503, "5"},
|
||||
|
||||
@@ -62,7 +62,13 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
// The cluster was busy for as long as the band may wait, not broken:
|
||||
// said so a client comes back. Without it llmgw-kb, which retries
|
||||
// only what says when to, counted every such passage as failed.
|
||||
status = http.StatusGatewayTimeout
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrQueueFull):
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
// Turned away for higher-priority work, not for good: said so the
|
||||
// client backs off and comes back rather than giving up.
|
||||
@@ -102,6 +108,8 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/score", r.handleInference("/v1/score"))
|
||||
mux.HandleFunc("POST /v1/audio/transcriptions", r.handleInference("/v1/audio/transcriptions"))
|
||||
mux.HandleFunc("POST /v1/audio/translations", r.handleInference("/v1/audio/translations"))
|
||||
// Pictures, made by ComfyUI from a workflow the catalog names.
|
||||
mux.HandleFunc("POST "+imagesPath, r.handleInference(imagesPath))
|
||||
// Answered by the guardrails' judge model.
|
||||
mux.HandleFunc("POST /v1/moderations", r.handleModerations)
|
||||
// Codex, and whatever else speaks only the Responses API, translated to
|
||||
@@ -121,6 +129,12 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /v1/batches", r.handleListBatches)
|
||||
mux.HandleFunc("GET /v1/batches/{id}", r.handleGetBatch)
|
||||
mux.HandleFunc("POST /v1/batches/{id}/cancel", r.handleCancelBatch)
|
||||
// Jobs: 3D models, which take minutes and are asked after rather than
|
||||
// waited for. The result is a file.
|
||||
mux.HandleFunc("POST "+meshesPath, r.handleCreateJob)
|
||||
mux.HandleFunc("GET "+meshesPath, r.handleListJobs)
|
||||
mux.HandleFunc("GET "+meshesPath+"/{id}", r.handleGetJob)
|
||||
mux.HandleFunc("POST "+meshesPath+"/{id}/cancel", r.handleCancelJob)
|
||||
// Knowledge bases: documents a person's llmgw-kb read and embedded.
|
||||
r.registerKB(mux)
|
||||
// Liveness: this process is up. It says nothing about whether it can
|
||||
@@ -319,7 +333,7 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
exempt := []string{"/health", "/ready", "/leader", "/", "/about", "/_static/", "/_lang/", "/claim/",
|
||||
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/"}
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/org", "/org/", "/setup/"}
|
||||
// The docs are public unless the site said otherwise, and then guarded by
|
||||
// the session check in front of them — or, on a gateway with keys and no
|
||||
// accounts to sign in to, by the key like any other path.
|
||||
@@ -391,7 +405,11 @@ type parsedRequest struct {
|
||||
ProfileOverride string
|
||||
PromptPrefix string
|
||||
PromptTokens int
|
||||
MaxTokens int
|
||||
// ContextTokens is what has to fit in one context: the prompt, except
|
||||
// for embeddings, where each input is embedded on its own and the
|
||||
// longest one is what a context must hold.
|
||||
ContextTokens int
|
||||
MaxTokens int
|
||||
|
||||
// ThreadID is the conversation this request continues, empty for the
|
||||
// stateless path every request took before threads existed.
|
||||
@@ -473,11 +491,19 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// A picture asked of a model that writes text, or the other way
|
||||
// round, is refused before it waits for either.
|
||||
if why := r.imageRouteRefusal(path, parsed.Model, parsed.UpstreamBody); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
|
||||
// Before authentication work and before admission: a request that
|
||||
// cannot fit should not consume a quota reservation or a queue slot on
|
||||
// its way to being refused.
|
||||
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
|
||||
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
|
||||
if why := checkContextFits(entry, parsed.contextTokens(), parsed.MaxTokens); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
@@ -736,6 +762,8 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
|
||||
// the truth than charging nothing.
|
||||
u.PromptTokens = int64(req.PromptTokens)
|
||||
}
|
||||
u.Images = int64(res.outcome.Images)
|
||||
u.Meshes = int64(res.outcome.Meshes)
|
||||
r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
|
||||
r.acct.Record(req.reservation, u)
|
||||
}
|
||||
@@ -825,6 +853,12 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
|
||||
// plan §14). Measured over the whole prompt, not the routing prefix —
|
||||
// see promptBytes.
|
||||
p.PromptTokens = promptBytes(&b) / 4
|
||||
p.ContextTokens = p.PromptTokens
|
||||
if path == "/v1/embeddings" {
|
||||
if longest, ok := longestInput(b.Input); ok {
|
||||
p.ContextTokens = longest / 4
|
||||
}
|
||||
}
|
||||
|
||||
// Tags are the gateway's: an engine has no use for them, and its logs
|
||||
// are no place for what a client wrote there.
|
||||
@@ -897,6 +931,29 @@ func promptPrefix(b *inferenceBody) string {
|
||||
// bounded head of the conversation; a token estimate wants all of it, and using
|
||||
// one for the other meant TPM limits undercounted exactly the requests that
|
||||
// consume the most, and nothing could tell whether a prompt fit in a context.
|
||||
// longestInput is the length of an embeddings request's longest input, when
|
||||
// input is a list of strings: a batch of texts, each its own context.
|
||||
func longestInput(raw json.RawMessage) (int, bool) {
|
||||
var list []string
|
||||
if len(raw) == 0 || raw[0] != '[' || json.Unmarshal(raw, &list) != nil || len(list) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
n := 0
|
||||
for _, s := range list {
|
||||
n = max(n, len(s))
|
||||
}
|
||||
return n, true
|
||||
}
|
||||
|
||||
// contextTokens is what has to fit in one context; requests built inside
|
||||
// the router set only PromptTokens.
|
||||
func (p *parsedRequest) contextTokens() int {
|
||||
if p.ContextTokens > 0 {
|
||||
return p.ContextTokens
|
||||
}
|
||||
return p.PromptTokens
|
||||
}
|
||||
|
||||
func promptBytes(b *inferenceBody) int {
|
||||
n := 0
|
||||
add := func(raw json.RawMessage) {
|
||||
@@ -1086,9 +1143,12 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
|
||||
if m.Description != "" {
|
||||
entry["description"] = m.Description
|
||||
}
|
||||
if m.License != nil {
|
||||
entry["license"] = m.License
|
||||
}
|
||||
entry["capabilities"] = map[string]bool{
|
||||
"tools": m.Capabilities.Tools, "vision": m.Capabilities.Vision,
|
||||
"embeddings": m.Capabilities.Embeddings,
|
||||
"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(), "meshes": m.MakesMeshes(),
|
||||
}
|
||||
data = append(data, entry)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,307 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// /v1/images/generations, OpenAI's shape, made by ComfyUI: the request goes
|
||||
// through the same checks as any other — key, allowlist, limits, admission,
|
||||
// guardrails on its prompt — and only at the replica is it turned into a
|
||||
// workflow, queued and waited on, and its pictures sent back as base64.
|
||||
|
||||
const imagesPath = "/v1/images/generations"
|
||||
|
||||
// imageRequest is what a request for pictures asks. Beyond OpenAI's fields,
|
||||
// those a diffusion model has and DALL·E does not: seed, steps,
|
||||
// negative_prompt and guidance.
|
||||
type imageRequest struct {
|
||||
Prompt string `json:"prompt"`
|
||||
N int `json:"n"`
|
||||
Size string `json:"size"`
|
||||
ResponseFormat string `json:"response_format"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
Guidance *float64 `json:"guidance"`
|
||||
|
||||
width, height int
|
||||
}
|
||||
|
||||
// The bounds of a request. A FLUX latent is 1/16 of the picture, so a side
|
||||
// that is not a multiple of 16 is not one it can make; the rest keep one
|
||||
// request from holding a card for the length of a film.
|
||||
const (
|
||||
maxImages = 4
|
||||
minSide = 256
|
||||
maxSide = 2048
|
||||
maxSteps = 100
|
||||
defaultSide = 1024
|
||||
)
|
||||
|
||||
func parseImageRequest(body []byte) (imageRequest, error) {
|
||||
var in imageRequest
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
return in, err
|
||||
}
|
||||
if strings.TrimSpace(in.Prompt) == "" {
|
||||
return in, errors.New("prompt is required")
|
||||
}
|
||||
switch {
|
||||
case in.N == 0:
|
||||
in.N = 1
|
||||
case in.N < 0 || in.N > maxImages:
|
||||
return in, fmt.Errorf("n is between 1 and %d", maxImages)
|
||||
}
|
||||
switch in.ResponseFormat {
|
||||
case "", "b64_json":
|
||||
case "url":
|
||||
// A URL would have to be served by somebody and kept for some time,
|
||||
// and a picture made from a prompt is the prompt's owner's: it is
|
||||
// not kept here.
|
||||
return in, errors.New(`response_format "url" is not offered: the picture comes back as b64_json and is kept nowhere`)
|
||||
default:
|
||||
return in, fmt.Errorf("response_format %q is not b64_json", in.ResponseFormat)
|
||||
}
|
||||
in.width, in.height = defaultSide, defaultSide
|
||||
if s := strings.TrimSpace(in.Size); s != "" && s != "auto" {
|
||||
w, h, ok := strings.Cut(strings.ToLower(s), "x")
|
||||
wi, err1 := strconv.Atoi(w)
|
||||
hi, err2 := strconv.Atoi(h)
|
||||
if !ok || err1 != nil || err2 != nil {
|
||||
return in, fmt.Errorf("size %q is not WIDTHxHEIGHT", s)
|
||||
}
|
||||
in.width, in.height = wi, hi
|
||||
}
|
||||
for _, side := range []int{in.width, in.height} {
|
||||
if side < minSide || side > maxSide || side%16 != 0 {
|
||||
return in, fmt.Errorf("size %dx%d: each side is a multiple of 16 from %d to %d", in.width, in.height, minSide, maxSide)
|
||||
}
|
||||
}
|
||||
if in.Steps < 0 || in.Steps > maxSteps {
|
||||
return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
|
||||
}
|
||||
if in.Guidance != nil && (*in.Guidance <= 0 || *in.Guidance > 30) {
|
||||
return in, errors.New("guidance is above 0 and at most 30")
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// imageRouteRefusal says why a request cannot be answered at the path it was
|
||||
// sent to, before it takes a place in a queue: a picture asked of a model
|
||||
// that writes text, or text of one that makes pictures. Empty when it can,
|
||||
// and for a model the catalog does not know, which a hand-configured
|
||||
// endpoint may serve as it likes.
|
||||
func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
m, known := r.modelMap()[model]
|
||||
if path == meshesPath {
|
||||
if known && !m.MakesMeshes() {
|
||||
if m.MakesImages() {
|
||||
return fmt.Sprintf("%s makes pictures, not 3D models: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return fmt.Sprintf("%s does not make 3D models", m.Name)
|
||||
}
|
||||
if _, err := parseMeshRequest(body); err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if path == imagesPath {
|
||||
if known && m.MakesMeshes() && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s makes 3D models, not pictures: ask for one at %s, as a job", m.Name, meshesPath)
|
||||
}
|
||||
if known && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s does not make pictures", m.Name)
|
||||
}
|
||||
if _, err := parseImageRequest(body); err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if known && m.ServesComfyUI() {
|
||||
if m.MakesMeshes() {
|
||||
return fmt.Sprintf("%s makes 3D models: ask for one at %s, as a job", m.Name, meshesPath)
|
||||
}
|
||||
return fmt.Sprintf("%s makes pictures: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// generateImages answers a request at a ComfyUI replica. It returns what the
|
||||
// proxy would: a failure of this replica that another may not have is
|
||||
// retryable, and nothing is written until the pictures are all in hand.
|
||||
func (r *Router) generateImages(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
if req.Path == meshesPath {
|
||||
return r.generateMeshes(ctx, w, req, ep, body)
|
||||
}
|
||||
if req.Path != imagesPath {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s and %s only", ep.ID, imagesPath, meshesPath)
|
||||
}
|
||||
in, err := parseImageRequest(body)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
var upstream struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &upstream)
|
||||
tpl, err := comfy.Load(ep.Workflow)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
seed := comfy.RandomSeed()
|
||||
if in.Seed != nil {
|
||||
seed = *in.Seed
|
||||
}
|
||||
p := comfy.Params{
|
||||
Checkpoint: upstream.Model, Prompt: in.Prompt, NegativePrompt: in.NegativePrompt,
|
||||
Width: in.width, Height: in.height, N: in.N, Seed: seed, Steps: in.Steps,
|
||||
}
|
||||
if in.Guidance != nil {
|
||||
p.Guidance = *in.Guidance
|
||||
}
|
||||
workflow, err := tpl.Fill(p)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
|
||||
if t := r.cfg.Timeouts.Total; t > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, t)
|
||||
defer cancel()
|
||||
}
|
||||
start := time.Now()
|
||||
images, err := r.comfy.Generate(ctx, ep.URL, ep.APIKey, workflow)
|
||||
if err != nil {
|
||||
var ce *comfy.Error
|
||||
switch {
|
||||
case errors.As(err, &ce):
|
||||
return proxy.Outcome{Retryable: ce.Retryable}, err
|
||||
case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
|
||||
return proxy.Outcome{}, proxy.ErrClientGone
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return proxy.Outcome{}, fmt.Errorf("%s made no picture within %s", ep.ID, r.cfg.Timeouts.Total)
|
||||
}
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
|
||||
data := make([]map[string]any, 0, len(images))
|
||||
for _, im := range images {
|
||||
data = append(data, map[string]any{"b64_json": base64.StdEncoding.EncodeToString(im.Data)})
|
||||
}
|
||||
out, err := json.Marshal(map[string]any{"created": time.Now().Unix(), "data": data})
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
// The seed that made them, so a picture liked can be made again.
|
||||
w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(out)
|
||||
took := time.Since(start)
|
||||
return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(out)), TTFT: took, Decode: took,
|
||||
Images: len(images)}, nil
|
||||
}
|
||||
|
||||
// ctxCauseIsClient reports whether a context ended because the client went
|
||||
// away, rather than because a deadline set here passed.
|
||||
func ctxCauseIsClient(ctx context.Context) bool {
|
||||
return !errors.Is(context.Cause(ctx), context.DeadlineExceeded)
|
||||
}
|
||||
|
||||
// registerImagesPage mounts the page that makes pictures, beside the chat
|
||||
// and for the same reasons: a model tried without a client, with the
|
||||
// browser holding a session and never a key, and the request sent down the
|
||||
// one path every API request takes.
|
||||
func (r *Router) registerImagesPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetImages(func() bool {
|
||||
for _, m := range r.modelMap() {
|
||||
if m.MakesImages() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
mux.HandleFunc("GET /images", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
contexts, _ := r.chatContexts(req.Context(), user)
|
||||
current, err := r.pickContext(req.Context(), user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
current = ""
|
||||
}
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Images(user, r.modelsOf(req, user, catalog.Model.MakesImages), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /images/generate", pages.Guard(r.imagesSend))
|
||||
}
|
||||
|
||||
// imagesSend makes pictures on behalf of a signed-in person.
|
||||
func (r *Router) imagesSend(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
http.Error(w, "could not read the request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
Prompt string `json:"prompt"`
|
||||
Size string `json:"size"`
|
||||
N int `json:"n"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
// Context is what the pictures are charged to, checked as the
|
||||
// chat's is.
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Model == "" {
|
||||
http.Error(w, "a model and a prompt are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
principal, err := r.principalFor(req, user, in.Context)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
// Rebuilt from the fields the page has, so it can set nothing else.
|
||||
out := map[string]any{"model": in.Model, "prompt": in.Prompt, "size": in.Size, "n": in.N}
|
||||
if in.Seed != nil {
|
||||
out["seed"] = *in.Seed
|
||||
}
|
||||
if in.Steps > 0 {
|
||||
out["steps"] = in.Steps
|
||||
}
|
||||
if s := strings.TrimSpace(in.NegativePrompt); s != "" {
|
||||
out["negative_prompt"] = s
|
||||
}
|
||||
upstream, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
http.Error(w, "could not build the request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
next := req.Clone(auth.WithPrincipal(req.Context(), principal))
|
||||
next.Body = io.NopCloser(bytes.NewReader(upstream))
|
||||
next.ContentLength = int64(len(upstream))
|
||||
next.Header.Set("Content-Type", "application/json")
|
||||
next.Header.Del(knowledgeHeader)
|
||||
r.handleInference(imagesPath)(w, next)
|
||||
}
|
||||
@@ -0,0 +1,277 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// imageRouter has a ComfyUI model, flux, beside a text model, and returns
|
||||
// what the ComfyUI server was sent.
|
||||
func imageRouter(t *testing.T) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
return imageRouterWith(t, nil)
|
||||
}
|
||||
|
||||
func imageRouterWith(t *testing.T, edit func(*config.Server)) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
var mu sync.Mutex
|
||||
var queued map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/metrics":
|
||||
t.Error("ComfyUI was scraped for metrics")
|
||||
case "/prompt":
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
mu.Lock()
|
||||
queued = b.Prompt
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p1"}`)
|
||||
case "/history/p1":
|
||||
mu.Lock()
|
||||
n := 1
|
||||
if latent, ok := queued["4"].(map[string]any); ok {
|
||||
n = int(latent["inputs"].(map[string]any)["batch_size"].(float64))
|
||||
}
|
||||
mu.Unlock()
|
||||
var images []string
|
||||
for i := range n {
|
||||
images = append(images, fmt.Sprintf(`{"filename":"%d.png","type":"temp"}`, i))
|
||||
}
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[%s]}}}}`,
|
||||
strings.Join(images, ","))
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png-bytes")
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
|
||||
External: &catalog.External{URL: srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
|
||||
if edit != nil {
|
||||
edit(&cfg)
|
||||
}
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:flux", URL: srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
return rt, &queued, &mu
|
||||
}
|
||||
|
||||
func postImages(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestAPictureIsMadeByComfyUIAndComesBackAsBase64(t *testing.T) {
|
||||
rt, queued, mu := imageRouter(t)
|
||||
rec := postImages(t, rt.Handler(), imagesPath, map[string]any{
|
||||
"model": "flux", "prompt": "a lighthouse at dusk", "size": "768x512", "seed": 7})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var out struct {
|
||||
Data []struct {
|
||||
B64 string `json:"b64_json"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil || len(out.Data) != 1 {
|
||||
t.Fatalf("body %s: %v", rec.Body.String(), err)
|
||||
}
|
||||
if b, _ := base64.StdEncoding.DecodeString(out.Data[0].B64); string(b) != "png-bytes" {
|
||||
t.Errorf("picture = %q", b)
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "7" {
|
||||
t.Errorf("seed header = %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
wf, _ := json.Marshal(*queued)
|
||||
for _, want := range []string{`"ckpt_name":"flux1-schnell-fp8.safetensors"`, `"text":"a lighthouse at dusk"`,
|
||||
`"width":768`, `"height":512`, `"seed":7`} {
|
||||
if !strings.Contains(string(wf), want) {
|
||||
t.Errorf("%s not in the workflow sent: %s", want, wf)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAndTextAreAskedForAtTheirOwnPaths(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
h := rt.Handler()
|
||||
for name, c := range map[string]struct {
|
||||
path string
|
||||
body map[string]any
|
||||
want string
|
||||
}{
|
||||
"text of flux": {"/v1/chat/completions", map[string]any{"model": "flux", "messages": []any{}}, "ask for them at " + imagesPath},
|
||||
"picture of chat": {imagesPath, map[string]any{"model": "chat", "prompt": "x"}, "does not make pictures"},
|
||||
"no prompt": {imagesPath, map[string]any{"model": "flux"}, "prompt is required"},
|
||||
"odd size": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "size": "1000x1000"}, "multiple of 16"},
|
||||
"url wanted": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "response_format": "url"}, "b64_json"},
|
||||
"too many at once": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "n": 9}, "n is between"},
|
||||
} {
|
||||
rec := postImages(t, h, c.path, c.body)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), c.want) {
|
||||
t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelSaysItMakesPictures(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if !strings.Contains(rec.Body.String(), `"images":true`) {
|
||||
t.Errorf("/v1/models: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsCheckedInTheCatalog(t *testing.T) {
|
||||
ok := catalog.Model{Name: "f", Source: "external", External: &catalog.External{
|
||||
URL: "http://10.0.0.147:8188", Model: "flux1-dev.safetensors", API: catalog.APIComfyUI, Workflow: "flux-dev"}}
|
||||
if _, err := catalog.New([]catalog.Model{ok}); err != nil {
|
||||
t.Fatalf("refused: %v", err)
|
||||
}
|
||||
for name, e := range map[string]catalog.External{
|
||||
"unknown api": {URL: "http://a:1", Model: "m", API: "a1111"},
|
||||
"unknown workflow": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Workflow: "sdxl"},
|
||||
"workflow, openai": {URL: "http://a:1", Model: "m", Workflow: "flux-dev"},
|
||||
"comfyui from peer": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Peer: "other"},
|
||||
} {
|
||||
e := e
|
||||
if _, err := catalog.New([]catalog.Model{{Name: "f", Source: "external", External: &e}}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAreChargedByThePictureAndRecorded(t *testing.T) {
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"flux": {PromptPer1k: 1000, PerImage: 50000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-1", Hash: sum[:], Prefix: "bbbb", Name: "t",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
b, _ := json.Marshal(map[string]any{"model": "flux", "prompt": "three posters", "n": 3})
|
||||
req := httptest.NewRequest(http.MethodPost, imagesPath, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
id := rec.Header().Get("X-Request-Id")
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, err := rt.store.UsageByRequest(t.Context(), id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok {
|
||||
if u.Images != 3 {
|
||||
t.Errorf("recorded %d pictures, want 3", u.Images)
|
||||
}
|
||||
if want := (u.PromptTokens*1000+999)/1000 + 3*50000; u.CostMicros != want {
|
||||
t.Errorf("charged %d micros, want %d (%d prompt tokens and three pictures)", u.CostMicros, want, u.PromptTokens)
|
||||
}
|
||||
return
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("no usage was recorded")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThePicturesPageOffersPictureModelsAndTheChatTheOthers(t *testing.T) {
|
||||
e := newFakeEngine(t, "chat")
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
})
|
||||
rt.SetEndpoints(t.Context(), append(rt.reg.Infos(), endpoint.Info{ID: "c", URL: e.srv.URL, Models: []string{"chat"}}))
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
user := store.User{ID: "usr-1", ProjectID: project}
|
||||
names := func(pictures bool) []string {
|
||||
var out []string
|
||||
var models []web.ChatModel
|
||||
if pictures {
|
||||
models = rt.modelsOf(httptest.NewRequest("GET", "/", nil), user, catalog.Model.MakesImages)
|
||||
} else {
|
||||
models = rt.modelsFor(httptest.NewRequest("GET", "/", nil), user)
|
||||
}
|
||||
for _, m := range models {
|
||||
out = append(out, m.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := names(false); len(got) != 1 || got[0] != "chat" {
|
||||
t.Errorf("the chat offers %v", got)
|
||||
}
|
||||
if got := names(true); len(got) != 1 || got[0] != "flux" {
|
||||
t.Errorf("the pictures page offers %v", got)
|
||||
}
|
||||
|
||||
// The page's request is rebuilt: a field it has no business sending
|
||||
// does not reach the model.
|
||||
body := `{"model":"flux","prompt":"a kite","size":"512x512","n":1,"seed":3,"response_format":"url"}`
|
||||
rec := httptest.NewRecorder()
|
||||
rt.imagesSend(rec, httptest.NewRequest("POST", "/images/generate", strings.NewReader(body)), user)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "b64_json") {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "3" {
|
||||
t.Errorf("seed %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,209 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Jobs: a request that takes minutes, answered with an id and asked after.
|
||||
//
|
||||
// A 3D model from a picture takes minutes on a card that first has to be
|
||||
// emptied of whatever else was on it. That is not a request to hold a socket
|
||||
// open for (activate.go says why), so it is a job: queued with 202, its
|
||||
// status and progress read back, its result downloaded as a file. The
|
||||
// request is kept as the client sent it and replayed, by whichever gateway
|
||||
// holds the job's lease, through the same path a client's request takes —
|
||||
// key, allowlist, limits, admission, guardrails, accounting — so a job is
|
||||
// charged and judged like any request, tagged job=<id>.
|
||||
|
||||
// meshesPath is where 3D models are asked for. The synchronous path behind
|
||||
// it is served to the job worker only; clients get a job.
|
||||
const meshesPath = "/v1/3d/generations"
|
||||
|
||||
const (
|
||||
purposeJobOutput = "job_output"
|
||||
// jobWindow is how long a job may take, queue included, before it is
|
||||
// given up on.
|
||||
jobWindow = 24 * time.Hour
|
||||
)
|
||||
|
||||
func (r *Router) jobJSON(ctx context.Context, j store.Job) map[string]any {
|
||||
out := map[string]any{
|
||||
"id": j.ID, "object": "job", "endpoint": j.Endpoint, "model": j.Model, "status": j.Status,
|
||||
"progress": nil, "output_file_id": strOrNil(j.OutputFileID), "content_type": strOrNil(j.ContentType),
|
||||
"error": nil, "created_at": j.CreatedAt.Unix(), "started_at": unixOrNil(j.StartedAt),
|
||||
"finished_at": unixOrNil(j.FinishedAt), "expires_at": unixOrNil(j.ExpiresAt), "usage": nil,
|
||||
}
|
||||
if j.Progress != "" && json.Valid([]byte(j.Progress)) {
|
||||
out["progress"] = json.RawMessage(j.Progress)
|
||||
}
|
||||
if j.Error != "" {
|
||||
out["error"] = map[string]string{"message": j.Error}
|
||||
}
|
||||
// The terms what it makes comes under, beside the result: whoever ships
|
||||
// the mesh answers for them.
|
||||
if m, ok := r.modelMap()[j.Model]; ok && m.License != nil {
|
||||
out["license"] = m.License
|
||||
}
|
||||
// What the work cost, from the usage record the replayed request left
|
||||
// under the job's id.
|
||||
if store.JobDone(j.Status) && r.store != nil {
|
||||
if u, ok, err := r.store.UsageByRequest(ctx, j.ID); err == nil && ok {
|
||||
out["usage"] = map[string]int64{"prompt_tokens": u.PromptTokens, "completion_tokens": u.CompletionTokens,
|
||||
"images": u.Images, "meshes": u.Meshes}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// handleCreateJob queues a request for a 3D model. Refused now: no key, a
|
||||
// model that is not there or not allowed, a request the model cannot serve.
|
||||
// Refused when it is served, and said as the job's error: limits, the
|
||||
// guardrails, a model that will not come up.
|
||||
func (r *Router) handleCreateJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "could not read the request: "+err.Error())
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if in.Model == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "model is required")
|
||||
return
|
||||
}
|
||||
if _, known := r.modelMap()[in.Model]; !known || r.hiddenModel(in.Model) {
|
||||
writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
|
||||
fmt.Sprintf("the model %q does not exist", in.Model))
|
||||
return
|
||||
}
|
||||
if !r.permits(p, in.Model) {
|
||||
auth.WriteForbiddenModel(w, in.Model)
|
||||
return
|
||||
}
|
||||
if why := r.imageRouteRefusal(meshesPath, in.Model, body); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
j := store.Job{ID: newID("job_"), Owner: ownerOf(p), KeyID: p.KeyID, Endpoint: meshesPath, Model: in.Model,
|
||||
Input: body, Status: store.JobQueued, CreatedAt: now, ExpiresAt: now.Add(jobWindow)}
|
||||
if err := r.store.PutJob(req.Context(), j); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
r.wakeJobs()
|
||||
writeJSON(w, http.StatusAccepted, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
|
||||
// ownedJob is a job the caller owns, or a 404 said.
|
||||
func (r *Router) ownedJob(w http.ResponseWriter, req *http.Request, owner string) (store.Job, bool) {
|
||||
j, err := r.store.JobByID(req.Context(), req.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) || err == nil && j.Owner != owner {
|
||||
writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "job_not_found",
|
||||
fmt.Sprintf("no job %q", req.PathValue("id")))
|
||||
return store.Job{}, false
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return store.Job{}, false
|
||||
}
|
||||
return j, true
|
||||
}
|
||||
|
||||
func (r *Router) handleGetJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if j, ok := r.ownedJob(w, req, ownerOf(p)); ok {
|
||||
writeJSON(w, http.StatusOK, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Router) handleCancelJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
j, ok := r.ownedJob(w, req, ownerOf(p))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
j, err := r.store.CancelJob(req.Context(), j.ID, time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
r.wakeJobs()
|
||||
writeJSON(w, http.StatusOK, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
|
||||
func (r *Router) handleListJobs(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
limit, _ := strconv.Atoi(req.URL.Query().Get("limit"))
|
||||
if limit <= 0 || limit > 100 {
|
||||
limit = 20
|
||||
}
|
||||
js, err := r.store.Jobs(req.Context(), ownerOf(p), req.URL.Query().Get("after"), limit+1)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
more := len(js) > limit
|
||||
if more {
|
||||
js = js[:limit]
|
||||
}
|
||||
data := make([]map[string]any, 0, len(js))
|
||||
for _, j := range js {
|
||||
data = append(data, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
out := map[string]any{"object": "list", "data": data, "has_more": more, "first_id": nil, "last_id": nil}
|
||||
if len(js) > 0 {
|
||||
out["first_id"], out["last_id"] = js[0].ID, js[len(js)-1].ID
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// errorMessage is what an answer said went wrong: the message of an OpenAI
|
||||
// error body, or the body itself, kept short.
|
||||
func errorMessage(body []byte) string {
|
||||
var e struct {
|
||||
Error struct {
|
||||
Message string `json:"message"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(body, &e) == nil && e.Error.Message != "" {
|
||||
return e.Error.Message
|
||||
}
|
||||
s := strings.TrimSpace(string(body))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
if s == "" {
|
||||
return "no reason given"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,285 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// jobComfy is a ComfyUI that makes one picture a workflow, slowly when
|
||||
// asked: stall holds every workflow until it is cleared, and the paths the
|
||||
// router took to stop one are counted.
|
||||
type jobComfy struct {
|
||||
srv *httptest.Server
|
||||
stall atomic.Bool
|
||||
prompts atomic.Int32
|
||||
cancelled atomic.Int32
|
||||
}
|
||||
|
||||
func newJobComfy(t *testing.T) *jobComfy {
|
||||
t.Helper()
|
||||
c := &jobComfy{}
|
||||
c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.URL.Path == "/prompt":
|
||||
c.prompts.Add(1)
|
||||
fmt.Fprint(w, `{"prompt_id":"p1"}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
if c.stall.Load() {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[{"filename":"0.png","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png-bytes")
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost, r.URL.Path == "/interrupt":
|
||||
c.cancelled.Add(1)
|
||||
fmt.Fprint(w, `{}`)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[0,"p1"]],"queue_pending":[]}`)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(c.srv.Close)
|
||||
return c
|
||||
}
|
||||
|
||||
// jobRouter is a gateway with the store, a key, a text model and a ComfyUI
|
||||
// model priced by the picture.
|
||||
func jobRouter(t *testing.T, c *jobComfy) (*Router, string) {
|
||||
t.Helper()
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
|
||||
External: &catalog.External{URL: c.srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"flux": {PerImage: 5000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:flux", URL: c.srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "platform")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "gateway")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("a", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-1", Hash: sum[:], Prefix: "aaaa", Name: "test",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rt, secret
|
||||
}
|
||||
|
||||
// queueJob puts a job in the store as the create endpoint would, for an
|
||||
// endpoint that serves today: until a model makes 3D models, the worker is
|
||||
// tried with pictures.
|
||||
func queueJob(t *testing.T, rt *Router, id, status string, body string) store.Job {
|
||||
t.Helper()
|
||||
now := time.Now()
|
||||
j := store.Job{ID: id, Owner: "project:" + mustProjectOf(t, rt, "key-1"), KeyID: "key-1", Endpoint: imagesPath,
|
||||
Model: "flux", Input: []byte(body), Status: status, CreatedAt: now, ExpiresAt: now.Add(jobWindow)}
|
||||
if err := rt.store.PutJob(t.Context(), j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
func waitJob(t *testing.T, h http.Handler, secret, id string, until func(map[string]any) bool) map[string]any {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
_, j, raw := call(t, h, secret, http.MethodGet, meshesPath+"/"+id, nil)
|
||||
if until(j) {
|
||||
return j
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the job did not get there: %s", raw)
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func jobDone(j map[string]any) bool { return store.JobDone(fmt.Sprint(j["status"])) }
|
||||
|
||||
func TestAJobIsServedItsResultKeptAndItsWorkCharged(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_1", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
|
||||
got := waitJob(t, h, secret, j.ID, jobDone)
|
||||
if got["status"] != "completed" || got["content_type"] != "application/json" || got["error"] != nil {
|
||||
t.Fatalf("%v", got)
|
||||
}
|
||||
if got["started_at"] == nil || got["finished_at"] == nil {
|
||||
t.Errorf("times: %v", got)
|
||||
}
|
||||
code, _, out := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string)+"/content", nil)
|
||||
if code != 200 || !strings.Contains(out, `"b64_json":"cG5nLWJ5dGVz"`) {
|
||||
t.Errorf("result %d: %s", code, out)
|
||||
}
|
||||
_, f, _ := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string), nil)
|
||||
if f["purpose"] != purposeJobOutput || f["filename"] != "job_1.json" || f["expires_at"] == nil {
|
||||
t.Errorf("file %v", f)
|
||||
}
|
||||
|
||||
// Charged like any request, by the picture, and findable by the job.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, _ := rt.store.UsageByRequest(t.Context(), j.ID)
|
||||
if ok {
|
||||
if u.Tags["job"] != j.ID || u.Images != 1 || u.CostMicros != 5000 {
|
||||
t.Errorf("usage %+v", u)
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the job left no usage")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
got = waitJob(t, h, secret, j.ID, func(j map[string]any) bool { return j["usage"] != nil })
|
||||
if u := got["usage"].(map[string]any); u["images"] != 1.0 {
|
||||
t.Errorf("usage shown %v", u)
|
||||
}
|
||||
_, list, _ := call(t, h, secret, http.MethodGet, meshesPath, nil)
|
||||
if d := list["data"].([]any); len(d) != 1 {
|
||||
t.Errorf("listed %v", list)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobIsRefusedBeforeItIsQueuedWhenItCannotBeServed(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
code, _, raw := call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"model": "chat", "prompt": "a cube"})
|
||||
if code != 400 || !strings.Contains(raw, "does not make 3D models") {
|
||||
t.Errorf("text model: %d %s", code, raw)
|
||||
}
|
||||
code, _, raw = call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"model": "nope", "prompt": "a cube"})
|
||||
if code != 404 || !strings.Contains(raw, "model_not_found") {
|
||||
t.Errorf("unknown model: %d %s", code, raw)
|
||||
}
|
||||
code, _, raw = call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"prompt": "a cube"})
|
||||
if code != 400 || !strings.Contains(raw, "model is required") {
|
||||
t.Errorf("no model: %d %s", code, raw)
|
||||
}
|
||||
if code, _, _ := call(t, h, "", http.MethodPost, meshesPath, map[string]any{"model": "chat"}); code != 401 {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if _, list, _ := call(t, h, secret, http.MethodGet, meshesPath, nil); len(list["data"].([]any)) != 0 {
|
||||
t.Errorf("a refused job was queued: %v", list)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARunningJobIsCancelledAndTheEngineToldToStop(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
c.stall.Store(true)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_slow", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
waitJob(t, h, secret, j.ID, func(j map[string]any) bool { return j["status"] == "running" })
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for c.prompts.Load() == 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
|
||||
code, got, raw := call(t, h, secret, http.MethodPost, meshesPath+"/"+j.ID+"/cancel", nil)
|
||||
if code != 200 || got["status"] != "cancelling" {
|
||||
t.Fatalf("cancel: %d %s", code, raw)
|
||||
}
|
||||
got = waitJob(t, h, secret, j.ID, jobDone)
|
||||
if got["status"] != "cancelled" || got["output_file_id"] != nil {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
if c.cancelled.Load() == 0 {
|
||||
t.Error("ComfyUI was not told to stop")
|
||||
}
|
||||
|
||||
// One not yet taken ends at once, and is never sent.
|
||||
before := c.prompts.Load()
|
||||
j2 := queueJob(t, rt, "job_never", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
_, got, _ = call(t, h, secret, http.MethodPost, meshesPath+"/"+j2.ID+"/cancel", nil)
|
||||
if got["status"] != "cancelled" {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
rt.wakeJobs()
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
if c.prompts.Load() != before {
|
||||
t.Error("a cancelled job was sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobWhoseGatewayDiedIsTakenByAnother(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
// As a gateway that died mid-job left it: running, its lease run out.
|
||||
j := queueJob(t, rt, "job_orphan", store.JobRunning, `{"model":"flux","prompt":"a cube"}`)
|
||||
j.StartedAt = time.Now().Add(-time.Minute)
|
||||
_ = rt.store.UpdateJob(t.Context(), j)
|
||||
rt.wakeJobs()
|
||||
if got := waitJob(t, h, secret, j.ID, jobDone); got["status"] != "completed" {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
|
||||
// A refusal of the request is the job's end, said.
|
||||
bad := queueJob(t, rt, "job_bad", store.JobQueued, `{"model":"flux","prompt":"a cube","n":99}`)
|
||||
rt.wakeJobs()
|
||||
got := waitJob(t, h, secret, bad.ID, jobDone)
|
||||
if got["status"] != "failed" || !strings.Contains(fmt.Sprint(got["error"]), "n is between 1 and 4") {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobsAreTheirOwners(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_mine", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
waitJob(t, h, secret, j.ID, jobDone)
|
||||
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "other")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "t")
|
||||
project, _ := rt.store.CreateProject(t.Context(), team, "p")
|
||||
other := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(other))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "other",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code, _, _ := call(t, h, other, http.MethodGet, meshesPath+"/"+j.ID, nil); code != 404 {
|
||||
t.Errorf("another project read a job: %d", code)
|
||||
}
|
||||
if code, _, _ := call(t, h, other, http.MethodPost, meshesPath+"/"+j.ID+"/cancel", nil); code != 404 {
|
||||
t.Errorf("another project cancelled a job: %d", code)
|
||||
}
|
||||
_, mine, _ := call(t, h, secret, http.MethodGet, meshesPath+"/"+j.ID, nil)
|
||||
if code, _, _ := call(t, h, other, http.MethodGet, "/v1/files/"+mine["output_file_id"].(string)+"/content", nil); code != 404 {
|
||||
t.Errorf("another project read a job's result: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The worker that serves jobs. Any gateway on the store may run one: a job
|
||||
// is worked on by whichever holds its lease, and one whose gateway died is
|
||||
// taken by another when the lease runs out.
|
||||
|
||||
const (
|
||||
jobLease = 2 * time.Minute
|
||||
jobPoll = 5 * time.Second
|
||||
// jobCancelCheck is how often a running job looks for a cancel.
|
||||
jobCancelCheck = 2 * time.Second
|
||||
// jobMaxResult bounds what one job may answer: a mesh, not a film.
|
||||
jobMaxResult = 256 << 20
|
||||
)
|
||||
|
||||
// wakeJobs has the worker look for work now rather than at its next poll.
|
||||
func (r *Router) wakeJobs() {
|
||||
select {
|
||||
case r.jobWake <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Router) runJobs(ctx context.Context) {
|
||||
host, _ := os.Hostname()
|
||||
owner := fmt.Sprintf("%s:%d:%s", host, os.Getpid(), newID(""))
|
||||
t := time.NewTicker(jobPoll)
|
||||
defer t.Stop()
|
||||
for {
|
||||
for {
|
||||
now := time.Now()
|
||||
j, ok, err := r.store.ClaimJob(ctx, owner, now, now.Add(jobLease))
|
||||
if err != nil {
|
||||
if ctx.Err() == nil {
|
||||
r.log.Warn("could not look for jobs to serve", "err", err)
|
||||
}
|
||||
break
|
||||
}
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
r.workJob(ctx, owner, j)
|
||||
}
|
||||
// A job's record outlives its file by nothing: both go at the
|
||||
// retention.
|
||||
_, _, _, days := r.batchCfg()
|
||||
if n, err := r.store.DeleteJobsBefore(ctx, time.Now().AddDate(0, 0, -days)); err == nil && n > 0 {
|
||||
r.log.Info("jobs past their retention forgotten", "jobs", n)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-r.jobWake:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// workJob takes one job as far as it goes, holding its lease the while.
|
||||
func (r *Router) workJob(ctx context.Context, owner string, j store.Job) {
|
||||
jctx, stop := context.WithCancel(ctx)
|
||||
defer stop()
|
||||
defer func() { _ = r.store.ReleaseJob(context.WithoutCancel(ctx), j.ID, owner) }()
|
||||
// The lease kept while working; lost, the work stops and another
|
||||
// gateway has it.
|
||||
go func() {
|
||||
t := time.NewTicker(jobLease / 4)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-jctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if held, err := r.store.RenewJobLease(jctx, j.ID, owner, time.Now().Add(jobLease)); err == nil && !held {
|
||||
stop()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
if err := r.serveJob(jctx, &j); err != nil && jctx.Err() == nil {
|
||||
r.log.Warn("a job stopped with an error; it is picked up again", "job", j.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// serveJob replays the job's request and keeps what came back. A failure
|
||||
// of the gateway's own (the store unreachable) is returned, and the job is
|
||||
// picked up again; a refusal of the request is the job's end.
|
||||
func (r *Router) serveJob(ctx context.Context, j *store.Job) error {
|
||||
now := time.Now()
|
||||
switch {
|
||||
case j.Status == store.JobCancelling:
|
||||
j.Status, j.FinishedAt = store.JobCancelled, now
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
case now.After(j.ExpiresAt):
|
||||
j.Status, j.FinishedAt = store.JobExpired, now
|
||||
j.Error = fmt.Sprintf("not finished within %s", jobWindow)
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
if j.Status == store.JobQueued {
|
||||
j.Status, j.StartedAt = store.JobRunning, now
|
||||
if err := r.store.UpdateJob(ctx, *j); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// A job found running was another gateway's, which died with it. It is
|
||||
// run again from the start: the engine's queue does not outlive the
|
||||
// request that was waiting on it.
|
||||
fail := func(msg string) error {
|
||||
j.Status, j.Error, j.FinishedAt = store.JobFailed, msg, time.Now()
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
p, err := r.principalOfJob(ctx, *j)
|
||||
if err != nil {
|
||||
return fail("the key or account this job was made with can no longer be used: " + err.Error())
|
||||
}
|
||||
|
||||
// A cancel asked for while it runs ends the request, which takes the
|
||||
// work out of the engine's queue.
|
||||
rctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
go func() {
|
||||
t := time.NewTicker(jobCancelCheck)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-rctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if cur, err := r.store.JobByID(rctx, j.ID); err == nil && cur.Status == store.JobCancelling {
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
handler := r.handleInference(j.Endpoint)
|
||||
for attempt := 1; ; attempt++ {
|
||||
hreq, err := http.NewRequestWithContext(auth.WithPrincipal(rctx, p), http.MethodPost, j.Endpoint, bytes.NewReader(j.Input))
|
||||
if err != nil {
|
||||
return fail(err.Error())
|
||||
}
|
||||
hreq.Header.Set("Content-Type", "application/json")
|
||||
hreq.Header.Set("X-Request-Id", j.ID)
|
||||
hreq.Header.Set(tagsHeader, "job="+j.ID)
|
||||
w := &bufferWriter{header: http.Header{}, max: jobMaxResult}
|
||||
handler(w, hreq)
|
||||
if rctx.Err() != nil {
|
||||
if ctx.Err() != nil {
|
||||
// The lease was lost or the gateway is stopping: another
|
||||
// gateway's to pick up.
|
||||
return ctx.Err()
|
||||
}
|
||||
j.Status, j.FinishedAt = store.JobCancelled, time.Now()
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
status := w.status
|
||||
if status == 0 {
|
||||
status = http.StatusOK
|
||||
}
|
||||
// Waited out, for as long as the job has: a model loading, a rate
|
||||
// limit, a queue full. Nobody is holding a connection, which is the
|
||||
// point of a job. Not a spent budget, which says no Retry-After
|
||||
// because waiting does not help.
|
||||
if wait := w.header.Get("Retry-After"); wait != "" && (status == http.StatusTooManyRequests || status >= 500) &&
|
||||
time.Now().Before(j.ExpiresAt) {
|
||||
secs, _ := strconv.Atoi(wait)
|
||||
select {
|
||||
case <-rctx.Done():
|
||||
continue // said above
|
||||
case <-time.After(time.Duration(max(secs, 1)) * time.Second):
|
||||
}
|
||||
continue
|
||||
}
|
||||
if status >= 300 {
|
||||
return fail(errorMessage(w.buf.Bytes()))
|
||||
}
|
||||
ct := w.header.Get("Content-Type")
|
||||
if ct == "" {
|
||||
ct = http.DetectContentType(w.buf.Bytes())
|
||||
}
|
||||
_, _, _, days := r.batchCfg()
|
||||
done := time.Now()
|
||||
f := store.File{ID: newID("file-"), Owner: j.Owner, Purpose: purposeJobOutput,
|
||||
Filename: j.ID + extensionFor(ct), Bytes: int64(w.buf.Len()), CreatedAt: done, ExpiresAt: done.AddDate(0, 0, days)}
|
||||
if err := r.store.PutFile(ctx, f, w.buf.Bytes()); err != nil {
|
||||
return err
|
||||
}
|
||||
// Said done while it may still be said: the last word heard was the
|
||||
// node running, and a finished job saying it is running reads as
|
||||
// one that stopped halfway.
|
||||
_ = r.store.SetJobProgress(ctx, j.ID, `{"state":"done","percent":100}`, "")
|
||||
j.Status, j.OutputFileID, j.ContentType, j.FinishedAt = store.JobCompleted, f.ID, ct, done
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
}
|
||||
|
||||
// extensionFor names a result's file by what it is.
|
||||
func extensionFor(contentType string) string {
|
||||
ct, _, _ := strings.Cut(contentType, ";")
|
||||
switch strings.TrimSpace(ct) {
|
||||
case "application/json":
|
||||
return ".json"
|
||||
case "model/gltf-binary":
|
||||
return ".glb"
|
||||
case "model/gltf+json":
|
||||
return ".gltf"
|
||||
case "model/obj":
|
||||
return ".obj"
|
||||
case "image/png":
|
||||
return ".png"
|
||||
case "image/jpeg":
|
||||
return ".jpg"
|
||||
case "image/webp":
|
||||
return ".webp"
|
||||
}
|
||||
return ".bin"
|
||||
}
|
||||
|
||||
// principalOfJob is who a job acts for, as they stand now: the key it was
|
||||
// made with, or — for one made from the portal — the person whose session
|
||||
// made it, while they may still use the gateway and the project the job is
|
||||
// charged to.
|
||||
func (r *Router) principalOfJob(ctx context.Context, j store.Job) (auth.Principal, error) {
|
||||
uid, session := strings.CutPrefix(j.KeyID, "session:")
|
||||
if !session {
|
||||
return r.principalOfKey(ctx, j.KeyID, "")
|
||||
}
|
||||
u, err := r.store.UserByID(ctx, uid)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
if u.State != store.UserActive {
|
||||
return auth.Principal{}, fmt.Errorf("the account that made it is %s", u.State)
|
||||
}
|
||||
project, _ := strings.CutPrefix(j.Owner, "project:")
|
||||
if !strings.HasPrefix(j.Owner, "project:") {
|
||||
project = ""
|
||||
} else if ok, err := r.store.CanUseProject(ctx, uid, project); err != nil || !ok {
|
||||
return auth.Principal{}, errors.New("its account is no longer a member of the project it is charged to")
|
||||
}
|
||||
eff, err := r.store.ResolveFor(ctx, j.KeyID, uid, project)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
return auth.Principal{KeyID: j.KeyID, KeyPrefix: "session", UserID: uid, ProjectID: eff.ProjectID,
|
||||
TeamID: eff.TeamID, OrgID: eff.OrgID, Band: eff.Band, RoutingProfile: eff.RoutingProfile, Limits: eff}, nil
|
||||
}
|
||||
@@ -3,7 +3,6 @@ package router
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
@@ -60,6 +59,9 @@ type bufferWriter struct {
|
||||
header http.Header
|
||||
status int
|
||||
buf bytes.Buffer
|
||||
// max bounds what is kept; 1 MiB when unset, which is a judge's answer
|
||||
// many times over. A job's result is a file, and sets more.
|
||||
max int
|
||||
}
|
||||
|
||||
func (b *bufferWriter) Header() http.Header { return b.header }
|
||||
@@ -72,8 +74,12 @@ func (b *bufferWriter) Write(p []byte) (int, error) {
|
||||
if b.status == 0 {
|
||||
b.status = http.StatusOK
|
||||
}
|
||||
if b.buf.Len()+len(p) > 1<<20 {
|
||||
return 0, errors.New("a judge's answer over 1 MiB")
|
||||
limit := b.max
|
||||
if limit <= 0 {
|
||||
limit = 1 << 20
|
||||
}
|
||||
if b.buf.Len()+len(p) > limit {
|
||||
return 0, fmt.Errorf("an answer over %d bytes", limit)
|
||||
}
|
||||
return b.buf.Write(p)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,383 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// 3D models from a picture, made by ComfyUI. Clients ask for them as a job
|
||||
// (jobs.go); the job worker replays the request through the ordinary path,
|
||||
// and at a ComfyUI replica it arrives here: the picture is put in ComfyUI's
|
||||
// input directory, the model's workflow is filled in with its name and run,
|
||||
// and the mesh it saves comes back as the answer — the file the job keeps.
|
||||
|
||||
// meshRequest is what a request for a 3D model asks. The picture is the
|
||||
// whole of the prompt: base64, or a data URL.
|
||||
type meshRequest struct {
|
||||
Model string `json:"model"`
|
||||
Image string `json:"image"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
// Resolution is the model's own grid, where it has a choice: TripoSR's
|
||||
// marching cubes take 256 by default, TRELLIS.2 512 to 1536. Zero leaves
|
||||
// it to the workflow.
|
||||
Resolution int `json:"resolution"`
|
||||
|
||||
picture []byte
|
||||
ext string
|
||||
}
|
||||
|
||||
// maxPicture bounds the picture a mesh is made from.
|
||||
const maxPicture = 16 << 20
|
||||
|
||||
func parseMeshRequest(body []byte) (meshRequest, error) {
|
||||
var in meshRequest
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
return in, err
|
||||
}
|
||||
s := strings.TrimSpace(in.Image)
|
||||
if s == "" {
|
||||
return in, errors.New("image is required: the picture to make a 3D model of, base64 or a data URL")
|
||||
}
|
||||
if strings.HasPrefix(s, "data:") {
|
||||
_, after, ok := strings.Cut(s, ";base64,")
|
||||
if !ok {
|
||||
return in, errors.New("image is a data URL without base64 in it")
|
||||
}
|
||||
s = after
|
||||
}
|
||||
if base64.StdEncoding.DecodedLen(len(s)) > maxPicture+3 {
|
||||
return in, fmt.Errorf("image is past %d MB", maxPicture>>20)
|
||||
}
|
||||
b, err := base64.StdEncoding.DecodeString(s)
|
||||
if err != nil {
|
||||
return in, errors.New("image is not base64: " + err.Error())
|
||||
}
|
||||
switch http.DetectContentType(b) {
|
||||
case "image/png":
|
||||
in.ext = ".png"
|
||||
case "image/jpeg":
|
||||
in.ext = ".jpg"
|
||||
case "image/webp":
|
||||
in.ext = ".webp"
|
||||
default:
|
||||
return in, errors.New("image is not a PNG, JPEG or WebP picture")
|
||||
}
|
||||
in.picture = b
|
||||
if in.Steps < 0 || in.Steps > maxSteps {
|
||||
return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
|
||||
}
|
||||
if r := in.Resolution; r != 0 && (r < 64 || r > 2048 || r%32 != 0) {
|
||||
return in, fmt.Errorf("resolution %d is not a multiple of 32 from 64 to 2048", r)
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// generateMeshes answers a request for a 3D model at a ComfyUI replica. It
|
||||
// writes the mesh itself, with its type, which is what the job keeps.
|
||||
func (r *Router) generateMeshes(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
in, err := parseMeshRequest(body)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
tpl, err := comfy.Load(ep.Workflow)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
if t := r.cfg.Timeouts.Total; t > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, t)
|
||||
defer cancel()
|
||||
}
|
||||
start := time.Now()
|
||||
fail := func(err error) (proxy.Outcome, error) {
|
||||
var ce *comfy.Error
|
||||
switch {
|
||||
case errors.As(err, &ce):
|
||||
return proxy.Outcome{Retryable: ce.Retryable}, err
|
||||
case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
|
||||
return proxy.Outcome{}, proxy.ErrClientGone
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return proxy.Outcome{}, fmt.Errorf("%s made no 3D model within %s", ep.ID, r.cfg.Timeouts.Total)
|
||||
}
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
// A name of its own: two requests' pictures must not overwrite each
|
||||
// other in the replica's input directory.
|
||||
rnd := make([]byte, 8)
|
||||
_, _ = rand.Read(rnd)
|
||||
name, err := r.comfy.Upload(ctx, ep.URL, ep.APIKey, "llmgw-"+hex.EncodeToString(rnd)+in.ext, in.picture)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
seed := comfy.RandomSeed()
|
||||
if in.Seed != nil {
|
||||
seed = *in.Seed
|
||||
}
|
||||
var upstream struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &upstream)
|
||||
p := comfy.Params{Checkpoint: upstream.Model, Image: name, Seed: seed, Steps: in.Steps, Extra: map[string]any{}}
|
||||
if in.Resolution > 0 {
|
||||
p.Extra["resolution"] = in.Resolution
|
||||
}
|
||||
workflow, err := tpl.Fill(p)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
files, err := r.comfy.GenerateWatch(ctx, ep.URL, ep.APIKey, workflow, r.jobProgress(ctx, req))
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
var mesh *comfy.Image
|
||||
for i := range files {
|
||||
if isMeshType(files[i].ContentType) {
|
||||
mesh = &files[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if mesh == nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the workflow saved no mesh (it saved %d other files): "+
|
||||
"its last node should save a .glb", ep.ID, len(files))
|
||||
}
|
||||
data := mesh.Data
|
||||
if mesh.ContentType == "model/obj" {
|
||||
glb, err := objToGLB(data)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the mesh it saved: %w", ep.ID, err)
|
||||
}
|
||||
data, mesh.ContentType = glb, "model/gltf-binary"
|
||||
}
|
||||
m := r.modelMap()[req.Model]
|
||||
if m.License != nil && mesh.ContentType == "model/gltf-binary" {
|
||||
if with, err := glbWithLicense(data, *m.License); err == nil {
|
||||
data = with
|
||||
} else {
|
||||
// Refused rather than sent without it: a mesh travels without
|
||||
// the API, and the notice is what the terms ask to travel too.
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the licence could not be written into the mesh: %w", ep.ID, err)
|
||||
}
|
||||
}
|
||||
w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
|
||||
if m.License != nil {
|
||||
w.Header().Set("x-llmgw-license", m.License.Name)
|
||||
}
|
||||
w.Header().Set("Content-Type", mesh.ContentType)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(data)
|
||||
took := time.Since(start)
|
||||
return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(data)), TTFT: took, Decode: took,
|
||||
Meshes: 1}, nil
|
||||
}
|
||||
|
||||
func isMeshType(ct string) bool {
|
||||
return strings.HasPrefix(ct, "model/") || ct == "application/ply"
|
||||
}
|
||||
|
||||
// glbWithLicense is a GLB with the licence in asset.extras.license, where a
|
||||
// file's terms travel with it: glTF's own place for what an application
|
||||
// adds, kept by every tool that loads and saves the file.
|
||||
func glbWithLicense(glb []byte, lic catalog.License) ([]byte, error) {
|
||||
if len(glb) < 20 || string(glb[:4]) != "glTF" || binary.LittleEndian.Uint32(glb[4:]) != 2 {
|
||||
return nil, errors.New("not a glTF 2 binary")
|
||||
}
|
||||
n := int(binary.LittleEndian.Uint32(glb[12:]))
|
||||
if string(glb[16:20]) != "JSON" || 20+n > len(glb) {
|
||||
return nil, errors.New("its first chunk is not its JSON")
|
||||
}
|
||||
var doc map[string]any
|
||||
d := json.NewDecoder(bytes.NewReader(glb[20 : 20+n]))
|
||||
d.UseNumber() // every number as it was written
|
||||
if err := d.Decode(&doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
asset, _ := doc["asset"].(map[string]any)
|
||||
if asset == nil {
|
||||
asset = map[string]any{"version": "2.0"}
|
||||
}
|
||||
extras, _ := asset["extras"].(map[string]any)
|
||||
if extras == nil {
|
||||
extras = map[string]any{}
|
||||
}
|
||||
l := map[string]any{"name": lic.Name}
|
||||
if lic.URL != "" {
|
||||
l["url"] = lic.URL
|
||||
}
|
||||
if lic.Notice != "" {
|
||||
l["notice"] = lic.Notice
|
||||
}
|
||||
extras["license"] = l
|
||||
asset["extras"] = extras
|
||||
doc["asset"] = asset
|
||||
js, err := json.Marshal(doc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// A chunk is padded to four bytes, the JSON one with spaces.
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
rest := glb[20+n:]
|
||||
out := make([]byte, 0, 20+len(js)+len(rest))
|
||||
out = append(out, "glTF"...)
|
||||
out = binary.LittleEndian.AppendUint32(out, 2)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(20+len(js)+len(rest))) //nolint:gosec // bounded by the file read
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(len(js))) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "JSON"...)
|
||||
out = append(out, js...)
|
||||
return append(out, rest...), nil
|
||||
}
|
||||
|
||||
// jobProgress is where a job's request says how far along it is: into the
|
||||
// job, for whoever asks after it. Nil for a request that is no job's — and
|
||||
// for one that only says it is: the job must be the same key's, so nobody
|
||||
// writes into another's progress by naming it in a tag.
|
||||
func (r *Router) jobProgress(ctx context.Context, req *parsedRequest) func(comfy.Progress) {
|
||||
id := req.Tags["job"]
|
||||
if id == "" || r.store == nil || id != req.RequestID {
|
||||
return nil
|
||||
}
|
||||
j, err := r.store.JobByID(ctx, id)
|
||||
if err != nil || j.KeyID != req.KeyID {
|
||||
return nil
|
||||
}
|
||||
return func(p comfy.Progress) {
|
||||
b, err := json.Marshal(p)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := r.store.SetJobProgress(context.WithoutCancel(ctx), id, string(b), p.PromptID); err != nil {
|
||||
r.log.Warn("a job's progress could not be kept", "job", id, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// registerMeshesPage mounts the page that makes 3D models, beside the
|
||||
// pictures page and for the same reasons: a session, never a key, and the
|
||||
// request sent down the path an API client's takes — here, a job.
|
||||
func (r *Router) registerMeshesPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetMeshes(func() bool {
|
||||
for _, m := range r.modelMap() {
|
||||
if m.MakesMeshes() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
mux.HandleFunc("GET /3d", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
contexts, _ := r.chatContexts(req.Context(), user)
|
||||
current, err := r.pickContext(req.Context(), user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
current = ""
|
||||
}
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Meshes(user, r.modelsOf(req, user, catalog.Model.MakesMeshes), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /3d/generate", pages.Guard(r.meshesSend))
|
||||
mux.HandleFunc("GET /3d/jobs/{id}", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
if next, ok := r.asPerson(w, req, user, req.URL.Query().Get("context")); ok {
|
||||
r.handleGetJob(w, next)
|
||||
}
|
||||
}))
|
||||
// The mesh itself, from where the page is, for its viewer to load.
|
||||
mux.HandleFunc("GET /3d/jobs/{id}/file", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
next, ok := r.asPerson(w, req, user, req.URL.Query().Get("context"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, _ := auth.FromContext(next.Context())
|
||||
j, ok := r.ownedJob(w, next, ownerOf(p))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if j.OutputFileID == "" {
|
||||
http.Error(w, "the job has made nothing yet", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
b, err := r.store.FileContent(req.Context(), j.OutputFileID)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", j.ContentType)
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("inline; filename=%q", j.ID+".glb"))
|
||||
_, _ = w.Write(b)
|
||||
}))
|
||||
}
|
||||
|
||||
// asPerson is a page's request as the signed-in person's, in the place they
|
||||
// chose: what the API handlers read a key's principal from.
|
||||
func (r *Router) asPerson(w http.ResponseWriter, req *http.Request, user store.User, context string) (*http.Request, bool) {
|
||||
p, err := r.principalFor(req, user, context)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return nil, false
|
||||
}
|
||||
return req.WithContext(auth.WithPrincipal(req.Context(), p)), true
|
||||
}
|
||||
|
||||
// meshesSend asks for a 3D model on behalf of a signed-in person.
|
||||
func (r *Router) meshesSend(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
http.Error(w, "could not read the request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
Image string `json:"image"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Resolution int `json:"resolution"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Model == "" {
|
||||
http.Error(w, "a model and a picture are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Rebuilt from the fields the page has, so it can set nothing else.
|
||||
out := map[string]any{"model": in.Model, "image": in.Image}
|
||||
if in.Seed != nil {
|
||||
out["seed"] = *in.Seed
|
||||
}
|
||||
if in.Resolution > 0 {
|
||||
out["resolution"] = in.Resolution
|
||||
}
|
||||
upstream, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
http.Error(w, "could not build the request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
next, ok := r.asPerson(w, req, user, in.Context)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
next = next.Clone(next.Context())
|
||||
next.Body = io.NopCloser(bytes.NewReader(upstream))
|
||||
next.ContentLength = int64(len(upstream))
|
||||
next.Header.Set("Content-Type", "application/json")
|
||||
r.handleCreateJob(w, next)
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"image"
|
||||
"image/png"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// testGLB is the smallest glTF binary: an asset, and a binary chunk after
|
||||
// its JSON that must come through untouched.
|
||||
func testGLB(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
js := []byte(`{"asset":{"version":"2.0","generator":"test"},"buffers":[{"byteLength":4}]}`)
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
bin := []byte{1, 2, 3, 4}
|
||||
var b []byte
|
||||
b = append(b, "glTF"...)
|
||||
b = binary.LittleEndian.AppendUint32(b, 2)
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(12+8+len(js)+8+len(bin)))
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(len(js)))
|
||||
b = append(b, "JSON"...)
|
||||
b = append(b, js...)
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(len(bin)))
|
||||
b = append(b, "BIN\x00"...)
|
||||
return append(b, bin...)
|
||||
}
|
||||
|
||||
// meshComfy is a ComfyUI that takes a picture and saves a mesh, as a 3D
|
||||
// node's SaveGLB does: under "3d", not "images".
|
||||
type meshComfy struct {
|
||||
srv *httptest.Server
|
||||
mu sync.Mutex
|
||||
uploaded []string
|
||||
workflow map[string]any
|
||||
}
|
||||
|
||||
func newMeshComfy(t *testing.T) *meshComfy {
|
||||
glb := testGLB(t)
|
||||
c := &meshComfy{}
|
||||
c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/upload/image":
|
||||
f, h, err := r.FormFile("image")
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 400)
|
||||
return
|
||||
}
|
||||
b, _ := io.ReadAll(f)
|
||||
c.mu.Lock()
|
||||
c.uploaded = append(c.uploaded, h.Filename+":"+http.DetectContentType(b))
|
||||
c.mu.Unlock()
|
||||
fmt.Fprintf(w, `{"name":%q,"subfolder":"","type":"input"}`, h.Filename)
|
||||
case "/prompt":
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
c.mu.Lock()
|
||||
c.workflow = b.Prompt
|
||||
c.mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"m1"}`)
|
||||
case "/history/m1":
|
||||
fmt.Fprint(w, `{"m1":{"status":{"status_str":"success","completed":true},"outputs":{
|
||||
"3":{"text":["a preview"]},"4":{"3d":[{"filename":"mesh_00001_.glb","subfolder":"3d","type":"output"}]}}}}`)
|
||||
case "/view":
|
||||
// Served as ComfyUI serves a file it has no type for.
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
_, _ = w.Write(glb)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(c.srv.Close)
|
||||
return c
|
||||
}
|
||||
|
||||
const meshWorkflow = `{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"2": {"class_type": "Trellis2ImageToMesh", "inputs": {"image": ["1", 0], "seed": "{{seed}}", "resolution": "{{resolution|512}}"}},
|
||||
"4": {"class_type": "SaveGLB", "inputs": {"mesh": ["2", 0], "filename_prefix": "3d/mesh"}}
|
||||
}`
|
||||
|
||||
func testPicture(t *testing.T) string {
|
||||
var b bytes.Buffer
|
||||
_ = png.Encode(&b, image.NewRGBA(image.Rect(0, 0, 8, 8)))
|
||||
return base64.StdEncoding.EncodeToString(b.Bytes())
|
||||
}
|
||||
|
||||
func meshRouter(t *testing.T, c *meshComfy) (*Router, string) {
|
||||
t.Helper()
|
||||
cfg := baseConfig()
|
||||
mit := &catalog.License{Name: "MIT", URL: "https://github.com/microsoft/TRELLIS.2/blob/main/LICENSE",
|
||||
Notice: "Made with TRELLIS.2 (MIT)."}
|
||||
cfg.Models = []catalog.Model{
|
||||
{Name: "trellis2", Source: "external", License: mit, External: &catalog.External{URL: c.srv.URL,
|
||||
Model: "trellis2", API: catalog.APIComfyUI, Output: catalog.OutputMeshes, Workflow: "w.json"}},
|
||||
{Name: "flux", Source: "external", External: &catalog.External{URL: c.srv.URL,
|
||||
Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}},
|
||||
}
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"trellis2": {PerMesh: 250000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:trellis2", URL: c.srv.URL, Models: []string{"trellis2"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "trellis2", Workflow: meshWorkflow},
|
||||
{ID: "external:flux", URL: c.srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "studio")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "art")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "props")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("m", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-m", Hash: sum[:], Prefix: "mmmm", Name: "art",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rt, secret
|
||||
}
|
||||
|
||||
func TestA3DModelIsMadeFromAPictureWithItsLicenceInside(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, secret := meshRouter(t, c)
|
||||
h := rt.Handler()
|
||||
|
||||
code, j, raw := call(t, h, secret, http.MethodPost, meshesPath,
|
||||
map[string]any{"model": "trellis2", "image": "data:image/png;base64," + testPicture(t), "seed": 11, "resolution": 1024})
|
||||
if code != http.StatusAccepted {
|
||||
t.Fatalf("%d %s", code, raw)
|
||||
}
|
||||
got := waitJob(t, h, secret, j["id"].(string), jobDone)
|
||||
if got["status"] != "completed" || got["content_type"] != "model/gltf-binary" {
|
||||
t.Fatalf("job %v", got)
|
||||
}
|
||||
if p, _ := got["progress"].(map[string]any); p["state"] != "done" {
|
||||
t.Errorf("a finished job says %v", got["progress"])
|
||||
}
|
||||
if lic, _ := got["license"].(map[string]any); lic["name"] != "MIT" {
|
||||
t.Errorf("the job does not say the licence: %v", got["license"])
|
||||
}
|
||||
code, _, file := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string)+"/content", nil)
|
||||
if code != http.StatusOK || !strings.HasPrefix(file, "glTF") {
|
||||
t.Fatalf("the result is not the mesh: %d %q", code, file[:min(len(file), 40)])
|
||||
}
|
||||
b := []byte(file)
|
||||
n := binary.LittleEndian.Uint32(b[12:])
|
||||
var doc struct {
|
||||
Asset struct {
|
||||
Generator string `json:"generator"`
|
||||
Extras struct {
|
||||
License catalog.License `json:"license"`
|
||||
} `json:"extras"`
|
||||
} `json:"asset"`
|
||||
}
|
||||
if err := json.Unmarshal(b[20:20+n], &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if doc.Asset.Extras.License.Notice != "Made with TRELLIS.2 (MIT)." || doc.Asset.Generator != "test" {
|
||||
t.Errorf("asset %+v: the notice is not in the mesh, or what was there is lost", doc.Asset)
|
||||
}
|
||||
if int(binary.LittleEndian.Uint32(b[8:])) != len(b) || !bytes.HasSuffix(b, []byte{1, 2, 3, 4}) {
|
||||
t.Error("the mesh's length or its binary chunk did not survive")
|
||||
}
|
||||
|
||||
c.mu.Lock()
|
||||
wf, _ := json.Marshal(c.workflow)
|
||||
uploaded := strings.Join(c.uploaded, ",")
|
||||
c.mu.Unlock()
|
||||
if !strings.HasSuffix(uploaded, ".png:image/png") || !strings.HasPrefix(uploaded, "llmgw-") {
|
||||
t.Errorf("uploaded %q", uploaded)
|
||||
}
|
||||
name := strings.TrimSuffix(uploaded, ":image/png")
|
||||
for _, want := range []string{`"image":"` + name + `"`, `"seed":11`, `"resolution":1024`} {
|
||||
if !strings.Contains(string(wf), want) {
|
||||
t.Errorf("%s not in the workflow sent: %s", want, wf)
|
||||
}
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, _ := rt.store.UsageByRequest(t.Context(), j["id"].(string))
|
||||
if ok {
|
||||
if u.Meshes != 1 || u.CostMicros < 250000 {
|
||||
t.Errorf("recorded %d meshes, %d micros; want one at its price", u.Meshes, u.CostMicros)
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("no usage recorded")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestA3DModelIsAskedOfAModelThatMakesThem(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, secret := meshRouter(t, c)
|
||||
h := rt.Handler()
|
||||
for name, c := range map[string]struct {
|
||||
path string
|
||||
body map[string]any
|
||||
want string
|
||||
}{
|
||||
"a mesh of flux": {meshesPath, map[string]any{"model": "flux", "image": testPicture(t)}, "makes pictures, not 3D models"},
|
||||
"no picture": {meshesPath, map[string]any{"model": "trellis2"}, "image is required"},
|
||||
"not a picture": {meshesPath, map[string]any{"model": "trellis2", "image": base64.StdEncoding.EncodeToString([]byte("hello"))}, "not a PNG"},
|
||||
"an odd resolution": {meshesPath, map[string]any{"model": "trellis2", "image": testPicture(t), "resolution": 700}, "multiple of 32"},
|
||||
"a picture of trellis2": {imagesPath, map[string]any{"model": "trellis2", "prompt": "a cube"}, "makes 3D models, not pictures"},
|
||||
"a chat with trellis2": {"/v1/chat/completions", map[string]any{"model": "trellis2", "messages": []any{}}, "makes 3D models"},
|
||||
} {
|
||||
code, _, raw := call(t, h, secret, http.MethodPost, c.path, c.body)
|
||||
if code != http.StatusBadRequest || !strings.Contains(raw, c.want) {
|
||||
t.Errorf("%s: %d %s", name, code, raw)
|
||||
}
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
if !strings.Contains(rec.Body.String(), `"meshes":true`) || !strings.Contains(rec.Body.String(), `"license":{"name":"MIT"`) {
|
||||
t.Errorf("/v1/models: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOBJLeavesTheGatewayAsAGLB(t *testing.T) {
|
||||
// TripoSR saves an OBJ with a colour after each position, and a quad
|
||||
// here stands for the polygons other nodes write.
|
||||
obj := "# made by a test\nv 0 0 0 1 0 0\nv 1 0 0 0 1 0\nv 1 1 0 0 0 1\nv 0 1 0 1 1 1\nf 1 2 3 4\nf -4 -2 -1\n"
|
||||
glb, err := objToGLB([]byte(obj))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(glb[:4]) != "glTF" || int(binary.LittleEndian.Uint32(glb[8:])) != len(glb) {
|
||||
t.Fatal("not a glTF binary of its own length")
|
||||
}
|
||||
n := binary.LittleEndian.Uint32(glb[12:])
|
||||
var doc struct {
|
||||
Accessors []struct {
|
||||
Count int `json:"count"`
|
||||
Max []float64 `json:"max"`
|
||||
} `json:"accessors"`
|
||||
Meshes []struct {
|
||||
Primitives []struct {
|
||||
Attributes map[string]int `json:"attributes"`
|
||||
} `json:"primitives"`
|
||||
} `json:"meshes"`
|
||||
}
|
||||
if err := json.Unmarshal(glb[20:20+n], &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if doc.Accessors[0].Count != 4 || doc.Accessors[1].Count != 9 || doc.Accessors[0].Max[1] != 1 {
|
||||
t.Errorf("accessors %+v: four vertices and three triangles wanted", doc.Accessors)
|
||||
}
|
||||
if _, ok := doc.Meshes[0].Primitives[0].Attributes["COLOR_0"]; !ok {
|
||||
t.Error("the vertices' colours were lost")
|
||||
}
|
||||
if _, err := glbWithLicense(glb, catalog.License{Name: "MIT"}); err != nil {
|
||||
t.Errorf("the licence cannot be written into it: %v", err)
|
||||
}
|
||||
for _, bad := range []string{"v 0 0 0\n", "v 0 0 0\nf 1 2 3\n", "f x y z\n"} {
|
||||
if _, err := objToGLB([]byte(bad)); err == nil {
|
||||
t.Errorf("%q made a GLB", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestThe3DPageMakesAJobAndShowsItsMeshToItsOwnerOnly(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, _ := meshRouter(t, c)
|
||||
project := mustProjectOf(t, rt, "key-m")
|
||||
user := store.User{ID: "usr-1", Email: "ann@studio.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
if err := rt.store.CreateUser(t.Context(), user); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := rt.store.AddMembership(t.Context(), user.ID, store.ScopeProject, project, "member", "test", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "other")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "t")
|
||||
otherProject, _ := rt.store.CreateProject(t.Context(), team, "p")
|
||||
stranger := store.User{ID: "usr-2", Email: "bob@else.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
_ = rt.store.CreateUser(t.Context(), stranger)
|
||||
_ = rt.store.AddMembership(t.Context(), stranger.ID, store.ScopeProject, otherProject, "member", "test", time.Now())
|
||||
|
||||
body := `{"model":"trellis2","image":"` + testPicture(t) + `","resolution":256,"seed":5}`
|
||||
rec := httptest.NewRecorder()
|
||||
rt.meshesSend(rec, httptest.NewRequest("POST", "/3d/generate", strings.NewReader(body)), user)
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var j map[string]any
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &j)
|
||||
id := j["id"].(string)
|
||||
rt.wakeJobs()
|
||||
|
||||
get := func(u store.User, path string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest("GET", path, nil)
|
||||
req.SetPathValue("id", id)
|
||||
rec := httptest.NewRecorder()
|
||||
next, ok := rt.asPerson(rec, req, u, "")
|
||||
if !ok {
|
||||
return rec
|
||||
}
|
||||
if strings.HasSuffix(path, "/file") {
|
||||
p, _ := auth.FromContext(next.Context())
|
||||
jj, ok := rt.ownedJob(rec, next, ownerOf(p))
|
||||
if ok {
|
||||
b, _ := rt.store.FileContent(next.Context(), jj.OutputFileID)
|
||||
rec.Header().Set("Content-Type", jj.ContentType)
|
||||
_, _ = rec.Write(b)
|
||||
}
|
||||
return rec
|
||||
}
|
||||
rt.handleGetJob(rec, next)
|
||||
return rec
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
rec := get(user, "/3d/jobs/"+id)
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &j)
|
||||
if j["status"] == "completed" {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("not made: %s", rec.Body.String())
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if rec := get(user, "/3d/jobs/"+id+"/file"); !strings.HasPrefix(rec.Body.String(), "glTF") {
|
||||
t.Errorf("the mesh: %q", rec.Body.String()[:min(20, rec.Body.Len())])
|
||||
}
|
||||
if rec := get(stranger, "/3d/jobs/"+id); rec.Code != http.StatusNotFound {
|
||||
t.Errorf("another project's person read the job: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// objToGLB turns a Wavefront OBJ — what a 3D node saves when it saves no
|
||||
// GLB — into a glTF binary: one mesh of its vertices, their colours where
|
||||
// the file gives them (TripoSR's do, after each position), and its faces
|
||||
// as triangles. A GLB is the one file a game engine, a browser and the
|
||||
// licence notice all take, so every mesh leaves the gateway as one.
|
||||
func objToGLB(obj []byte) ([]byte, error) {
|
||||
var pos, col []float32
|
||||
var idx []uint32
|
||||
colours := true
|
||||
sc := bufio.NewScanner(bytes.NewReader(obj))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Fields(sc.Text())
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
switch f[0] {
|
||||
case "v":
|
||||
if len(f) < 4 {
|
||||
return nil, errors.New("a vertex with fewer than three coordinates")
|
||||
}
|
||||
for _, s := range f[1:4] {
|
||||
v, err := strconv.ParseFloat(s, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("a vertex coordinate %q: %w", s, err)
|
||||
}
|
||||
pos = append(pos, float32(v))
|
||||
}
|
||||
if len(f) >= 7 {
|
||||
for _, s := range f[4:7] {
|
||||
v, err := strconv.ParseFloat(s, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("a vertex colour %q: %w", s, err)
|
||||
}
|
||||
col = append(col, float32(v))
|
||||
}
|
||||
} else {
|
||||
colours = false
|
||||
}
|
||||
case "f":
|
||||
n := uint32(len(pos) / 3) //nolint:gosec // bounded by the file read
|
||||
var face []uint32
|
||||
for _, s := range f[1:] {
|
||||
ref, _, _ := strings.Cut(s, "/")
|
||||
i, err := strconv.Atoi(ref)
|
||||
if err != nil || i == 0 {
|
||||
return nil, fmt.Errorf("a face index %q", s)
|
||||
}
|
||||
if i < 0 {
|
||||
i = int(n) + i + 1
|
||||
}
|
||||
if i < 1 || uint32(i) > n { //nolint:gosec // checked positive
|
||||
return nil, fmt.Errorf("a face index %d past the %d vertices", i, n)
|
||||
}
|
||||
face = append(face, uint32(i-1)) //nolint:gosec // checked positive
|
||||
}
|
||||
// A polygon as a fan of triangles.
|
||||
for k := 1; k+1 < len(face); k++ {
|
||||
idx = append(idx, face[0], face[k], face[k+1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(pos) == 0 || len(idx) == 0 {
|
||||
return nil, errors.New("the OBJ has no faces")
|
||||
}
|
||||
if !colours || len(col) != len(pos) {
|
||||
col = nil
|
||||
}
|
||||
|
||||
var bin bytes.Buffer
|
||||
view := func(data any) (offset, length int) {
|
||||
offset = bin.Len()
|
||||
_ = binary.Write(&bin, binary.LittleEndian, data)
|
||||
length = bin.Len() - offset
|
||||
for bin.Len()%4 != 0 {
|
||||
bin.WriteByte(0)
|
||||
}
|
||||
return offset, length
|
||||
}
|
||||
lo := []float32{math.MaxFloat32, math.MaxFloat32, math.MaxFloat32}
|
||||
hi := []float32{-math.MaxFloat32, -math.MaxFloat32, -math.MaxFloat32}
|
||||
for i, v := range pos {
|
||||
lo[i%3], hi[i%3] = min(lo[i%3], v), max(hi[i%3], v)
|
||||
}
|
||||
po, pl := view(pos)
|
||||
io, il := view(idx)
|
||||
views := []map[string]any{
|
||||
{"buffer": 0, "byteOffset": po, "byteLength": pl, "target": 34962},
|
||||
{"buffer": 0, "byteOffset": io, "byteLength": il, "target": 34963},
|
||||
}
|
||||
accessors := []map[string]any{
|
||||
{"bufferView": 0, "componentType": 5126, "count": len(pos) / 3, "type": "VEC3", "min": lo, "max": hi},
|
||||
{"bufferView": 1, "componentType": 5125, "count": len(idx), "type": "SCALAR"},
|
||||
}
|
||||
attrs := map[string]any{"POSITION": 0}
|
||||
if col != nil {
|
||||
co, cl := view(col)
|
||||
views = append(views, map[string]any{"buffer": 0, "byteOffset": co, "byteLength": cl, "target": 34962})
|
||||
accessors = append(accessors, map[string]any{"bufferView": 2, "componentType": 5126, "count": len(col) / 3, "type": "VEC3"})
|
||||
attrs["COLOR_0"] = 2
|
||||
}
|
||||
doc := map[string]any{
|
||||
"asset": map[string]any{"version": "2.0", "generator": "llmgw"},
|
||||
"scene": 0,
|
||||
"scenes": []any{map[string]any{"nodes": []int{0}}},
|
||||
"nodes": []any{map[string]any{"mesh": 0}},
|
||||
"meshes": []any{map[string]any{"primitives": []any{map[string]any{"attributes": attrs, "indices": 1}}}},
|
||||
"accessors": accessors,
|
||||
"bufferViews": views,
|
||||
"buffers": []any{map[string]any{"byteLength": bin.Len()}},
|
||||
}
|
||||
js, err := json.Marshal(doc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
out := make([]byte, 0, 28+len(js)+bin.Len())
|
||||
out = append(out, "glTF"...)
|
||||
out = binary.LittleEndian.AppendUint32(out, 2)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(28+len(js)+bin.Len())) //nolint:gosec // bounded by the file read
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(len(js))) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "JSON"...)
|
||||
out = append(out, js...)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(bin.Len())) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "BIN\x00"...)
|
||||
return append(out, bin.Bytes()...), nil
|
||||
}
|
||||
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
|
||||
h := rt.Handler()
|
||||
// The /admin pages are mounted only with a cluster view, which this
|
||||
// router has none of; these are the session pages it does mount.
|
||||
for _, path := range []string{"/profile", "/chat", "/org", "/org/org-x"} {
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/org", "/org/org-x"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if strings.Contains(w.Body.String(), "api key") {
|
||||
|
||||
@@ -224,6 +224,10 @@ type Outcome struct {
|
||||
Decode time.Duration
|
||||
// Usage is the engine-reported token usage, if it was present.
|
||||
Usage *Usage
|
||||
// Images is how many pictures were made, by an endpoint that makes them.
|
||||
Images int
|
||||
// Meshes is how many 3D models were made.
|
||||
Meshes int
|
||||
// Bytes is how many bytes were relayed to the client.
|
||||
Bytes int64
|
||||
// Reply is the assistant's text, when CaptureReply asked for it. Truncated
|
||||
|
||||
@@ -276,6 +276,10 @@ func splitList(v string) []string {
|
||||
// ordinary path, so the caller's limits, budget and guardrails apply to it
|
||||
// and it is counted to them.
|
||||
func (r *Router) embedFor(ctx context.Context, p auth.Principal, model, text string) ([]float32, error) {
|
||||
// A question, not a document: what the model wants before a query.
|
||||
if _, m := r.resolve(model); m.QueryPrefix != "" {
|
||||
text = m.QueryPrefix + text
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"model": model, "input": []string{text}})
|
||||
hreq, err := http.NewRequestWithContext(auth.WithPrincipal(ctx, p), http.MethodPost, "/v1/embeddings", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kb/kbsync"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -253,3 +255,25 @@ func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
|
||||
t.Errorf("a salt without passages:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuestionIsEmbeddedWithTheModelsQueryPrefix(t *testing.T) {
|
||||
// Qwen3-Embedding and its like are trained with an instruction before
|
||||
// the query, never before the documents: the documents come through
|
||||
// /v1/embeddings as they are; the question gets the prefix here.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.Capabilities.Embeddings = true
|
||||
m.QueryPrefix = "Instruct: retrieve passages\nQuery: "
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
// The test engine answers with the input's length in bytes and words.
|
||||
v, err := rt.embedFor(t.Context(), auth.Principal{}, "emb", "kaç gün izin var")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "Instruct: retrieve passages\nQuery: kaç gün izin var"
|
||||
if int(v[0]) != len(want) || int(v[1]) != len(strings.Fields(want)) {
|
||||
t.Errorf("embedded %v, not %q", v, want)
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -53,7 +54,9 @@ type Router struct {
|
||||
reg *endpoint.Registry
|
||||
poller *scrape.Poller
|
||||
px *proxy.Proxy
|
||||
m *metrics
|
||||
// comfy asks ComfyUI replicas for pictures.
|
||||
comfy *comfy.Client
|
||||
m *metrics
|
||||
// adm is nil when admission control is disabled.
|
||||
adm *admission.Controller
|
||||
|
||||
@@ -74,8 +77,10 @@ type Router struct {
|
||||
canaries canaryStats
|
||||
// slo keeps each model's recent token timings, for admission.
|
||||
slo sloTracker
|
||||
// batchWake has the batch worker look for work at once.
|
||||
// batchWake has the batch worker look for work at once; jobWake the
|
||||
// job worker.
|
||||
batchWake chan struct{}
|
||||
jobWake chan struct{}
|
||||
// reqLogs is each project's keeping of requests, as last read.
|
||||
reqLogs requestLogs
|
||||
// answers are the answers projects reuse; nil reuses none.
|
||||
@@ -301,6 +306,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
|
||||
r := &Router{
|
||||
batchWake: make(chan struct{}, 1),
|
||||
jobWake: make(chan struct{}, 1),
|
||||
cfg: cfg.Router,
|
||||
landing: cfg.Landing,
|
||||
log: log,
|
||||
@@ -311,6 +317,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
kv: kvindex.New(),
|
||||
kvClient: newKVClient(),
|
||||
m: metrics,
|
||||
comfy: &comfy.Client{HTTP: &http.Client{Transport: &http.Transport{DialContext: (&net.Dialer{Timeout: cfg.Router.Timeouts.Dial}).DialContext}}},
|
||||
px: proxy.New(proxy.Config{
|
||||
DialTimeout: cfg.Router.Timeouts.Dial,
|
||||
ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader,
|
||||
@@ -370,7 +377,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
prices := make(map[string]accounting.Price, len(cfg.Auth.Pricing))
|
||||
for name, p := range cfg.Auth.Pricing {
|
||||
prices[name] = accounting.Price{
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k,
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, PerImage: p.PerImage, PerMesh: p.PerMesh,
|
||||
}
|
||||
}
|
||||
r.recorder = accounting.NewRecorder(st, r.auth, accounting.RecorderConfig{
|
||||
@@ -386,6 +393,8 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
accounting.NewPricing(prices, accounting.Price{
|
||||
PromptPer1k: cfg.Auth.DefaultPrice.PromptPer1k,
|
||||
CompletionPer1k: cfg.Auth.DefaultPrice.CompletionPer1k,
|
||||
PerImage: cfg.Auth.DefaultPrice.PerImage,
|
||||
PerMesh: cfg.Auth.DefaultPrice.PerMesh,
|
||||
}),
|
||||
r.recorder, log)
|
||||
r.threads = r.store
|
||||
@@ -427,6 +436,7 @@ func admissionConfig(rc config.Router) admission.Config {
|
||||
Name: b.Name,
|
||||
Priority: b.Priority,
|
||||
SaturationCeiling: b.SaturationCeiling,
|
||||
QueueTTL: b.QueueTTL,
|
||||
})
|
||||
}
|
||||
return admission.Config{
|
||||
@@ -487,6 +497,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
|
||||
Pool: e.Pool,
|
||||
Weight: e.Weight,
|
||||
Priority: e.Priority,
|
||||
Workflow: e.Workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -522,6 +533,7 @@ func (r *Router) Start(ctx context.Context) {
|
||||
}
|
||||
if r.store != nil {
|
||||
go r.runBatches(ctx)
|
||||
go r.runJobs(ctx)
|
||||
go r.pruneRequestBodies(ctx)
|
||||
go r.refreshTenants(ctx)
|
||||
go r.purgeSessions(ctx)
|
||||
@@ -1162,7 +1174,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
}
|
||||
// A filter per attempt, for the model this attempt is served by.
|
||||
filter := r.outputFilter(uctx, req, model, &preq)
|
||||
out, perr := r.px.Do(uctx, w, preq)
|
||||
var out proxy.Outcome
|
||||
var perr error
|
||||
if dec.Endpoint.Engine == endpoint.EngineComfyUI {
|
||||
out, perr = r.generateImages(uctx, w, req, dec.Endpoint.Info, body)
|
||||
} else {
|
||||
out, perr = r.px.Do(uctx, w, preq)
|
||||
}
|
||||
releaseAdapter()
|
||||
uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
|
||||
endSpan(uspan, perr)
|
||||
|
||||
@@ -783,3 +783,28 @@ func TestAStoppedServiceAnswers503AndStillListsModels(t *testing.T) {
|
||||
t.Errorf("started again, still refused: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestABatchOfEmbeddingsIsCheckedTextByText(t *testing.T) {
|
||||
// llmgw-kb sends many passages in one request; each is embedded on its
|
||||
// own, so their sum over the context is no reason to refuse them. One
|
||||
// passage over the context still is.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.ContextLength = 2048
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
send := func(input []string) *httptest.ResponseRecorder {
|
||||
b, _ := json.Marshal(map[string]any{"model": "emb", "input": input})
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/embeddings", bytes.NewReader(b)))
|
||||
return rec
|
||||
}
|
||||
passage := strings.Repeat("x", 6000) // about 1500 tokens
|
||||
if rec := send([]string{passage, passage, passage, passage}); rec.Code == http.StatusBadRequest {
|
||||
t.Errorf("four passages that each fit were refused: %s", rec.Body.String())
|
||||
}
|
||||
if rec := send([]string{"short", strings.Repeat("x", 40_000)}); rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("a passage over the context was let through: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -96,8 +96,8 @@ func (p *Poller) Sync(ctx context.Context) {
|
||||
|
||||
want := make(map[string]endpoint.Info, len(infos))
|
||||
for _, in := range infos {
|
||||
if in.Engine == endpoint.EngineExternal {
|
||||
continue // a provider elsewhere publishes no /metrics
|
||||
if !endpoint.Scraped(in.Engine) {
|
||||
continue // a provider elsewhere, or ComfyUI, publishes no /metrics
|
||||
}
|
||||
want[in.ID] = in
|
||||
}
|
||||
|
||||
@@ -68,6 +68,8 @@ var copyOrder = []string{
|
||||
"files",
|
||||
"batches",
|
||||
"batch_results",
|
||||
// After files, which a finished job's result is one of.
|
||||
"jobs",
|
||||
"request_bodies",
|
||||
// Runs before their cases, which point at them.
|
||||
"eval_runs",
|
||||
@@ -88,6 +90,7 @@ var copyOrder = []string{
|
||||
var notCopied = map[string]string{
|
||||
"leadership": "a lease on a running process, meaningless in a new store",
|
||||
"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
|
||||
"notify_sent": "when notifications were last sent; at worst one is sent again an hour early",
|
||||
}
|
||||
|
||||
// CopyResult is how many rows each table received.
|
||||
|
||||
@@ -25,6 +25,8 @@ type ExportRow struct {
|
||||
Prompt int64
|
||||
Completion int64
|
||||
CostMicros int64
|
||||
Images int64
|
||||
Meshes int64
|
||||
}
|
||||
|
||||
// ExportDims are what an export can be broken down by.
|
||||
@@ -51,16 +53,18 @@ func (s *DB) UsageExport(ctx context.Context, from, to time.Time, period string,
|
||||
bucket int64
|
||||
key, user, project, model string
|
||||
requests, errors, limited, prompt, completion, cost int64
|
||||
images, meshes int64
|
||||
}
|
||||
//nolint:gosec // table is one of two literals above
|
||||
rows, err := queryAll(ctx, s.db, `SELECT bucket, key_id, user_id, project_id, served_model,
|
||||
SUM(requests), SUM(errors), SUM(limited), SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
|
||||
SUM(requests), SUM(errors), SUM(limited), SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
|
||||
SUM(images), SUM(meshes)
|
||||
FROM `+table+` WHERE bucket >= ? AND bucket < ?
|
||||
GROUP BY bucket, key_id, user_id, project_id, served_model`,
|
||||
[]any{millis(from), millis(to)}, func(r *sql.Rows) (raw, error) {
|
||||
var x raw
|
||||
err := r.Scan(&x.bucket, &x.key, &x.user, &x.project, &x.model,
|
||||
&x.requests, &x.errors, &x.limited, &x.prompt, &x.completion, &x.cost)
|
||||
&x.requests, &x.errors, &x.limited, &x.prompt, &x.completion, &x.cost, &x.images, &x.meshes)
|
||||
return x, err
|
||||
})
|
||||
if err != nil {
|
||||
@@ -137,6 +141,8 @@ func (s *DB) UsageExport(ctx context.Context, from, to time.Time, period string,
|
||||
t.Prompt += x.prompt
|
||||
t.Completion += x.completion
|
||||
t.CostMicros += x.cost
|
||||
t.Images += x.images
|
||||
t.Meshes += x.meshes
|
||||
}
|
||||
out := make([]ExportRow, 0, len(order))
|
||||
for _, k := range order {
|
||||
|
||||
@@ -46,3 +46,36 @@ func TestUsageIsExportedByPeriodAndWhatWasAskedFor(t *testing.T) {
|
||||
t.Error("an unknown dimension was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAreCountedInTheSumsAndTheExport(t *testing.T) {
|
||||
// The request rows said how many pictures each made, and the sums did
|
||||
// not: once the rows were past their retention, an export said what the
|
||||
// pictures cost and not how many there were.
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
at := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
|
||||
if err := db.WriteUsage(ctx, []Usage{
|
||||
{At: at, KeyID: "k", Model: "flux", ServedModel: "flux", PromptTokens: 9, Images: 2, CostMicros: 40000, Status: 200},
|
||||
{At: at.Add(time.Minute), KeyID: "k", Model: "flux", ServedModel: "flux", PromptTokens: 9, Images: 3, CostMicros: 60000, Status: 200},
|
||||
{At: at.Add(2 * time.Minute), KeyID: "k", Model: "chat", ServedModel: "chat", PromptTokens: 9, Status: 200},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Past their retention: only the sums are left.
|
||||
if _, err := db.PruneUsage(ctx, at.Add(time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, period := range []string{"hour", "day"} {
|
||||
rows, err := db.UsageExport(ctx, at.Add(-24*time.Hour), at.Add(48*time.Hour), period, []string{"model"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := map[string]int64{}
|
||||
for _, r := range rows {
|
||||
got[r.Model] += r.Images
|
||||
}
|
||||
if got["flux"] != 5 || got["chat"] != 0 {
|
||||
t.Errorf("%s: pictures by model %v, want flux 5", period, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Job is one request that is answered with an id and asked after: work that
|
||||
// takes minutes, which no client should hold a connection for. The request
|
||||
// is kept as it was sent and replayed by whichever gateway holds the lease.
|
||||
type Job struct {
|
||||
ID string
|
||||
Owner string
|
||||
KeyID string
|
||||
Endpoint string
|
||||
Model string
|
||||
// Input is the request's body, as the client sent it.
|
||||
Input []byte
|
||||
Status string
|
||||
// Progress is what the engine said of the work so far, as JSON; empty
|
||||
// until it said anything.
|
||||
Progress string
|
||||
// PromptID is the engine's own id for the work, so a gateway that takes
|
||||
// over a running job asks after the same work rather than queueing it
|
||||
// again.
|
||||
PromptID string
|
||||
OutputFileID string
|
||||
ContentType string
|
||||
Error string
|
||||
CreatedAt time.Time
|
||||
StartedAt time.Time
|
||||
FinishedAt time.Time
|
||||
// ExpiresAt is when a job not yet finished is given up on.
|
||||
ExpiresAt time.Time
|
||||
}
|
||||
|
||||
// Job states.
|
||||
const (
|
||||
JobQueued = "queued"
|
||||
JobRunning = "running"
|
||||
JobCompleted = "completed"
|
||||
JobFailed = "failed"
|
||||
JobCancelling = "cancelling"
|
||||
JobCancelled = "cancelled"
|
||||
JobExpired = "expired"
|
||||
)
|
||||
|
||||
// JobDone is whether a state is an end.
|
||||
func JobDone(status string) bool {
|
||||
switch status {
|
||||
case JobCompleted, JobFailed, JobCancelled, JobExpired:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
const jobCols = `id, owner, key_id, endpoint, model, input, status, progress, prompt_id, output_file_id,
|
||||
content_type, error, created_at, started_at, finished_at, expires_at`
|
||||
|
||||
func scanJob(sc interface{ Scan(...any) error }) (Job, error) {
|
||||
var j Job
|
||||
var created, started, finished, expires int64
|
||||
err := sc.Scan(&j.ID, &j.Owner, &j.KeyID, &j.Endpoint, &j.Model, &j.Input, &j.Status, &j.Progress, &j.PromptID,
|
||||
&j.OutputFileID, &j.ContentType, &j.Error, &created, &started, &finished, &expires)
|
||||
if err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
at := func(v int64) time.Time {
|
||||
if v == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.UnixMilli(v)
|
||||
}
|
||||
j.CreatedAt, j.StartedAt, j.FinishedAt, j.ExpiresAt = at(created), at(started), at(finished), at(expires)
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// PutJob records a new job.
|
||||
func (s *DB) PutJob(ctx context.Context, j Job) error {
|
||||
if j.Input == nil {
|
||||
j.Input = []byte{}
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx, `INSERT INTO jobs (id, owner, key_id, endpoint, model, input, status, created_at, expires_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
j.ID, j.Owner, j.KeyID, j.Endpoint, j.Model, j.Input, j.Status, millis(j.CreatedAt), millis(j.ExpiresAt))
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateJob writes how a job stands: its state, result, error and times.
|
||||
// Its progress and prompt id are written by SetJobProgress, which the work
|
||||
// calls often and which must not undo an end written here.
|
||||
func (s *DB) UpdateJob(ctx context.Context, j Job) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, output_file_id = ?, content_type = ?, error = ?,
|
||||
started_at = ?, finished_at = ? WHERE id = ?`,
|
||||
j.Status, j.OutputFileID, j.ContentType, j.Error, optMillis(j.StartedAt), optMillis(j.FinishedAt), j.ID)
|
||||
return err
|
||||
}
|
||||
|
||||
// SetJobProgress writes what the engine said of a running job, and its id
|
||||
// for the work when it is known. A job that has ended keeps its end.
|
||||
func (s *DB) SetJobProgress(ctx context.Context, id, progress, promptID string) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET progress = ?,
|
||||
prompt_id = CASE WHEN ? = '' THEN prompt_id ELSE ? END
|
||||
WHERE id = ? AND status IN (?, ?)`, progress, promptID, promptID, id, JobRunning, JobCancelling)
|
||||
return err
|
||||
}
|
||||
|
||||
// CancelJob asks for a job to stop: cancelling, unless it has ended already.
|
||||
// It says the job as it stands.
|
||||
func (s *DB) CancelJob(ctx context.Context, id string, now time.Time) (Job, error) {
|
||||
// One not yet taken ends at once: there is nothing to stop.
|
||||
if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, finished_at = ?
|
||||
WHERE id = ? AND status = ? AND lease_until < ?`, JobCancelled, millis(now), id, JobQueued, millis(now)); err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ? WHERE id = ? AND status IN (?, ?)`,
|
||||
JobCancelling, id, JobQueued, JobRunning); err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
return s.JobByID(ctx, id)
|
||||
}
|
||||
|
||||
// JobByID finds a job, ErrNotFound when there is none.
|
||||
func (s *DB) JobByID(ctx context.Context, id string) (Job, error) {
|
||||
j, err := scanJob(s.db.QueryRowContext(ctx, `SELECT `+jobCols+` FROM jobs WHERE id = ?`, id))
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Job{}, ErrNotFound
|
||||
}
|
||||
return j, err
|
||||
}
|
||||
|
||||
// Jobs lists an owner's jobs, the newest first, those made before the job
|
||||
// after when it is given.
|
||||
func (s *DB) Jobs(ctx context.Context, owner, after string, limit int) ([]Job, error) {
|
||||
q := `SELECT ` + jobCols + ` FROM jobs WHERE owner = ?`
|
||||
args := []any{owner}
|
||||
if after != "" {
|
||||
q += ` AND created_at < (SELECT created_at FROM jobs WHERE id = ?)`
|
||||
args = append(args, after)
|
||||
}
|
||||
if limit <= 0 || limit > 100 {
|
||||
limit = 20
|
||||
}
|
||||
q += ` ORDER BY created_at DESC LIMIT ?`
|
||||
args = append(args, limit)
|
||||
rows, err := s.db.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Job
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, j)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// AllJobs is every owner's jobs, newest first, those in one status when it
|
||||
// is named: what an operator looks over.
|
||||
func (s *DB) AllJobs(ctx context.Context, status string, limit int) ([]Job, error) {
|
||||
q := `SELECT ` + jobCols + ` FROM jobs`
|
||||
var args []any
|
||||
if status != "" {
|
||||
q += ` WHERE status = ?`
|
||||
args = append(args, status)
|
||||
}
|
||||
if limit <= 0 || limit > 500 {
|
||||
limit = 50
|
||||
}
|
||||
q += ` ORDER BY created_at DESC LIMIT ?`
|
||||
args = append(args, limit)
|
||||
rows, err := s.db.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Job
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, j)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// ClaimJob takes the lease on the oldest job with work left that no gateway
|
||||
// holds, for owner until the time given. ok is false when there is none.
|
||||
func (s *DB) ClaimJob(ctx context.Context, owner string, now, until time.Time) (Job, bool, error) {
|
||||
var id string
|
||||
err := s.db.QueryRowContext(ctx, `UPDATE jobs SET lease_owner = ?, lease_until = ?
|
||||
WHERE id = (SELECT id FROM jobs WHERE status IN (?, ?, ?) AND lease_until < ?
|
||||
ORDER BY created_at LIMIT 1)
|
||||
AND lease_until < ?
|
||||
RETURNING id`, owner, millis(until), JobQueued, JobRunning, JobCancelling, millis(now), millis(now)).Scan(&id)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Job{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return Job{}, false, err
|
||||
}
|
||||
j, err := s.JobByID(ctx, id)
|
||||
return j, err == nil, err
|
||||
}
|
||||
|
||||
// RenewJobLease keeps a lease held; false when it is held no longer.
|
||||
func (s *DB) RenewJobLease(ctx context.Context, id, owner string, until time.Time) (bool, error) {
|
||||
res, err := s.db.ExecContext(ctx, `UPDATE jobs SET lease_until = ? WHERE id = ? AND lease_owner = ?`,
|
||||
millis(until), id, owner)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// ReleaseJob gives a lease up.
|
||||
func (s *DB) ReleaseJob(ctx context.Context, id, owner string) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET lease_until = 0, lease_owner = '' WHERE id = ? AND lease_owner = ?`, id, owner)
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteJobsBefore forgets the jobs that ended before the time given, whose
|
||||
// result files are gone by then too.
|
||||
func (s *DB) DeleteJobsBefore(ctx context.Context, before time.Time) (int64, error) {
|
||||
res, err := s.db.ExecContext(ctx, `DELETE FROM jobs WHERE finished_at > 0 AND finished_at < ?`, millis(before))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return res.RowsAffected()
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJobs(t *testing.T) {
|
||||
checkJobs(t, capacityStore(t))
|
||||
}
|
||||
|
||||
func TestPostgresJobs(t *testing.T) {
|
||||
checkJobs(t, openPG(t))
|
||||
}
|
||||
|
||||
func checkJobs(t *testing.T, db *DB) {
|
||||
ctx := t.Context()
|
||||
now := time.Now()
|
||||
j := Job{ID: "job-1", Owner: "p1", KeyID: "k", Endpoint: "/v1/3d/generations", Model: "trellis2",
|
||||
Input: []byte(`{"model":"trellis2"}`), Status: JobQueued, CreatedAt: now, ExpiresAt: now.Add(24 * time.Hour)}
|
||||
if err := db.PutJob(ctx, j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, ok, err := db.ClaimJob(ctx, "gw-a", now, now.Add(time.Minute))
|
||||
if err != nil || !ok || got.ID != "job-1" || string(got.Input) != `{"model":"trellis2"}` {
|
||||
t.Fatalf("claim: %+v %v %v", got, ok, err)
|
||||
}
|
||||
if _, ok, _ := db.ClaimJob(ctx, "gw-b", now, now.Add(time.Minute)); ok {
|
||||
t.Error("a leased job was claimed twice")
|
||||
}
|
||||
if held, _ := db.RenewJobLease(ctx, "job-1", "gw-b", now.Add(time.Hour)); held {
|
||||
t.Error("another gateway renewed the lease")
|
||||
}
|
||||
if held, _ := db.RenewJobLease(ctx, "job-1", "gw-a", now.Add(time.Hour)); !held {
|
||||
t.Error("the holder could not renew")
|
||||
}
|
||||
|
||||
// Progress is written while it runs, and never over an end.
|
||||
got.Status, got.StartedAt = JobRunning, now
|
||||
if err := db.UpdateJob(ctx, got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":40}`, "p-7"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":60}`, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cur, _ := db.JobByID(ctx, "job-1")
|
||||
if cur.Progress != `{"percent":60}` || cur.PromptID != "p-7" || cur.Status != JobRunning {
|
||||
t.Errorf("%+v", cur)
|
||||
}
|
||||
|
||||
// A running job asked to stop is cancelling, for the worker to stop it.
|
||||
cur, err = db.CancelJob(ctx, "job-1", now)
|
||||
if err != nil || cur.Status != JobCancelling {
|
||||
t.Fatalf("%+v %v", cur, err)
|
||||
}
|
||||
cur.Status, cur.FinishedAt = JobCancelled, now
|
||||
_ = db.UpdateJob(ctx, cur)
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":99}`, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cur, _ = db.JobByID(ctx, "job-1"); cur.Progress != `{"percent":60}` || cur.Status != JobCancelled {
|
||||
t.Errorf("progress written over an end: %+v", cur)
|
||||
}
|
||||
_ = db.ReleaseJob(ctx, "job-1", "gw-a")
|
||||
if _, ok, _ := db.ClaimJob(ctx, "gw-b", now.Add(2*time.Hour), now.Add(3*time.Hour)); ok {
|
||||
t.Error("an ended job was claimed")
|
||||
}
|
||||
|
||||
// One not yet taken ends at once when cancelled.
|
||||
j2 := j
|
||||
j2.ID, j2.CreatedAt = "job-2", now.Add(time.Second)
|
||||
_ = db.PutJob(ctx, j2)
|
||||
if cur, _ = db.CancelJob(ctx, "job-2", now.Add(2*time.Second)); cur.Status != JobCancelled || cur.FinishedAt.IsZero() {
|
||||
t.Errorf("%+v", cur)
|
||||
}
|
||||
|
||||
// Listed newest first, another owner's not at all.
|
||||
j3 := j
|
||||
j3.ID, j3.Owner, j3.CreatedAt = "job-3", "p2", now.Add(3*time.Second)
|
||||
_ = db.PutJob(ctx, j3)
|
||||
js, _ := db.Jobs(ctx, "p1", "", 10)
|
||||
if len(js) != 2 || js[0].ID != "job-2" || js[1].ID != "job-1" {
|
||||
t.Errorf("listed %+v", js)
|
||||
}
|
||||
if js, _ = db.Jobs(ctx, "p1", "job-2", 10); len(js) != 1 || js[0].ID != "job-1" {
|
||||
t.Errorf("after: %+v", js)
|
||||
}
|
||||
if _, err := db.JobByID(ctx, "nope"); err != ErrNotFound {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
if n, _ := db.DeleteJobsBefore(ctx, now.Add(time.Hour)); n != 2 {
|
||||
t.Errorf("deleted %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOperatorSeesEveryOwnersJobs(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
now := time.Now()
|
||||
for i, owner := range []string{"project:a", "project:b", "user:c"} {
|
||||
status := JobQueued
|
||||
if i == 2 {
|
||||
status = JobCompleted
|
||||
}
|
||||
if err := db.PutJob(t.Context(), Job{ID: fmt.Sprint("job_", i), Owner: owner, KeyID: "k", Endpoint: "/v1/3d/generations",
|
||||
Model: "m", Input: []byte("{}"), Status: status, CreatedAt: now.Add(time.Duration(i) * time.Second),
|
||||
ExpiresAt: now.Add(time.Hour)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
all, err := db.AllJobs(t.Context(), "", 0)
|
||||
if err != nil || len(all) != 3 || all[0].ID != "job_2" {
|
||||
t.Fatalf("%v %v: every owner's, newest first", all, err)
|
||||
}
|
||||
queued, _ := db.AllJobs(t.Context(), JobQueued, 0)
|
||||
if len(queued) != 2 {
|
||||
t.Errorf("%d queued", len(queued))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many pictures a request made, for a model that makes them: what a
|
||||
-- per-picture price is charged by, and what its tokens do not say.
|
||||
ALTER TABLE usage ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,23 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many pictures each slice's requests made, beside their tokens and
|
||||
-- cost, so the sums and exports say it after the request rows are gone.
|
||||
-- Pictures made before this were counted only in those rows: added from
|
||||
-- them, for the slices since the first.
|
||||
ALTER TABLE usage_5m ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
UPDATE usage_5m SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_5m.bucket AND u.ts < usage_5m.bucket + 300000
|
||||
AND u.key_id = usage_5m.key_id AND u.user_id = usage_5m.user_id AND u.project_id = usage_5m.project_id
|
||||
AND u.model = usage_5m.model AND u.served_model = usage_5m.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 300000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1h SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1h.bucket AND u.ts < usage_1h.bucket + 3600000
|
||||
AND u.key_id = usage_1h.key_id AND u.user_id = usage_1h.user_id AND u.project_id = usage_1h.project_id
|
||||
AND u.model = usage_1h.model AND u.served_model = usage_1h.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 3600000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1d SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1d.bucket AND u.ts < usage_1d.bucket + 86400000
|
||||
AND u.key_id = usage_1d.key_id AND u.user_id = usage_1d.user_id AND u.project_id = usage_1d.project_id
|
||||
AND u.model = usage_1d.model AND u.served_model = usage_1d.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 86400000 FROM usage WHERE images > 0);
|
||||
@@ -0,0 +1,8 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- When each notification was last sent, by its kind and subject: every
|
||||
-- control plane raises the same events — a budget spent is seen by each one
|
||||
-- serving its requests — and the one that claims an event here sends it.
|
||||
CREATE TABLE notify_sent (
|
||||
key TEXT PRIMARY KEY,
|
||||
sent_at BIGINT NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- Jobs (BACKLOG G1): a request that takes minutes — a 3D model from a
|
||||
-- picture — answered with an id and asked after, rather than held open.
|
||||
-- The request is kept as it was sent, so whichever gateway holds the lease
|
||||
-- replays it through the same path a client's request takes; the result is
|
||||
-- a file in the files table, kept for the batch files' retention.
|
||||
CREATE TABLE jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
key_id TEXT NOT NULL,
|
||||
endpoint TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
input BYTEA NOT NULL,
|
||||
-- status is queued, running, completed, failed, cancelling, cancelled or
|
||||
-- expired.
|
||||
status TEXT NOT NULL,
|
||||
-- progress is what the engine said of the work so far, as JSON: queue
|
||||
-- position, percent, step. Empty until it says anything.
|
||||
progress TEXT NOT NULL DEFAULT '',
|
||||
-- prompt_id is the engine's own id for the work (ComfyUI's prompt_id),
|
||||
-- so a gateway that takes over a running job asks after the same work.
|
||||
prompt_id TEXT NOT NULL DEFAULT '',
|
||||
output_file_id TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL DEFAULT '',
|
||||
error TEXT NOT NULL DEFAULT '',
|
||||
created_at BIGINT NOT NULL,
|
||||
started_at BIGINT NOT NULL DEFAULT 0,
|
||||
finished_at BIGINT NOT NULL DEFAULT 0,
|
||||
expires_at BIGINT NOT NULL,
|
||||
lease_owner TEXT NOT NULL DEFAULT '',
|
||||
lease_until BIGINT NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX jobs_owner ON jobs (owner, created_at);
|
||||
CREATE INDEX jobs_status ON jobs (status, lease_until);
|
||||
@@ -0,0 +1,7 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many 3D models a request made, as images says how many pictures: what
|
||||
-- per_mesh is charged by, in the request rows and in the sums.
|
||||
ALTER TABLE usage ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_5m ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,3 @@
|
||||
-- How many pictures a request made, for a model that makes them: what a
|
||||
-- per-picture price is charged by, and what its tokens do not say.
|
||||
ALTER TABLE usage ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,22 @@
|
||||
-- How many pictures each slice's requests made, beside their tokens and
|
||||
-- cost, so the sums and exports say it after the request rows are gone.
|
||||
-- Pictures made before this were counted only in those rows: added from
|
||||
-- them, for the slices since the first.
|
||||
ALTER TABLE usage_5m ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
UPDATE usage_5m SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_5m.bucket AND u.ts < usage_5m.bucket + 300000
|
||||
AND u.key_id = usage_5m.key_id AND u.user_id = usage_5m.user_id AND u.project_id = usage_5m.project_id
|
||||
AND u.model = usage_5m.model AND u.served_model = usage_5m.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 300000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1h SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1h.bucket AND u.ts < usage_1h.bucket + 3600000
|
||||
AND u.key_id = usage_1h.key_id AND u.user_id = usage_1h.user_id AND u.project_id = usage_1h.project_id
|
||||
AND u.model = usage_1h.model AND u.served_model = usage_1h.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 3600000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1d SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1d.bucket AND u.ts < usage_1d.bucket + 86400000
|
||||
AND u.key_id = usage_1d.key_id AND u.user_id = usage_1d.user_id AND u.project_id = usage_1d.project_id
|
||||
AND u.model = usage_1d.model AND u.served_model = usage_1d.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 86400000 FROM usage WHERE images > 0);
|
||||
@@ -0,0 +1,7 @@
|
||||
-- When each notification was last sent, by its kind and subject: every
|
||||
-- control plane raises the same events — a budget spent is seen by each one
|
||||
-- serving its requests — and the one that claims an event here sends it.
|
||||
CREATE TABLE notify_sent (
|
||||
key TEXT PRIMARY KEY,
|
||||
sent_at BIGINT NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,33 @@
|
||||
-- Jobs (BACKLOG G1): a request that takes minutes — a 3D model from a
|
||||
-- picture — answered with an id and asked after, rather than held open.
|
||||
-- The request is kept as it was sent, so whichever gateway holds the lease
|
||||
-- replays it through the same path a client's request takes; the result is
|
||||
-- a file in the files table, kept for the batch files' retention.
|
||||
CREATE TABLE jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
key_id TEXT NOT NULL,
|
||||
endpoint TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
input BLOB NOT NULL,
|
||||
-- status is queued, running, completed, failed, cancelling, cancelled or
|
||||
-- expired.
|
||||
status TEXT NOT NULL,
|
||||
-- progress is what the engine said of the work so far, as JSON: queue
|
||||
-- position, percent, step. Empty until it says anything.
|
||||
progress TEXT NOT NULL DEFAULT '',
|
||||
-- prompt_id is the engine's own id for the work (ComfyUI's prompt_id),
|
||||
-- so a gateway that takes over a running job asks after the same work.
|
||||
prompt_id TEXT NOT NULL DEFAULT '',
|
||||
output_file_id TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL DEFAULT '',
|
||||
error TEXT NOT NULL DEFAULT '',
|
||||
created_at INTEGER NOT NULL,
|
||||
started_at INTEGER NOT NULL DEFAULT 0,
|
||||
finished_at INTEGER NOT NULL DEFAULT 0,
|
||||
expires_at INTEGER NOT NULL,
|
||||
lease_owner TEXT NOT NULL DEFAULT '',
|
||||
lease_until INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX jobs_owner ON jobs (owner, created_at);
|
||||
CREATE INDEX jobs_status ON jobs (status, lease_until);
|
||||
@@ -0,0 +1,6 @@
|
||||
-- How many 3D models a request made, as images says how many pictures: what
|
||||
-- per_mesh is charged by, in the request rows and in the sums.
|
||||
ALTER TABLE usage ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_5m ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,23 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ClaimNotice claims the sending of a notification: true when nobody has
|
||||
// sent the same one within the window, and from now on, nobody else may
|
||||
// until it has passed. One statement, so two control planes asking at once
|
||||
// cannot both be told yes.
|
||||
func (s *DB) ClaimNotice(ctx context.Context, key string, at time.Time, window time.Duration) (bool, error) {
|
||||
now := at.UnixMilli()
|
||||
res, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO notify_sent (key, sent_at) VALUES (?, ?)
|
||||
ON CONFLICT (key) DO UPDATE SET sent_at = excluded.sent_at WHERE notify_sent.sent_at < ?`,
|
||||
key, now, now-window.Milliseconds())
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
return n == 1, err
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestANoticeIsClaimedByOneOfThoseAskingAtOnce(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
at := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
|
||||
var yes atomic.Int64
|
||||
var wg sync.WaitGroup
|
||||
for range 8 {
|
||||
wg.Go(func() {
|
||||
ok, err := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at, time.Hour)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if ok {
|
||||
yes.Add(1)
|
||||
}
|
||||
})
|
||||
}
|
||||
wg.Wait()
|
||||
if yes.Load() != 1 {
|
||||
t.Fatalf("%d of eight claimed the same notice", yes.Load())
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at.Add(59*time.Minute), time.Hour); ok {
|
||||
t.Error("claimed again within the hour")
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t2", at, time.Hour); !ok {
|
||||
t.Error("another subject could not be claimed")
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at.Add(61*time.Minute), time.Hour); !ok {
|
||||
t.Error("not claimable once the hour had passed")
|
||||
}
|
||||
}
|
||||
@@ -69,7 +69,7 @@ type rollupKey struct {
|
||||
// rollupRow is what a batch adds to one row.
|
||||
type rollupRow struct {
|
||||
requests, errors, limited, prompt int64
|
||||
completion, cost int64
|
||||
completion, cost, images, meshes int64
|
||||
latency, queue, ttft, ttftN, last int64
|
||||
lat, ttftH []int64
|
||||
}
|
||||
@@ -98,6 +98,8 @@ func rollupBatch(t tier, records []Usage) (map[rollupKey]*rollupRow, []rollupKey
|
||||
row.prompt += r.PromptTokens
|
||||
row.completion += r.CompletionTokens
|
||||
row.cost += r.CostMicros
|
||||
row.images += r.Images
|
||||
row.meshes += r.Meshes
|
||||
row.latency += r.LatencyMs
|
||||
row.queue += r.QueueMs
|
||||
row.lat[latencyBucket(r.LatencyMs)]++
|
||||
@@ -116,7 +118,7 @@ var rollupColumns = func() []string {
|
||||
n := len(LatencyBounds) + 1
|
||||
cols := []string{"bucket", "key_id", "user_id", "project_id", "model", "served_model",
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros",
|
||||
"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts"}
|
||||
"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts", "images", "meshes"}
|
||||
for i := range n {
|
||||
cols = append(cols, fmt.Sprintf("lat_%d", i))
|
||||
}
|
||||
@@ -156,7 +158,7 @@ func writeRollup(ctx context.Context, tx conn, records []Usage) error {
|
||||
r := rows[k]
|
||||
args := []any{k.bucket, k.keyID, k.userID, k.projectID, k.model, k.servedModel,
|
||||
r.requests, r.errors, r.limited, r.prompt, r.completion, r.cost,
|
||||
r.latency, r.queue, r.ttft, r.ttftN, r.last}
|
||||
r.latency, r.queue, r.ttft, r.ttftN, r.last, r.images, r.meshes}
|
||||
for _, v := range r.lat {
|
||||
args = append(args, v)
|
||||
}
|
||||
@@ -187,11 +189,11 @@ func usageSince(since time.Time) (string, []any) {
|
||||
from := millis(since)
|
||||
to5m, to1h, to1d := tier5m.ceil(since), tier1h.ceil(since), tier1d.ceil(since)
|
||||
const cols = `key_id, user_id, project_id, model, served_model, requests, errors, limited,
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts`
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts, images, meshes`
|
||||
return `(SELECT key_id, user_id, project_id, model, served_model, 1 AS requests,
|
||||
CASE WHEN status >= 400 THEN 1 ELSE 0 END AS errors,
|
||||
CASE WHEN status = 429 THEN 1 ELSE 0 END AS limited,
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts, images, meshes
|
||||
FROM usage WHERE ts >= ? AND ts < ?
|
||||
UNION ALL SELECT ` + cols + ` FROM usage_5m WHERE bucket >= ? AND bucket < ?
|
||||
UNION ALL SELECT ` + cols + ` FROM usage_1h WHERE bucket >= ? AND bucket < ?
|
||||
|
||||
Loaded 100 of 123 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user