Compare commits
20
Commits
adapters-f4
...
v0.14.2
No files matched your search
+36
@@ -1065,6 +1065,42 @@ training yet; quality gating of a published adapter is by hand (ctl eval).
|
||||
|
||||
---
|
||||
|
||||
### F5. Large imports (M) — measured 2026-10-09, open
|
||||
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
|
||||
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
|
||||
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
|
||||
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
- Done: a batch of embeddings checked text by text against the context
|
||||
(branch `embeddings-batch`); it was refused as one over-long prompt.
|
||||
- Done: `query_prefix` on an embedding model, before questions only.
|
||||
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
|
||||
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU
|
||||
profile on llama.cpp, which needs a source per profile (the catalog has
|
||||
one per model; the import swapped the entry by hand), and an admission
|
||||
queue limit that suits bulk imports (30 s turned some away).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||||
chat model, and a reranker.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+70
-1
@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
@@ -26,6 +26,75 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## v0.14.2 — 2026-10-10
|
||||
|
||||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||||
cluster's own cards. The agents install ComfyUI themselves, the gateway
|
||||
prices pictures by the picture, and the portal has a page to make them.
|
||||
Also: on-demand models that survive a failover, and knowledge-base
|
||||
embeddings fixes.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains a column (`usage.images`); nothing to do.
|
||||
- **ComfyUI is installed only where asked**: `llmgw ctl config set
|
||||
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
|
||||
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
|
||||
or run engine `comfyui`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
|
||||
key, allowlist, limit, admission and guardrail checks as any request. A
|
||||
model with profiles on engine `comfyui` is placed on a GPU node like any
|
||||
other. A ComfyUI server outside the cluster can be an external model with
|
||||
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
|
||||
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
|
||||
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
|
||||
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
|
||||
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
|
||||
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
|
||||
that leaves takes its workflow off the queue. `/v1/models` says `images`
|
||||
for such a model.
|
||||
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
|
||||
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
|
||||
gate listens on the replica's port, demands the replica's key, and lets
|
||||
through only what the router asks. The model's `comfyui:` block names
|
||||
the workflow and, for a source that is a set of files, the checkpoint.
|
||||
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
|
||||
checked against a digest llmgw holds, since GitHub publishes none, and a
|
||||
version without one is refused. Its requirements are resolved as PyPI
|
||||
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
|
||||
CUDA 12.9 one for an older driver, and the result is checked on the card.
|
||||
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
|
||||
agent.yaml.
|
||||
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
|
||||
prompt's tokens. A model priced only per picture is not free, so a spent
|
||||
budget still stops it. Each usage row keeps how many pictures were made,
|
||||
shown by `llmgw ctl request`.
|
||||
- **`/images` on the portal**, beside the chat, for the models that make
|
||||
pictures. The chat no longer offers them. The page offers size, how many,
|
||||
seed, steps and a negative prompt, and shows each picture with its seed
|
||||
and a download link. Nothing is kept but the usage.
|
||||
- **An embedding model's `query_prefix`**, put before the question a
|
||||
knowledge-base search embeds and never before documents, for Qwen3-Embedding
|
||||
and models trained like it.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An awake on-demand model survived no restart or change of leader of the
|
||||
control plane.** A control plane that had heard no requests read every
|
||||
on-demand replica its nodes reported as wanted by nobody, and drained them
|
||||
all. An adopted replica now gets a whole idle timeout.
|
||||
- `ctl models` said `not placed` for an on-demand model that had been woken
|
||||
and was serving; it now says `ready (on demand)`.
|
||||
- A batch of embeddings was refused as one over-long prompt when the
|
||||
passages together exceeded the context, though each one fit. The longest
|
||||
input is what has to fit.
|
||||
- The chat's kept conversation sat in the browser under one key for
|
||||
everybody, so the next person to sign in on the same browser opened the
|
||||
last one's conversation. It is kept per account, and signing out clears
|
||||
what the pages stored.
|
||||
|
||||
## v0.14.1 — 2026-10-09
|
||||
|
||||
Adapters trained on the nodes from a knowledge base and kept private to
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
|
||||
@@ -7595,6 +7595,10 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"images": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"cost_micros": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
@@ -8565,6 +8569,24 @@
|
||||
},
|
||||
"embeddings": {
|
||||
"type": "boolean"
|
||||
},
|
||||
"images": {
|
||||
"type": "boolean",
|
||||
"description": "Images is a model that makes pictures, asked for at /v1/images/generations. Implied by an external model whose API is ComfyUI's."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.ComfyUI": {
|
||||
"type": "object",
|
||||
"description": "ComfyUI is a ComfyUI model's workflow and checkpoint.",
|
||||
"properties": {
|
||||
"workflow": {
|
||||
"type": "string",
|
||||
"description": "Workflow is a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane, in ComfyUI's API format."
|
||||
},
|
||||
"checkpoint": {
|
||||
"type": "string",
|
||||
"description": "Checkpoint is the file the workflow's {{checkpoint}} names: the source's own name unless set, which a source that is a set of files must."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8605,6 +8627,14 @@
|
||||
"peer": {
|
||||
"type": "string",
|
||||
"description": "Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up."
|
||||
},
|
||||
"api": {
|
||||
"type": "string",
|
||||
"description": "API is what the provider speaks: OpenAI's, unless this says comfyui — a ComfyUI server, asked for pictures at /v1/images/generations and sent the workflow below with the request's prompt in it. Model is then the checkpoint file the workflow loads, as ComfyUI lists it."
|
||||
},
|
||||
"workflow": {
|
||||
"type": "string",
|
||||
"description": "Workflow is the ComfyUI workflow, in its API format: a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8667,6 +8697,10 @@
|
||||
"$ref": "#/components/schemas/catalog.External",
|
||||
"description": "External is a model served by someone else — an OpenAI-compatible API outside the cluster — rather than by replicas placed here. Such a model has no profiles and nothing is placed for it; the router sends its requests to the provider. Most useful named in another model's fallbacks: used only when the cluster has no capacity left. Its prompts leave the cluster, so which keys may name it is worth an allowlist."
|
||||
},
|
||||
"comfyui": {
|
||||
"$ref": "#/components/schemas/catalog.ComfyUI",
|
||||
"description": "ComfyUI is how a model whose profiles run engine comfyui makes its pictures: the workflow, and the file it loads."
|
||||
},
|
||||
"adapters": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
@@ -8710,6 +8744,10 @@
|
||||
"$ref": "#/components/schemas/catalog.Capabilities",
|
||||
"description": "Capabilities is what this model can do beyond plain completion."
|
||||
},
|
||||
"query_prefix": {
|
||||
"type": "string",
|
||||
"description": "QueryPrefix is put before a question an embedding model embeds for a knowledge-base search; documents are embedded without it. Models trained with an instruction on the query side want it, Qwen3-Embedding \"Instruct: Given a web search query, retrieve relevant passages that answer the query\\nQuery: \" for one."
|
||||
},
|
||||
"role": {
|
||||
"type": "string",
|
||||
"description": "Role \"guard\" is a model that judges the gateway's traffic for the guardrails rather than serving clients: hidden from /v1/models and the chat page, refused to any key that names it, critical by default, and always running. Guard says how its answers are read."
|
||||
|
||||
+122
@@ -1270,6 +1270,128 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Pictures from ComfyUI
|
||||
|
||||
Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
|
||||
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
|
||||
waits, and sends the pictures back as `b64_json`.
|
||||
|
||||
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
|
||||
GPU node like any other, started when asked for, and stopped when unused.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
|
||||
```
|
||||
llmgw ctl config set engines.comfyui 0.39.2
|
||||
```
|
||||
|
||||
The agent fetches ComfyUI's source from GitHub and checks it against the
|
||||
digest llmgw holds for that version. GitHub publishes none for a tag's archive,
|
||||
so a version llmgw does not know is refused. The agent then builds a Python
|
||||
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
|
||||
dependencies are resolved as PyPI stood the day after the release. torch comes
|
||||
from PyPI (CUDA 13), or from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580. The install is checked on the node's GPU, and takes about 10 GB.
|
||||
`engine_overrides` can keep it off nodes that should not have it.
|
||||
|
||||
Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
|
||||
its environment in `.venv`, readable by the user engines run as
|
||||
(`llmgw-engine`). Name it in that node's `agent.yaml`; a node with one named
|
||||
there is not given the installed one:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
comfyui:
|
||||
dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python
|
||||
```
|
||||
|
||||
Then add the model:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
|
||||
comfyui:
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
scaling: {min: 0, max: 1, idle_timeout: 15m}
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: comfyui
|
||||
gpu_class: 6000-24g
|
||||
tp: 1
|
||||
```
|
||||
|
||||
The source is fetched as for any model. It can be one file, such as the
|
||||
single-file checkpoints the built-in workflows load. It can also be a mirror
|
||||
set (a URL ending in `/`) for a model in several files, such as a diffusion
|
||||
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
|
||||
own name, so a workflow of your own can load each one. A set must name the
|
||||
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
|
||||
is named by its own file name.
|
||||
|
||||
ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
|
||||
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
|
||||
listens on the replica's port, demands the replica's key, and lets through only
|
||||
what the router asks: queueing, waiting, fetching a picture and cancelling.
|
||||
Uploads and node installs are refused. Each replica has its own directory,
|
||||
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
|
||||
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
|
||||
`--disable-smart-memory` frees the card between pictures, at the cost of loading
|
||||
the model again for each one.
|
||||
|
||||
**On a ComfyUI server outside the cluster.** Make it an external model with
|
||||
`api: comfyui`:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: external
|
||||
external:
|
||||
url: http://10.0.0.147:8188 # ComfyUI, started with --listen
|
||||
model: flux1-schnell-fp8.safetensors # the checkpoint, as ComfyUI lists it
|
||||
api: comfyui
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
```
|
||||
|
||||
Like any external model, its requests count as leaving the cluster.
|
||||
|
||||
`flux-schnell` (the default) and `flux-dev` are built in. They suit the
|
||||
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
|
||||
For any other model, export the workflow from ComfyUI with **Save (API)** and
|
||||
name the file. In the file, put placeholders where the request's values go:
|
||||
|
||||
- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
|
||||
- `{{width}}` and `{{height}}` are the size.
|
||||
- `{{n}}` is the batch size.
|
||||
- `{{seed}}` is the seed.
|
||||
- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
|
||||
request leaves them out.
|
||||
- `{{checkpoint}}` is the `model` above.
|
||||
|
||||
A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
|
||||
of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
|
||||
send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
|
||||
back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
|
||||
nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
|
||||
before the picture is sent.
|
||||
|
||||
Requests for pictures pass the same checks as any other request: keys,
|
||||
allowlists, limits, admission, and the guardrails on the prompt. A client that
|
||||
leaves takes its workflow off the ComfyUI queue.
|
||||
|
||||
Each request's usage row records how many pictures it made (`llmgw ctl request
|
||||
ID` shows them). A picture's price is set with `per_image`, in micro-units like
|
||||
the other prices, and is charged on top of the prompt's tokens:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
pricing:
|
||||
flux-schnell: {prompt_per_1k: 0, per_image: 20000}
|
||||
```
|
||||
|
||||
An external ComfyUI on a card that llmgw also places models on should be
|
||||
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
|
||||
the card after each picture, and llmgw sees that memory as somebody else's.
|
||||
|
||||
### Federating clusters
|
||||
|
||||
Another llmgw cluster — another site, another team's GPUs — can be a provider
|
||||
|
||||
@@ -45,6 +45,15 @@ is available to everyone who may use them.
|
||||
A collection stays with the model it was made with: vectors from two
|
||||
models cannot be compared.
|
||||
|
||||
Some embedding models are trained with an instruction before the
|
||||
question and none before the documents. Give it as the entry's
|
||||
`query_prefix`; llmgw puts it before questions only. For
|
||||
Qwen3-Embedding:
|
||||
|
||||
```yaml
|
||||
query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
|
||||
```
|
||||
|
||||
With pgvector 0.7 or later a collection's index is kept at half
|
||||
precision (`halfvec`), which halves its memory and takes models of up to
|
||||
4000 dimensions; with an older pgvector the limit is 2000. A collection
|
||||
|
||||
@@ -60,6 +60,11 @@ rather than repeated.
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
| `images` | `bool` | [Capabilities](#capabilities) |
|
||||
| `workflow` | `string` | [ComfyUI](#comfyui) |
|
||||
| `checkpoint` | `string` | [ComfyUI](#comfyui) |
|
||||
| `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `python` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `interval` | `time.Duration` | [Control](#control) |
|
||||
| `base_backoff` | `time.Duration` | [Control](#control) |
|
||||
| `max_backoff` | `time.Duration` | [Control](#control) |
|
||||
@@ -76,6 +81,7 @@ rather than repeated.
|
||||
| `api_key` | `string` | [Endpoint](#endpoint) |
|
||||
| `models` | `[]string` | [Endpoint](#endpoint) |
|
||||
| `engine` | `string` | [Endpoint](#endpoint) |
|
||||
| `workflow` | `string` | [Endpoint](#endpoint) |
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
@@ -90,6 +96,8 @@ rather than repeated.
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -135,6 +143,7 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `comfyui` | `*ComfyUI` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
@@ -145,6 +154,7 @@ rather than repeated.
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
| `max_output` | `int` | [Model](#model) |
|
||||
| `capabilities` | `Capabilities` | [Model](#model) |
|
||||
| `query_prefix` | `string` | [Model](#model) |
|
||||
| `role` | `string` | [Model](#model) |
|
||||
| `guard` | `*GuardModel` | [Model](#model) |
|
||||
| `origin` | `*Origin` | [Model](#model) |
|
||||
@@ -342,6 +352,7 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
@@ -376,12 +387,14 @@ rather than repeated.
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -715,6 +728,47 @@ token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
`bool`
|
||||
|
||||
### `images`
|
||||
|
||||
`bool`
|
||||
|
||||
Images is a model that makes pictures, asked for at
|
||||
/v1/images/generations. Implied by an external model whose API is
|
||||
ComfyUI's.
|
||||
|
||||
## ComfyUI
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is a built-in's name (flux-schnell, the default, or
|
||||
flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
format.
|
||||
|
||||
### `checkpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
source's own name unless set, which a source that is a set of files
|
||||
must.
|
||||
|
||||
## ComfyUIConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
engines run as.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
## Control
|
||||
|
||||
### `interval`
|
||||
@@ -828,6 +882,13 @@ with one.
|
||||
Engine names the inference engine, which decides what its metrics are
|
||||
called. Empty means vLLM's names.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
name or its JSON; flux-schnell unless set.
|
||||
|
||||
### `pool`
|
||||
|
||||
`string`
|
||||
@@ -919,6 +980,23 @@ a model that federation added (`llmgw ctl federation add`) rather
|
||||
than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
goes with the peer's own list, and is sent only while the peer is up.
|
||||
|
||||
### `api`
|
||||
|
||||
`string`
|
||||
|
||||
API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
sent the workflow below with the request's prompt in it. Model is then
|
||||
the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
(flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
plane.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1305,6 +1383,15 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`*ComfyUI`
|
||||
|
||||
ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
pictures: the workflow, and the file it loads.
|
||||
|
||||
Takes a [ComfyUI](#comfyui) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
@@ -1389,6 +1476,16 @@ Capabilities is what this model can do beyond plain completion.
|
||||
|
||||
Takes a [Capabilities](#capabilities) block.
|
||||
|
||||
### `query_prefix`
|
||||
|
||||
`string`
|
||||
|
||||
QueryPrefix is put before a question an embedding model embeds for a
|
||||
knowledge-base search; documents are embedded without it. Models
|
||||
trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
"Instruct: Given a web search query, retrieve relevant passages that
|
||||
answer the query\nQuery: " for one.
|
||||
|
||||
### `role`
|
||||
|
||||
`string`
|
||||
@@ -2889,6 +2986,14 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is what one picture costs, for a model that makes them: a
|
||||
picture's cost is the card's time, which its prompt's few tokens do
|
||||
not measure.
|
||||
|
||||
## admission.Band
|
||||
|
||||
### `name`
|
||||
@@ -3131,6 +3236,13 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is the cost of one picture, for a model that makes them
|
||||
(/v1/images/generations), on top of its prompt's tokens.
|
||||
|
||||
## engines.Config
|
||||
|
||||
### `disabled`
|
||||
@@ -3174,6 +3286,14 @@ Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
Takes a [SimConfig](#simconfig) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`ComfyUIConfig`
|
||||
|
||||
ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,17 +39,27 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
t.Error("a model priced per picture reads as free, and would pass a spent budget")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroPricedModelsStayAvailableWhenTheBudgetIsSpent(t *testing.T) {
|
||||
// This is what makes budget fallback work: a key with nothing left can
|
||||
// still be served something that costs nothing.
|
||||
|
||||
@@ -18,6 +18,10 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
}
|
||||
|
||||
// Zero reports whether this model spends nothing.
|
||||
@@ -25,14 +29,15 @@ type Price struct {
|
||||
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 }
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
|
||||
func (p Price) Cost(promptTokens, completionTokens, images int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage
|
||||
}
|
||||
|
||||
func divUp(n, d int64) int64 {
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// ComfyUIConfig locates a ComfyUI installation.
|
||||
type ComfyUIConfig struct {
|
||||
// Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
// engines run as.
|
||||
Dir string `koanf:"dir"`
|
||||
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
Python string `koanf:"python"`
|
||||
}
|
||||
|
||||
// ComfyUI runs ComfyUI as an engine, behind llmgw's gate (llmgw
|
||||
// comfyui-gate): ComfyUI asks for no key, so it listens on the loopback and
|
||||
// the gate, which does, on the replica's port. Its replicas make pictures
|
||||
// for /v1/images/generations.
|
||||
type ComfyUI struct {
|
||||
cfg ComfyUIConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*ComfyUI) Name() string { return "comfyui" }
|
||||
|
||||
// HealthPath is the gate's, which answers once ComfyUI does.
|
||||
func (*ComfyUI) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is empty: ComfyUI publishes no metrics, and the router reads
|
||||
// none from it.
|
||||
func (*ComfyUI) MetricsPath() string { return "" }
|
||||
|
||||
// ServesHTTP is true: a ComfyUI replica is one process on one node.
|
||||
func (*ComfyUI) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the environment the agent installed:
|
||||
// its interpreter, with ComfyUI's code beside its bin, in comfyui/.
|
||||
func (c *ComfyUI) UseInstalled(python string) { c.installed.Store(&python) }
|
||||
|
||||
// Configured reports whether the configuration names a ComfyUI.
|
||||
func (c *ComfyUI) Configured() bool { return c.cfg.Dir != "" }
|
||||
|
||||
func (c *ComfyUI) dir() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return filepath.Join(filepath.Dir(filepath.Dir(*p)), "comfyui")
|
||||
}
|
||||
return c.cfg.Dir
|
||||
}
|
||||
|
||||
func (c *ComfyUI) python() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
if c.cfg.Python != "" {
|
||||
return c.cfg.Python
|
||||
}
|
||||
return filepath.Join(c.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var comfyVersion = regexp.MustCompile(`__version__\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports ComfyUI's version, from the file its build writes it to:
|
||||
// importing ComfyUI to ask would load torch and look for cards.
|
||||
func (c *ComfyUI) Probe(context.Context) (string, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no ComfyUI directory is configured")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "main.py")); err != nil {
|
||||
return "", fmt.Errorf("%s is not ComfyUI: %w", dir, err)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "comfyui_version.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // installed, of a version it does not say
|
||||
}
|
||||
if m := comfyVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. ComfyUI runs a
|
||||
// workflow on one device and has no tensor parallelism to spread it by.
|
||||
func (*ComfyUI) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: ComfyUI runs on one GPU of one node", p.Name)
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is the gate, which starts ComfyUI itself.
|
||||
func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return nil, fmt.Errorf("comfyui: no ComfyUI directory is configured (engines.comfyui.dir)")
|
||||
}
|
||||
bin, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := s.Profile
|
||||
args := []string{"comfyui-gate",
|
||||
"--listen", net.JoinHostPort(s.Host(), strconv.Itoa(s.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
"--weights", p.ModelPath,
|
||||
// Relative to where engines run — their user's home — so it is
|
||||
// theirs to write.
|
||||
"--state", filepath.Join("comfyui", s.ReplicaID),
|
||||
"--python", c.python(),
|
||||
"--comfyui", dir,
|
||||
"--",
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(s)
|
||||
if s.APIKey != "" {
|
||||
env["LLMGW_ENGINE_KEY"] = s.APIKey
|
||||
}
|
||||
return []Command{{Name: "comfyui", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestComfyUIRunsBehindTheGateWithItsKeyOutOfSight(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/comfyui"}}
|
||||
cmds, err := c.Commands(ShardSpec{
|
||||
ReplicaID: "r-flux-1", HTTPPort: 8100, BindAddr: "10.0.0.147", APIKey: "sekret", GPUIndices: []int{0},
|
||||
Profile: Profile{Name: "p", Engine: "comfyui", ModelPath: "/w/flux.safetensors", ServedModelName: "flux",
|
||||
Args: []string{"--disable-smart-memory"}},
|
||||
})
|
||||
if err != nil || len(cmds) != 1 {
|
||||
t.Fatalf("%v %v", cmds, err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
for _, want := range []string{"comfyui-gate", "--listen 10.0.0.147:8100", "--model flux", "--weights /w/flux.safetensors",
|
||||
"--state comfyui/r-flux-1", "--python /opt/comfyui/.venv/bin/python", "--comfyui /opt/comfyui", "-- --disable-smart-memory"} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("%q not in %q", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "sekret") {
|
||||
t.Error("the key is on the command line, where every user can read it")
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "LLMGW_ENGINE_KEY=sekret") || !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||
t.Error("the key or the card is not in the environment")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIRunsOnOneCard(t *testing.T) {
|
||||
c := &ComfyUI{}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 2}, h100(2)); err == nil {
|
||||
t.Error("a two-card ComfyUI profile was accepted")
|
||||
}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 1}, h100(1)); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if _, err := c.Commands(ShardSpec{Profile: Profile{ModelPath: "/w"}}); err == nil {
|
||||
t.Error("commands without a ComfyUI directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsOfferedByTheVersionItsBuildWrote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: dir}}
|
||||
if _, err := c.Probe(t.Context()); err == nil {
|
||||
t.Error("a directory without main.py passed for ComfyUI")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "main.py"), nil, 0o644)
|
||||
_ = os.WriteFile(filepath.Join(dir, "comfyui_version.py"), []byte("__version__ = \"0.39.2\"\n"), 0o644)
|
||||
if v, err := c.Probe(t.Context()); err != nil || v != "0.39.2" {
|
||||
t.Errorf("%q %v", v, err)
|
||||
}
|
||||
if got := NewRegistry(Config{ComfyUI: ComfyUIConfig{Dir: dir}}).ProbeConfigured(t.Context()); got["comfyui"] != "0.39.2" {
|
||||
t.Errorf("a configured ComfyUI is reported as %v", got)
|
||||
}
|
||||
if err := UnsafeArgs("comfyui", []string{"--enable-manager"}); err == nil {
|
||||
t.Error("a profile may install ComfyUI nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnInstalledComfyUIIsFoundBesideItsInterpreter(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/by-hand"}}
|
||||
c.UseInstalled("/opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python")
|
||||
cmds, err := c.Commands(ShardSpec{ReplicaID: "r", HTTPPort: 1, Profile: Profile{ModelPath: "/w", ServedModelName: "m"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
if !strings.Contains(args, "--python /opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python") ||
|
||||
!strings.Contains(args, "--comfyui /opt/llmgw/engines/comfyui/0.39.2-cu13/comfyui") {
|
||||
t.Errorf("the installed ComfyUI is not what runs: %s", args)
|
||||
}
|
||||
}
|
||||
@@ -214,6 +214,7 @@ func NewRegistry(cfg Config) *Registry {
|
||||
r.Add(&LlamaCpp{cfg: cfg.LlamaCpp})
|
||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||
r.Add(&Sim{cfg: cfg.Sim})
|
||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -262,6 +263,8 @@ type Config struct {
|
||||
// offers it only when its interpreter is named here.
|
||||
TRTLLM TRTLLMConfig `koanf:"trtllm"`
|
||||
Sim SimConfig `koanf:"sim"`
|
||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||
}
|
||||
|
||||
// --- shared helpers ---
|
||||
|
||||
@@ -477,7 +477,7 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
||||
|
||||
func TestRegistry(t *testing.T) {
|
||||
r := NewRegistry(Config{})
|
||||
if got := r.Names(); !slices.Equal(got, []string{"llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
t.Errorf("registry contents: %v", got)
|
||||
}
|
||||
if _, err := r.Get("vllm"); err != nil {
|
||||
|
||||
@@ -119,6 +119,19 @@ var flagsNever = map[string]map[string]string{
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
},
|
||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||
// where its files are.
|
||||
"comfyui": {
|
||||
"--listen": "is set by the agent", "--base-directory": "is set by the agent",
|
||||
"--extra-model-paths-config": "is set by the agent",
|
||||
"--output-directory": "writes files", "--temp-directory": "writes files",
|
||||
"--input-directory": "reads files", "--user-directory": "writes files",
|
||||
"--database-url": "writes a database elsewhere", "--front-end-root": "serves a directory",
|
||||
"--enable-cors-header": "lets other sites' pages call it",
|
||||
"--enable-manager": "installs nodes, which run Python",
|
||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||
"--comfy-api-base": "sends work to another service",
|
||||
},
|
||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||
// hyphenated names do not reach them.
|
||||
"trtllm": {
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// comfyuiSource is a ComfyUI release archive as GitHub makes one, and the
|
||||
// recipe pointed at it.
|
||||
func comfyuiSource(t *testing.T, in *Installer, rel *release, digest string) {
|
||||
t.Helper()
|
||||
archive := tarGz(t, map[string]string{
|
||||
"ComfyUI-0.39.2/main.py": "print('comfy')\n",
|
||||
"ComfyUI-0.39.2/comfyui_version.py": "__version__ = \"0.39.2\"\n",
|
||||
"ComfyUI-0.39.2/requirements.txt": "comfyui-frontend-package==1.53.10\ntorch\ntorchvision\n" +
|
||||
"numpy>=1.25.0\n\n#non essential dependencies:\nkornia>=0.7.1\n",
|
||||
}, nil)
|
||||
rel.assets["v0.39.2.tar.gz"] = archive
|
||||
sum := sha256.Sum256(archive)
|
||||
if digest == "" {
|
||||
digest = hex.EncodeToString(sum[:])
|
||||
}
|
||||
old := comfyuiReleases
|
||||
comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{"0.39.2": {digest, "2026-10-07T21:58:07Z"}}
|
||||
t.Cleanup(func() { comfyuiReleases = old })
|
||||
}
|
||||
|
||||
var turingNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
|
||||
NVIDIADriver: "580.82.07", MinComputeCapability: 7.5}
|
||||
|
||||
func TestComfyUIIsInstalledFromItsSourceCheckedAgainstTheRecipe(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready {
|
||||
t.Fatalf("not ready: %+v", got[0])
|
||||
}
|
||||
dir := filepath.Join(in.Root, "comfyui", "0.39.2-cu13")
|
||||
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "comfyui", "main.py")); err != nil {
|
||||
t.Errorf("ComfyUI's code is not beside the interpreter: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
for _, want := range []string{
|
||||
// Not PyPI's date, which ComfyUI has none of: the recipe's, a day on.
|
||||
"--exclude-newer 2026-10-08T21:58:07Z",
|
||||
"comfyui-frontend-package==1.53.10", "numpy>=1.25.0", "kornia>=0.7.1", "torch",
|
||||
"--require-hashes",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if strings.Contains(all, "/pypi/") {
|
||||
t.Error("PyPI was asked when ComfyUI was published")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIOnAnOlderDriverTakesPyTorchsCUDA12Torch(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
host := turingNode
|
||||
host.NVIDIADriver = "550.163.01"
|
||||
in, rel := pythonInstaller(t, host, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("got %+v", got[0])
|
||||
}
|
||||
var project string
|
||||
for _, c := range sys.commands {
|
||||
if strings.HasPrefix(c, "pyproject: ") {
|
||||
project = c
|
||||
}
|
||||
}
|
||||
// torch from PyTorch's index alone, named once, as the build names it.
|
||||
if !strings.Contains(project, `torch = { index = "extra" }`) || strings.Count(project, `"torch"`) != 1 {
|
||||
t.Errorf("torch is not from PyTorch's index alone:\n%s", project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIWhoseSourceIsNotTheOneTheRecipeNamesIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, strings.Repeat("ab", 32))
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match") {
|
||||
t.Fatalf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") {
|
||||
t.Errorf("installed from a source that did not match: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsInstalledOnlyWhereItCanRun(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, ubuntu, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}}); got[0].State != Skipped {
|
||||
t.Errorf("a node with no GPU: %+v", got[0])
|
||||
}
|
||||
in, _ = pythonInstaller(t, turingNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.1.0"}})
|
||||
if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "knows the source digest of ComfyUI 0.39.2") {
|
||||
t.Errorf("a version without a digest: %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequirementsFileIsReadForPackagesAndNothingElse(t *testing.T) {
|
||||
got, err := requirementsIn("torch\ntorchvision>=0.20\nPillow\n# a comment\naiohttp>=3.11 # inline\n", pytorchPackages)
|
||||
if err != nil || strings.Join(got, ",") != "Pillow,aiohttp>=3.11" {
|
||||
t.Errorf("%v %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"-r other.txt", "--index-url https://evil", "x @ https://evil/x.whl", "-e ."} {
|
||||
if _, err := requirementsIn(bad+"\n", nil); err == nil {
|
||||
t.Errorf("%q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -66,12 +66,17 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
|
||||
}
|
||||
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
var asOf string
|
||||
if !b.AsOf.IsZero() {
|
||||
asOf = b.AsOf.UTC().Format(time.RFC3339)
|
||||
} else {
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
asOf = published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
}
|
||||
asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
|
||||
cache := filepath.Join(in.root(), ".uv-cache")
|
||||
env := []string{
|
||||
@@ -96,7 +101,14 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
reqs := b.Requirements
|
||||
reqs := slices.Clone(b.Requirements)
|
||||
if b.Source != nil {
|
||||
more, err := in.unpackSource(ctx, w, b, tmp, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reqs = append(reqs, more...)
|
||||
}
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
@@ -320,3 +332,68 @@ func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Buil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackSource fetches an engine's source archive, checks it against the
|
||||
// digest the recipe holds, puts its code in the environment, and returns the
|
||||
// requirements its requirements file names — less the packages a second
|
||||
// index gives, which the build names itself.
|
||||
func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env string, st *Status) ([]string, error) {
|
||||
src := b.Source
|
||||
url := src.URL
|
||||
if strings.HasPrefix(url, "/") {
|
||||
url = strings.TrimSuffix(in.downloadBase(), "/") + url
|
||||
}
|
||||
scratch := env + ".source"
|
||||
_ = os.RemoveAll(scratch)
|
||||
if err := os.MkdirAll(scratch, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(scratch) //nolint:errcheck // scratch
|
||||
archive := filepath.Join(scratch, "source.tar.gz")
|
||||
name := w.Engine + " " + w.Version + "'s source"
|
||||
if err := in.download(ctx, url, name, archive, src.SHA256, st); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
in.progress(st, "unpacking %s", name)
|
||||
if err := untarGz(archive, scratch); err != nil {
|
||||
return nil, fmt.Errorf("unpacking %s: %w", name, err)
|
||||
}
|
||||
dst := filepath.Join(env, src.Dir)
|
||||
if err := os.Rename(filepath.Join(scratch, src.Top), dst); err != nil {
|
||||
return nil, fmt.Errorf("%s has no %s at its top: %w", name, src.Top, err)
|
||||
}
|
||||
if src.Requirements == "" {
|
||||
return nil, nil
|
||||
}
|
||||
f, err := os.ReadFile(filepath.Join(dst, src.Requirements))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return requirementsIn(string(f), b.IndexPackages)
|
||||
}
|
||||
|
||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||
// or install what the lock never saw.
|
||||
func requirementsIn(text string, skip []string) ([]string, error) {
|
||||
var out []string
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line, _, _ = strings.Cut(line, "#")
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "-") || strings.Contains(line, "://") || strings.Contains(line, " @ ") {
|
||||
return nil, fmt.Errorf("the requirements name %q, which is not a package from PyPI", line)
|
||||
}
|
||||
name := strings.ToLower(line)
|
||||
if i := strings.IndexAny(name, "<>=!~;[ "); i >= 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
if slices.Contains(skip, name) {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -53,6 +53,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return []byte("error: Hash mismatch for torch\n"), errExit
|
||||
}
|
||||
venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
|
||||
_ = os.WriteFile(filepath.Join(venv, "bin", "python"), []byte("#!/bin/sh\n"), 0o755)
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
@@ -68,12 +69,31 @@ type Build struct {
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// Source, for a Python engine that is not on PyPI, is its own code: an
|
||||
// archive pinned by its digest, unpacked into the environment, whose
|
||||
// requirements file is installed with Requirements. AsOf is then the
|
||||
// date its dependencies are resolved as of — the day after its version
|
||||
// was published, which PyPI cannot be asked for.
|
||||
Source *Source
|
||||
AsOf time.Time
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Source is an engine's code as an archive.
|
||||
type Source struct {
|
||||
// URL is fetched as is; SHA256 is what it must hash to, written into
|
||||
// the recipe because a project's source archives come with no published
|
||||
// digest of their own.
|
||||
URL, SHA256 string
|
||||
// Top is the archive's one top directory, put in the environment as Dir.
|
||||
Top, Dir string
|
||||
// Requirements is the requirements file, in Dir.
|
||||
Requirements string
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
@@ -104,6 +124,7 @@ var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
@@ -556,3 +577,78 @@ func orUnknown(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// --- ComfyUI ---
|
||||
|
||||
// comfyuiReleases are the ComfyUI versions this agent installs: each source
|
||||
// archive's digest, taken when it was added here and confirmed from a second
|
||||
// machine, and when the version was published. GitHub publishes no digest
|
||||
// for a tag's archive, so a version not listed is refused rather than
|
||||
// installed unchecked.
|
||||
var comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{
|
||||
"0.39.2": {"1d655e22cf546700fc1d6d2536e27701695c5832a51f63bb16c41866ac09be0b", "2026-10-07T21:58:07Z"},
|
||||
}
|
||||
|
||||
// ComfyUIRepo is where ComfyUI's source is published.
|
||||
const ComfyUIRepo = "Comfy-Org/ComfyUI"
|
||||
|
||||
// comfyuiDir is where ComfyUI's code is put in its environment: main.py is
|
||||
// <environment>/comfyui/main.py, beside bin/python.
|
||||
const comfyuiDir = "comfyui"
|
||||
|
||||
// comfyuiCandidates are the ComfyUI builds for a machine: its source, in a
|
||||
// Python environment with the torch for the node's driver — PyPI's, built
|
||||
// for CUDA 13, or PyTorch's CUDA 12.9 build for a driver CUDA 13 does not
|
||||
// run on.
|
||||
func comfyuiCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("ComfyUI", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rel, ok := comfyuiReleases[version]
|
||||
if !ok {
|
||||
known := make([]string, 0, len(comfyuiReleases))
|
||||
for v := range comfyuiReleases {
|
||||
known = append(known, v)
|
||||
}
|
||||
slices.Sort(known)
|
||||
return nil, fmt.Errorf("this agent knows the source digest of ComfyUI %s, not of %s; "+
|
||||
"install that one by hand and name it in agent.yaml (engines.comfyui.dir)",
|
||||
strings.Join(known, ", "), version)
|
||||
}
|
||||
published, err := time.Parse(time.RFC3339, rel.published)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
build := func(variant string) Build {
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Source: &Source{
|
||||
URL: "/" + ComfyUIRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||
SHA256: rel.sha256, Top: "ComfyUI-" + version, Dir: comfyuiDir,
|
||||
Requirements: "requirements.txt",
|
||||
},
|
||||
AsOf: published.Add(24 * time.Hour),
|
||||
// ComfyUI is started as a script, so what the agent is given is
|
||||
// the environment's interpreter; the proof is importing what it
|
||||
// imports first, from where its code is.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", `import os, sys, torch, aiohttp, safetensors
|
||||
d = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(sys.executable))), "` + comfyuiDir + `")
|
||||
assert os.path.isfile(os.path.join(d, "main.py")), d + " has no main.py"
|
||||
print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 15,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := build("cu129")
|
||||
cu129.ExtraIndex, cu129.IndexPackages = pytorchCU129, pytorchPackages
|
||||
return pick("ComfyUI "+version, variant, []candidate{
|
||||
{build("cu13"), gpuNeeds{minDriverForCUDA13, 7.5}, "ComfyUI " + version + " on CUDA 13's torch"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, 7.0}, "ComfyUI " + version + " on CUDA 12.9's torch"},
|
||||
}, h)
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
)
|
||||
|
||||
// comfyKeyEnv carries the replica's key to the gate: the environment, which
|
||||
// only the engine's user and root can read, rather than the command line,
|
||||
// which every user on the node can.
|
||||
const comfyKeyEnv = "LLMGW_ENGINE_KEY"
|
||||
|
||||
func newComfyUIGateCmd() *cobra.Command {
|
||||
var cfg comfy.GateConfig
|
||||
cmd := &cobra.Command{
|
||||
Use: "comfyui-gate [flags] [-- main.py flags]",
|
||||
Short: "Run ComfyUI as an engine: behind a key, on the loopback (started by the agent)",
|
||||
Hidden: true,
|
||||
Long: "Started by the node agent for a replica whose engine is comfyui. ComfyUI asks\n" +
|
||||
"nobody for a key, so it listens on the loopback only and this gate on the\n" +
|
||||
"replica's port, letting through, with the replica's key, only what the router\n" +
|
||||
"asks of it. It also lays the fetched weights out where ComfyUI looks for them.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
cfg.Key = os.Getenv(comfyKeyEnv)
|
||||
_ = os.Unsetenv(comfyKeyEnv)
|
||||
cfg.Args = args
|
||||
cfg.Log = logger()
|
||||
return comfy.RunGate(ctx, cfg)
|
||||
},
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&cfg.Listen, "listen", "", "the replica's address")
|
||||
f.StringVar(&cfg.Model, "model", "", "the model's name, for /v1/models")
|
||||
f.StringVar(&cfg.Weights, "weights", "", "the fetched weights: a file or a directory")
|
||||
f.StringVar(&cfg.State, "state", "", "the replica's own directory")
|
||||
f.StringVar(&cfg.Python, "python", "python3", "the interpreter of ComfyUI's environment")
|
||||
f.StringVar(&cfg.Dir, "comfyui", "", "ComfyUI's directory, where main.py is")
|
||||
for _, n := range []string{"listen", "weights", "state", "comfyui"} {
|
||||
_ = cmd.MarkFlagRequired(n)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
@@ -331,6 +331,9 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
return "stopped"
|
||||
case m.Ready >= m.Min && m.Min > 0:
|
||||
return "ready"
|
||||
case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total:
|
||||
// Asked for, and up until it goes unused for long enough.
|
||||
return "ready (on demand)"
|
||||
case m.Total > m.Ready:
|
||||
return "starting"
|
||||
default:
|
||||
|
||||
@@ -56,6 +56,9 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
fmt.Fprintf(out, "status %d\n", v.Status)
|
||||
fmt.Fprintf(out, "time %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
|
||||
fmt.Fprintf(out, "tokens %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
|
||||
if v.Images > 0 {
|
||||
fmt.Fprintf(out, "images %d\n", v.Images)
|
||||
}
|
||||
if v.CacheHit {
|
||||
fmt.Fprintf(out, "cache answered from the project's answer cache: nothing generated, nothing charged\n")
|
||||
}
|
||||
|
||||
@@ -54,6 +54,7 @@ func Root() *cobra.Command {
|
||||
newMirrorCmd(),
|
||||
newBundleCmd(),
|
||||
newSimEngineCmd(),
|
||||
newComfyUIGateCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
|
||||
@@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) {
|
||||
{"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"},
|
||||
{"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"},
|
||||
{"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"},
|
||||
{"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"},
|
||||
{"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := modelStatus(tc.view); got != tc.want {
|
||||
|
||||
@@ -0,0 +1,529 @@
|
||||
// Package comfy has images made by ComfyUI: a workflow filled in with what
|
||||
// was asked for, queued, waited on, and its pictures fetched.
|
||||
//
|
||||
// ComfyUI speaks no OpenAI. A workflow is a graph of nodes that is queued with
|
||||
// POST /prompt, is done when GET /history/{id} lists it, and leaves its
|
||||
// pictures to be read one by one from GET /view. What the graph is depends on
|
||||
// the model — FLUX schnell is not FLUX dev, and neither is SDXL — so the graph
|
||||
// is a template the catalog names, with placeholders where a request's
|
||||
// prompt, size and seed go.
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed workflows/*.json
|
||||
var builtin embed.FS
|
||||
|
||||
// DefaultWorkflow is the template used when the catalog names none.
|
||||
const DefaultWorkflow = "flux-schnell"
|
||||
|
||||
// Builtins are the workflows that come with llmgw, by name.
|
||||
func Builtins() []string {
|
||||
entries, _ := builtin.ReadDir("workflows")
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Template is a workflow in ComfyUI's API format, with placeholders.
|
||||
//
|
||||
// A placeholder is a string value "{{name}}", or "{{name|default}}" for one a
|
||||
// request may leave out. One that is the whole value becomes the value
|
||||
// itself, a number where the node wants a number; one inside a longer string
|
||||
// is written into it. The names are those of Params.
|
||||
type Template struct {
|
||||
graph map[string]any
|
||||
}
|
||||
|
||||
// Load reads a template: a built-in's name, or the workflow's JSON itself.
|
||||
func Load(nameOrJSON string) (*Template, error) {
|
||||
s := strings.TrimSpace(nameOrJSON)
|
||||
if s == "" {
|
||||
s = DefaultWorkflow
|
||||
}
|
||||
raw := []byte(s)
|
||||
if !strings.HasPrefix(s, "{") {
|
||||
b, err := builtin.ReadFile("workflows/" + s + ".json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("no built-in workflow %q (there are %s)", s, strings.Join(Builtins(), ", "))
|
||||
}
|
||||
raw = b
|
||||
}
|
||||
var g map[string]any
|
||||
if err := json.Unmarshal(raw, &g); err != nil {
|
||||
return nil, fmt.Errorf("the workflow is not JSON: %w", err)
|
||||
}
|
||||
if len(g) == 0 {
|
||||
return nil, errors.New("the workflow has no nodes")
|
||||
}
|
||||
for id, n := range g {
|
||||
node, ok := n.(map[string]any)
|
||||
if !ok || node["class_type"] == nil {
|
||||
// The UI's own save format has a "nodes" list instead, and is
|
||||
// not something /prompt accepts.
|
||||
return nil, fmt.Errorf("node %q has no class_type: the workflow must be saved in API format", id)
|
||||
}
|
||||
}
|
||||
return &Template{graph: g}, nil
|
||||
}
|
||||
|
||||
// Params are what a request decides. A zero number is one not given, which
|
||||
// leaves it to the template's default.
|
||||
type Params struct {
|
||||
Checkpoint string
|
||||
Prompt string
|
||||
NegativePrompt string
|
||||
Width, Height int
|
||||
N int
|
||||
Seed int64
|
||||
Steps int
|
||||
Guidance float64
|
||||
}
|
||||
|
||||
func (p Params) value(name string) (any, bool) {
|
||||
switch name {
|
||||
case "checkpoint":
|
||||
return p.Checkpoint, p.Checkpoint != ""
|
||||
case "prompt":
|
||||
return p.Prompt, true
|
||||
case "negative_prompt":
|
||||
return p.NegativePrompt, true
|
||||
case "width":
|
||||
return p.Width, p.Width > 0
|
||||
case "height":
|
||||
return p.Height, p.Height > 0
|
||||
case "n":
|
||||
return max(p.N, 1), true
|
||||
case "seed":
|
||||
return p.Seed, true
|
||||
case "steps":
|
||||
return p.Steps, p.Steps > 0
|
||||
case "guidance":
|
||||
return p.Guidance, p.Guidance > 0
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Fill is the workflow for one request.
|
||||
func (t *Template) Fill(p Params) (map[string]any, error) {
|
||||
var missing []string
|
||||
var fill func(v any) any
|
||||
fill = func(v any) any {
|
||||
switch v := v.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for k, x := range v {
|
||||
out[k] = fill(x)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, x := range v {
|
||||
out[i] = fill(x)
|
||||
}
|
||||
return out
|
||||
case string:
|
||||
return fillString(v, p, &missing)
|
||||
}
|
||||
return v
|
||||
}
|
||||
g := fill(t.graph).(map[string]any)
|
||||
if len(missing) > 0 {
|
||||
slices.Sort(missing)
|
||||
return nil, fmt.Errorf("the workflow wants %s, which the request did not give and the workflow has no default for",
|
||||
strings.Join(slices.Compact(missing), ", "))
|
||||
}
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func fillString(s string, p Params, missing *[]string) any {
|
||||
if !strings.Contains(s, "{{") {
|
||||
return s
|
||||
}
|
||||
resolve := func(ph string) (any, bool) {
|
||||
name, def, hasDef := strings.Cut(ph, "|")
|
||||
name = strings.TrimSpace(name)
|
||||
if v, ok := p.value(name); ok {
|
||||
return v, true
|
||||
}
|
||||
if hasDef {
|
||||
def = strings.TrimSpace(def)
|
||||
if f, err := strconv.ParseFloat(def, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
return def, true
|
||||
}
|
||||
*missing = append(*missing, name)
|
||||
return nil, false
|
||||
}
|
||||
// The whole value: what the node gets is the value, not its text.
|
||||
if strings.HasPrefix(s, "{{") && strings.HasSuffix(s, "}}") && strings.Count(s, "{{") == 1 {
|
||||
v, _ := resolve(s[2 : len(s)-2])
|
||||
return v
|
||||
}
|
||||
var b strings.Builder
|
||||
for {
|
||||
i := strings.Index(s, "{{")
|
||||
j := strings.Index(s, "}}")
|
||||
if i < 0 || j < i {
|
||||
b.WriteString(s)
|
||||
return b.String()
|
||||
}
|
||||
b.WriteString(s[:i])
|
||||
if v, ok := resolve(s[i+2 : j]); ok {
|
||||
fmt.Fprint(&b, v)
|
||||
}
|
||||
s = s[j+2:]
|
||||
}
|
||||
}
|
||||
|
||||
// RandomSeed is a seed for a request that named none. ComfyUI's are 64-bit,
|
||||
// but JSON numbers are read as doubles on the way: kept to 2^53.
|
||||
func RandomSeed() int64 {
|
||||
n, err := rand.Int(rand.Reader, big.NewInt(1<<53))
|
||||
if err != nil {
|
||||
return time.Now().UnixNano() & (1<<53 - 1)
|
||||
}
|
||||
return n.Int64()
|
||||
}
|
||||
|
||||
// Image is one picture made.
|
||||
type Image struct {
|
||||
Data []byte
|
||||
ContentType string
|
||||
}
|
||||
|
||||
// Error is a failure, said as what it means for the request.
|
||||
type Error struct {
|
||||
Msg string
|
||||
// Retryable is a failure of this server, which another one might not
|
||||
// have: it could not be reached, or it answered 5xx.
|
||||
Retryable bool
|
||||
}
|
||||
|
||||
func (e *Error) Error() string { return e.Msg }
|
||||
|
||||
// Client queues workflows and waits for them.
|
||||
type Client struct {
|
||||
HTTP *http.Client
|
||||
// Poll is how often a queued workflow is asked after.
|
||||
Poll time.Duration
|
||||
}
|
||||
|
||||
// Generate runs a workflow on the ComfyUI at base and returns its pictures,
|
||||
// in the order of the nodes that made them. A context that ends takes the
|
||||
// workflow out of the queue, or interrupts it if it is running, so a client
|
||||
// that went away does not leave the card working for nobody.
|
||||
func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
client := make([]byte, 8)
|
||||
_, _ = rand.Read(client)
|
||||
body, err := json.Marshal(map[string]any{"prompt": workflow, "client_id": "llmgw-" + hex.EncodeToString(client)})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var queued struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Error json.RawMessage `json:"error"`
|
||||
NodeErrors json.RawMessage `json:"node_errors"`
|
||||
}
|
||||
status, err := c.do(ctx, http.MethodPost, base+"/prompt", apiKey, body, &queued)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status != http.StatusOK || queued.PromptID == "" {
|
||||
// A workflow ComfyUI will not run: a checkpoint it does not have, a
|
||||
// node it does not know. The catalog's to fix, not the client's, and
|
||||
// no other replica set up the same way would take it either.
|
||||
return nil, &Error{Msg: "ComfyUI refused the workflow: " + refusal(queued.Error, queued.NodeErrors)}
|
||||
}
|
||||
id := queued.PromptID
|
||||
|
||||
poll := c.Poll
|
||||
if poll <= 0 {
|
||||
poll = 250 * time.Millisecond
|
||||
}
|
||||
t := time.NewTicker(poll)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.cancel(context.WithoutCancel(ctx), base, apiKey, id)
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
var hist map[string]historyEntry
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
continue // said by the select
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
h, ok := hist[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if h.Status.StatusStr == "error" {
|
||||
return nil, &Error{Msg: "the workflow failed in ComfyUI: " + h.failure()}
|
||||
}
|
||||
if !h.Status.Completed {
|
||||
continue
|
||||
}
|
||||
return c.fetch(ctx, base, apiKey, h)
|
||||
}
|
||||
}
|
||||
|
||||
type historyEntry struct {
|
||||
Status struct {
|
||||
StatusStr string `json:"status_str"`
|
||||
Completed bool `json:"completed"`
|
||||
Messages [][]json.RawMessage `json:"messages"`
|
||||
} `json:"status"`
|
||||
Outputs map[string]struct {
|
||||
Images []struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
} `json:"images"`
|
||||
} `json:"outputs"`
|
||||
}
|
||||
|
||||
// failure is what ComfyUI said went wrong, from its execution_error.
|
||||
func (h historyEntry) failure() string {
|
||||
for _, m := range h.Status.Messages {
|
||||
if len(m) < 2 || string(m[0]) != `"execution_error"` {
|
||||
continue
|
||||
}
|
||||
var e struct {
|
||||
NodeType string `json:"node_type"`
|
||||
Message string `json:"exception_message"`
|
||||
}
|
||||
if json.Unmarshal(m[1], &e) == nil && e.Message != "" {
|
||||
return strings.TrimSpace(e.NodeType + ": " + strings.TrimSpace(e.Message))
|
||||
}
|
||||
}
|
||||
return "no reason given"
|
||||
}
|
||||
|
||||
func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry) ([]Image, error) {
|
||||
nodes := make([]string, 0, len(h.Outputs))
|
||||
for id := range h.Outputs {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
// Numerically where they are numbers, which is how ComfyUI names them.
|
||||
slices.SortFunc(nodes, func(a, b string) int {
|
||||
x, ea := strconv.Atoi(a)
|
||||
y, eb := strconv.Atoi(b)
|
||||
if ea == nil && eb == nil {
|
||||
return x - y
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
var out []Image
|
||||
for _, n := range nodes {
|
||||
for _, im := range h.Outputs[n].Images {
|
||||
q := url.Values{"filename": {im.Filename}, "subfolder": {im.Subfolder}, "type": {im.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, &Error{Msg: "the workflow finished without a picture: it has no SaveImage or PreviewImage node"}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// cancel takes a workflow out of the queue, or stops it if it is the one
|
||||
// running. Best effort: the server may be gone, which also stops it.
|
||||
func (c *Client) cancel(ctx context.Context, base, apiKey, id string) {
|
||||
ctx, done := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer done()
|
||||
body, _ := json.Marshal(map[string]any{"delete": []string{id}})
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/queue", apiKey, body, nil)
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return
|
||||
}
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == strconv.Quote(id) {
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/interrupt", apiKey, []byte("{}"), nil)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
return http.DefaultClient
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method, u, apiKey string, body []byte) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return nil, &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// do sends a request and reads a JSON answer into out. A 5xx is an error;
|
||||
// any other status is returned with whatever body could be read.
|
||||
func (c *Client) do(ctx context.Context, method, u, apiKey string, body []byte, out any) (int, error) {
|
||||
resp, err := c.request(ctx, method, u, apiKey, body)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
|
||||
if err != nil {
|
||||
return 0, &Error{Msg: "reading ComfyUI's answer: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
return resp.StatusCode, &Error{Msg: fmt.Sprintf("ComfyUI answered %d: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
if out != nil && len(b) > 0 {
|
||||
if err := json.Unmarshal(b, out); err != nil && resp.StatusCode == http.StatusOK {
|
||||
return resp.StatusCode, &Error{Msg: "ComfyUI's answer is not JSON: " + snippet(b)}
|
||||
}
|
||||
}
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
|
||||
resp, err := c.request(ctx, http.MethodGet, u, apiKey, nil)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
|
||||
if err != nil {
|
||||
return Image{}, &Error{Msg: "reading a picture from ComfyUI: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a picture it made", resp.StatusCode)}
|
||||
}
|
||||
ct := resp.Header.Get("Content-Type")
|
||||
if ct == "" || ct == "application/octet-stream" {
|
||||
ct = http.DetectContentType(b)
|
||||
}
|
||||
if ct == "image/png" {
|
||||
b = withoutPNGText(b)
|
||||
}
|
||||
return Image{Data: b, ContentType: ct}, nil
|
||||
}
|
||||
|
||||
// withoutPNGText is a PNG without its text chunks. ComfyUI writes the whole
|
||||
// workflow into every picture it saves — the prompt, and the checkpoint and
|
||||
// nodes behind it — and what a picture says about how it was made is the
|
||||
// operator's to publish, not something to hand every client. A file that is
|
||||
// not a well-formed PNG is returned as it came.
|
||||
func withoutPNGText(b []byte) []byte {
|
||||
const sig = "\x89PNG\r\n\x1a\n"
|
||||
if len(b) < len(sig) || string(b[:len(sig)]) != sig {
|
||||
return b
|
||||
}
|
||||
out := make([]byte, 0, len(b))
|
||||
out = append(out, b[:len(sig)]...)
|
||||
for i := len(sig); i < len(b); {
|
||||
if i+12 > len(b) {
|
||||
return b
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(b[i:]))
|
||||
end := i + 12 + n
|
||||
if n < 0 || end > len(b) {
|
||||
return b
|
||||
}
|
||||
switch string(b[i+4 : i+8]) {
|
||||
case "tEXt", "iTXt", "zTXt":
|
||||
default:
|
||||
out = append(out, b[i:end]...)
|
||||
}
|
||||
i = end
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// refusal is what /prompt said when it would not queue a workflow.
|
||||
func refusal(e, nodes json.RawMessage) string {
|
||||
var top struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
}
|
||||
msg := snippet(e)
|
||||
if json.Unmarshal(e, &top) == nil && top.Message != "" {
|
||||
msg = top.Message
|
||||
if top.Details != "" {
|
||||
msg += ": " + top.Details
|
||||
}
|
||||
}
|
||||
var perNode map[string]struct {
|
||||
ClassType string `json:"class_type"`
|
||||
Errors []struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
} `json:"errors"`
|
||||
}
|
||||
if json.Unmarshal(nodes, &perNode) == nil {
|
||||
var parts []string
|
||||
for _, n := range perNode {
|
||||
for _, er := range n.Errors {
|
||||
parts = append(parts, strings.TrimSpace(n.ClassType+": "+er.Message+" "+er.Details))
|
||||
}
|
||||
}
|
||||
slices.Sort(parts)
|
||||
if len(parts) > 0 {
|
||||
msg += " (" + strings.Join(parts, "; ") + ")"
|
||||
}
|
||||
}
|
||||
if msg == "" {
|
||||
msg = "no reason given"
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
func snippet(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
for _, name := range Builtins() {
|
||||
tpl, err := Load(name)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "flux.safetensors", Prompt: `a "red" fox`, Width: 768, Height: 512, N: 2, Seed: 42})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
b, _ := json.Marshal(g)
|
||||
s := string(b)
|
||||
for _, want := range []string{`"ckpt_name":"flux.safetensors"`, `"text":"a \"red\" fox"`, `"width":768`,
|
||||
`"height":512`, `"batch_size":2`, `"seed":42`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("%s: %s not in %s", name, want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "{{") {
|
||||
t.Errorf("%s: a placeholder was left: %s", name, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
|
||||
tpl, _ := Load("flux-schnell")
|
||||
g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 4.0 {
|
||||
t.Errorf("default steps = %v, want 4", steps)
|
||||
}
|
||||
g, _ = tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512, Steps: 8})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 8 {
|
||||
t.Errorf("steps = %v, want 8", steps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPlaceholderWithoutAValueOrADefaultIsSaid(t *testing.T) {
|
||||
tpl, err := Load(`{"1":{"class_type":"X","inputs":{"a":"{{checkpoint}}","b":"size {{width}}x{{height}}"}}}`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tpl.Fill(Params{Width: 5}); err == nil || !strings.Contains(err.Error(), "checkpoint, height") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "c", Width: 5, Height: 6})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b := g["1"].(map[string]any)["inputs"].(map[string]any)["b"]; b != "size 5x6" {
|
||||
t.Errorf("b = %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowInTheUIsFormatIsRefused(t *testing.T) {
|
||||
if _, err := Load(`{"nodes":[{"id":1}],"links":[]}`); err == nil || !strings.Contains(err.Error(), "API format") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
if _, err := Load("sdxl-turbo"); err == nil {
|
||||
t.Fatal("an unknown built-in was loaded")
|
||||
}
|
||||
}
|
||||
|
||||
// fakeComfy is just enough of ComfyUI: /prompt queues, /history answers
|
||||
// after a few asks, /view serves a picture.
|
||||
type fakeComfy struct {
|
||||
mu sync.Mutex
|
||||
asked int
|
||||
fail string
|
||||
refuse bool
|
||||
deleted []string
|
||||
queued map[string]any
|
||||
readyAt int
|
||||
srv *httptest.Server
|
||||
interrupted bool
|
||||
}
|
||||
|
||||
func newFakeComfy(t *testing.T) *fakeComfy {
|
||||
f := &fakeComfy{readyAt: 2}
|
||||
f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
switch {
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/prompt":
|
||||
if f.refuse {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
fmt.Fprint(w, `{"error":{"message":"Prompt outputs failed validation","details":""},"node_errors":{"1":{"class_type":"CheckpointLoaderSimple","errors":[{"message":"Value not in list","details":"ckpt_name: 'x' not in []"}]}}}`)
|
||||
return
|
||||
}
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.queued = b.Prompt
|
||||
fmt.Fprint(w, `{"prompt_id":"p1","number":1,"node_errors":{}}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
f.asked++
|
||||
if f.asked < f.readyAt {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
if f.fail != "" {
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"error","completed":false,"messages":[["execution_start",{}],["execution_error",{"node_type":"KSampler","exception_message":%q}]]},"outputs":{}}}`, f.fail)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true,"messages":[]},"outputs":{"7":{"images":[{"filename":"a.png","subfolder":"","type":"temp"},{"filename":"b.png","subfolder":"","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "PNG:"+r.URL.Query().Get("filename"))
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost:
|
||||
var b struct {
|
||||
Delete []string `json:"delete"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.deleted = append(f.deleted, b.Delete...)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"p1",{},{},[]]],"queue_pending":[]}`)
|
||||
case r.URL.Path == "/interrupt":
|
||||
f.interrupted = true
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(f.srv.Close)
|
||||
return f
|
||||
}
|
||||
|
||||
func TestGenerateWaitsForTheWorkflowAndFetchesItsPictures(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
images, err := c.Generate(t.Context(), f.srv.URL, "", map[string]any{"1": map[string]any{"class_type": "X"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(images) != 2 || string(images[0].Data) != "PNG:a.png" || images[1].ContentType != "image/png" {
|
||||
t.Fatalf("images = %+v", images)
|
||||
}
|
||||
if f.queued["1"] == nil {
|
||||
t.Error("the workflow was not what was queued")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowThatFailsSaysWhy(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.fail = "CUDA out of memory"
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if err == nil || !strings.Contains(err.Error(), "KSampler: CUDA out of memory") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowComfyUIRefusesIsNotRetried(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.refuse = true
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
ce, ok := err.(*Error)
|
||||
if !ok || ce.Retryable || !strings.Contains(ce.Msg, "Value not in list") {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIThatCannotBeReachedIsRetryable(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.srv.Close()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if ce, ok := err.(*Error); !ok || !ce.Retryable {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientThatLeavesTakesItsWorkflowOffTheCard(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.readyAt = 1 << 30
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(ctx, f.srv.URL, "", map[string]any{})
|
||||
if err == nil {
|
||||
t.Fatal("no error")
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if len(f.deleted) != 1 || f.deleted[0] != "p1" || !f.interrupted {
|
||||
t.Errorf("deleted %v, interrupted %v", f.deleted, f.interrupted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
|
||||
chunk := func(kind, data string) []byte {
|
||||
b := make([]byte, 0, 12+len(data))
|
||||
b = binary.BigEndian.AppendUint32(b, uint32(len(data)))
|
||||
b = append(b, kind...)
|
||||
b = append(b, data...)
|
||||
return append(b, 0, 0, 0, 0) // a CRC nobody here checks
|
||||
}
|
||||
png := []byte("\x89PNG\r\n\x1a\n")
|
||||
png = append(png, chunk("IHDR", "0123456789abc")...)
|
||||
png = append(png, chunk("tEXt", "prompt\x00{\"1\":...}")...)
|
||||
png = append(png, chunk("IDAT", "pixels")...)
|
||||
png = append(png, chunk("iTXt", "workflow\x00...")...)
|
||||
png = append(png, chunk("IEND", "")...)
|
||||
got := string(withoutPNGText(png))
|
||||
if strings.Contains(got, "prompt") || strings.Contains(got, "workflow") {
|
||||
t.Errorf("text left in: %q", got)
|
||||
}
|
||||
for _, kind := range []string{"IHDR", "IDAT", "pixels", "IEND"} {
|
||||
if !strings.Contains(got, kind) {
|
||||
t.Errorf("%s gone: %q", kind, got)
|
||||
}
|
||||
}
|
||||
if broken := png[:30]; string(withoutPNGText(broken)) != string(broken) {
|
||||
t.Error("a truncated PNG was changed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,334 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The gate is how a node runs ComfyUI as one of llmgw's engines.
|
||||
//
|
||||
// ComfyUI asks nobody for a key: whoever reaches its port can queue work,
|
||||
// upload files and install nodes. Every other engine here demands the key
|
||||
// the control plane made for its replica, so ComfyUI is put behind a gate
|
||||
// that does: ComfyUI listens on the loopback only, and the gate on the
|
||||
// replica's port, letting through only the requests the router makes. The
|
||||
// gate also answers what the agent asks every engine — /health once
|
||||
// ComfyUI is up, and /v1/models naming the model — and lays the weights out
|
||||
// where ComfyUI looks for them.
|
||||
//
|
||||
// ComfyUI runs as the gate's child, in its process group, so stopping the
|
||||
// replica stops both, the card memory ComfyUI holds is counted as the
|
||||
// replica's, and either one exiting ends the other.
|
||||
|
||||
// GateConfig is one replica's gate.
|
||||
type GateConfig struct {
|
||||
// Listen is the replica's address, the one the router is given.
|
||||
Listen string
|
||||
// Model is the name the agent asks /v1/models for.
|
||||
Model string
|
||||
// Key is what every request must carry as a bearer token; empty lets
|
||||
// every request through, as an engine started without one does.
|
||||
Key string
|
||||
// Weights is the model as the agent fetched it: a file, or a directory
|
||||
// of files.
|
||||
Weights string
|
||||
// State is the replica's own directory: ComfyUI's base directory, with
|
||||
// its output, temp and user directories, and the models it is shown.
|
||||
State string
|
||||
// Python runs ComfyUI's main.py, found in Dir.
|
||||
Python, Dir string
|
||||
// Args are passed on to main.py, after the gate's own.
|
||||
Args []string
|
||||
// Output is where ComfyUI's output goes.
|
||||
Output io.Writer
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// The requests the router makes of ComfyUI: queueing, waiting, fetching and
|
||||
// cancelling. Nothing that uploads, installs or changes ComfyUI.
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info"}
|
||||
|
||||
// modelFolders are the kinds of file ComfyUI's loaders look for. Each is
|
||||
// shown the same directory: a workflow names its files, so a file being
|
||||
// listed where it is not wanted costs nothing.
|
||||
var modelFolders = []string{"checkpoints", "diffusion_models", "text_encoders", "vae", "loras",
|
||||
"clip_vision", "controlnet", "upscale_models", "embeddings", "style_models"}
|
||||
|
||||
// RunGate lays out the replica, starts ComfyUI and serves the gate until
|
||||
// ctx ends or ComfyUI exits.
|
||||
func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
log := cfg.Log
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
state, err := filepath.Abs(cfg.State)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What ComfyUI expects to find in its base directory, and dies
|
||||
// without: custom_nodes is listed before anything else is done.
|
||||
for _, d := range []string{"custom_nodes", "input", "output", "temp", "user"} {
|
||||
if err := os.MkdirAll(filepath.Join(state, d), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
return err
|
||||
}
|
||||
extra := filepath.Join(state, "extra_model_paths.yaml")
|
||||
if err := os.WriteFile(extra, []byte(extraModelPaths(models)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return err
|
||||
}
|
||||
|
||||
inner, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
args := append([]string{filepath.Join(cfg.Dir, "main.py"),
|
||||
"--listen", "127.0.0.1", "--port", strconv.Itoa(inner),
|
||||
"--base-directory", state, "--extra-model-paths-config", extra,
|
||||
"--disable-auto-launch"}, cfg.Args...)
|
||||
child := exec.Command(cfg.Python, args...) //nolint:gosec // the agent's own configuration
|
||||
child.Dir = cfg.Dir
|
||||
child.Env = os.Environ()
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
}
|
||||
// ComfyUI writes to a pipe of the gate's, never to what the gate was
|
||||
// given. That is the agent's, and an agent that restarts and adopts the
|
||||
// replica leaves it with no reader: ComfyUI's next progress bar would
|
||||
// fail with a broken pipe, and take the workflow it belongs to with it.
|
||||
// The gate reads the pipe for as long as ComfyUI writes, and passes on
|
||||
// what it can.
|
||||
signal.Ignore(syscall.SIGPIPE)
|
||||
pr, err := child.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
child.Stderr = child.Stdout
|
||||
if err := child.Start(); err != nil {
|
||||
return fmt.Errorf("starting ComfyUI: %w", err)
|
||||
}
|
||||
relayed := make(chan struct{})
|
||||
go func() { relay(pr, out); close(relayed) }()
|
||||
exited := make(chan error, 1)
|
||||
go func() {
|
||||
// Wait closes the pipe; what is in it is read first.
|
||||
<-relayed
|
||||
exited <- child.Wait()
|
||||
}()
|
||||
|
||||
upstream := &url.URL{Scheme: "http", Host: net.JoinHostPort("127.0.0.1", strconv.Itoa(inner))}
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: NewGate(upstream, cfg.Key, cfg.Model),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
go func() { served <- srv.ListenAndServe() }()
|
||||
log.Info("ComfyUI gate up", "listen", cfg.Listen, "comfyui", upstream.Host, "weights", cfg.Weights)
|
||||
|
||||
stop := func() {
|
||||
_ = child.Process.Signal(syscall.SIGTERM)
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(20 * time.Second):
|
||||
_ = child.Process.Kill()
|
||||
<-exited
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdown(srv)
|
||||
stop()
|
||||
return nil
|
||||
case err := <-exited:
|
||||
shutdown(srv)
|
||||
if err == nil {
|
||||
err = errors.New("exited")
|
||||
}
|
||||
return fmt.Errorf("ComfyUI %w", err)
|
||||
case err := <-served:
|
||||
stop()
|
||||
return fmt.Errorf("the gate stopped serving: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
func shutdown(srv *http.Server) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = srv.Shutdown(ctx)
|
||||
}
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
base(r)
|
||||
r.Header.Del("Authorization")
|
||||
// ComfyUI refuses a request whose Origin is not its own host; the
|
||||
// router sends none, and nothing else gets this far.
|
||||
r.Header.Del("Origin")
|
||||
r.Host = upstream.Host
|
||||
}
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
up := func(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, upstream.String()+"/system_stats", nil)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("ComfyUI answered %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if key != "" {
|
||||
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
|
||||
if !ok || subtle.ConstantTimeCompare([]byte(got), []byte(key)) != 1 {
|
||||
http.Error(w, "this engine wants its key", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
}
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
case "/v1/models":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
return
|
||||
}
|
||||
for _, p := range gatePaths {
|
||||
if r.URL.Path == p || strings.HasSuffix(p, "/") && strings.HasPrefix(r.URL.Path, p) {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// cachedName is a file the agent fetched by itself: its cache names it by
|
||||
// the source's hash, then the name it had.
|
||||
var cachedName = regexp.MustCompile(`^[0-9a-f]{16}-`)
|
||||
|
||||
// FileName is what a weights file is called where ComfyUI sees it: its own
|
||||
// name, without the prefix the agent's cache gives it.
|
||||
func FileName(path string) string {
|
||||
return cachedName.ReplaceAllString(filepath.Base(path), "")
|
||||
}
|
||||
|
||||
// LinkWeights shows ComfyUI the weights: a link in dir to the file, or to
|
||||
// every file of the directory. Links made before are taken away first, so
|
||||
// a replica started again on other weights sees only those.
|
||||
func LinkWeights(weights, dir string) error {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
old, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range old {
|
||||
if e.Type()&os.ModeSymlink != 0 {
|
||||
_ = os.Remove(filepath.Join(dir, e.Name()))
|
||||
}
|
||||
}
|
||||
weights, err = filepath.Abs(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
info, err := os.Stat(weights)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the weights: %w", err)
|
||||
}
|
||||
files := []string{weights}
|
||||
if info.IsDir() {
|
||||
entries, err := os.ReadDir(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files = files[:0]
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && !strings.HasPrefix(e.Name(), ".") {
|
||||
files = append(files, filepath.Join(weights, e.Name()))
|
||||
}
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return fmt.Errorf("the weights directory %s has no files", weights)
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if err := os.Symlink(f, filepath.Join(dir, FileName(f))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extraModelPaths(models string) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "llmgw:\n base_path: %q\n is_default: true\n", models)
|
||||
for _, f := range modelFolders {
|
||||
fmt.Fprintf(&b, " %s: .\n", f)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// relay copies r to w for as long as r has anything, and goes on reading
|
||||
// after w fails: a writer that stops reading must not stop the one writing.
|
||||
func relay(r io.Reader, w io.Writer) {
|
||||
buf := make([]byte, 32<<10)
|
||||
broken := false
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
if n > 0 && !broken {
|
||||
if _, werr := w.Write(buf[:n]); werr != nil {
|
||||
broken = true
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheGateWantsTheKeyAndLetsThroughOnlyTheRoutersRequests(t *testing.T) {
|
||||
var got []string
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = append(got, r.Method+" "+r.URL.Path+" auth="+r.Header.Get("Authorization"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
gate := httptest.NewServer(NewGate(u, "k", "flux"))
|
||||
defer gate.Close()
|
||||
|
||||
do := func(method, path, key string) (int, string) {
|
||||
req, _ := http.NewRequest(method, gate.URL+path, strings.NewReader("{}"))
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", ""); code != http.StatusUnauthorized {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", "wrong"); code != http.StatusUnauthorized {
|
||||
t.Errorf("wrong key: %d", code)
|
||||
}
|
||||
for _, p := range []string{"/upload/image", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusNotFound {
|
||||
t.Errorf("%s: %d, want it refused", p, code)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusOK {
|
||||
t.Errorf("%s: %d", p, code)
|
||||
}
|
||||
}
|
||||
if code, body := do("GET", "/v1/models", "k"); code != 200 || !strings.Contains(body, `"id":"flux"`) {
|
||||
t.Errorf("/v1/models: %d %s", code, body)
|
||||
}
|
||||
if code, _ := do("GET", "/health", "k"); code != 200 {
|
||||
t.Errorf("/health: %d", code)
|
||||
}
|
||||
for _, g := range got {
|
||||
if strings.Contains(g, "auth=Bearer") {
|
||||
t.Errorf("the key reached ComfyUI: %s", g)
|
||||
}
|
||||
}
|
||||
comfy.Close()
|
||||
if code, _ := do("GET", "/health", "k"); code != http.StatusServiceUnavailable {
|
||||
t.Errorf("/health with ComfyUI down: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheWeightsAreShownByTheirOwnNames(t *testing.T) {
|
||||
cache := t.TempDir()
|
||||
one := filepath.Join(cache, "0123456789abcdef-flux1-schnell-fp8.safetensors")
|
||||
_ = os.WriteFile(one, []byte("w"), 0o644)
|
||||
models := filepath.Join(t.TempDir(), "models")
|
||||
if err := LinkWeights(one, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(models, "flux1-schnell-fp8.safetensors")); err != nil || target != one {
|
||||
t.Fatalf("%q %v", target, err)
|
||||
}
|
||||
|
||||
set := t.TempDir()
|
||||
for _, n := range []string{"flux1-dev.safetensors", "ae.safetensors", "t5xxl_fp8.safetensors", ".complete"} {
|
||||
_ = os.WriteFile(filepath.Join(set, n), []byte("w"), 0o644)
|
||||
}
|
||||
if err := LinkWeights(set, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(models)
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
if strings.Join(names, ",") != "ae.safetensors,flux1-dev.safetensors,t5xxl_fp8.safetensors" {
|
||||
t.Errorf("shown %v: the old weights' link should be gone and the marker not shown", names)
|
||||
}
|
||||
if !strings.Contains(extraModelPaths(models), "checkpoints: .") {
|
||||
t.Error(extraModelPaths(models))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMain lets the test binary stand in for ComfyUI: started by the gate
|
||||
// as `python main.py --listen … --port …`, it answers /system_stats and
|
||||
// says what it was given.
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI") == "1" {
|
||||
fakeComfyUI()
|
||||
return
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func fakeComfyUI() {
|
||||
var port, base string
|
||||
for i, a := range os.Args {
|
||||
switch a {
|
||||
case "--port":
|
||||
port = os.Args[i+1]
|
||||
case "--base-directory":
|
||||
base = os.Args[i+1]
|
||||
}
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(base, "args"), []byte(strings.Join(os.Args[1:], " ")+"\nkey="+os.Getenv("LLMGW_ENGINE_KEY")), 0o644)
|
||||
http.HandleFunc("/system_stats", func(w http.ResponseWriter, _ *http.Request) {
|
||||
// A progress bar's worth of output with every answer: a write that
|
||||
// failed would end this process, as it fails ComfyUI's workflow.
|
||||
if _, err := os.Stdout.Write([]byte(strings.Repeat("progress ", 20000) + "\n")); err != nil {
|
||||
os.Exit(4)
|
||||
}
|
||||
_, _ = io.WriteString(w, "{}")
|
||||
})
|
||||
http.HandleFunc("/quit", func(w http.ResponseWriter, _ *http.Request) { os.Exit(3) })
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI_EXIT") == "1" {
|
||||
time.AfterFunc(200*time.Millisecond, func() { os.Exit(3) })
|
||||
}
|
||||
_ = http.ListenAndServe("127.0.0.1:"+port, nil)
|
||||
}
|
||||
|
||||
func TestTheGateRunsComfyUIAndEndsWithIt(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "flux.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
state := t.TempDir()
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Model: "flux", Weights: weights, State: state,
|
||||
Python: os.Args[0], Dir: t.TempDir(), Args: []string{"--disable-smart-memory"}, Output: io.Discard})
|
||||
}()
|
||||
get := func(path string) int {
|
||||
resp, err := http.Get("http://" + listen + path)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for get("/health") != http.StatusOK {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the gate never said ComfyUI was up")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "args"))
|
||||
for _, want := range []string{"main.py --listen 127.0.0.1 --port", "--base-directory " + state,
|
||||
"--extra-model-paths-config " + filepath.Join(state, "extra_model_paths.yaml"), "--disable-smart-memory"} {
|
||||
if !strings.Contains(string(b), want) {
|
||||
t.Errorf("%q not in what ComfyUI was given: %s", want, b)
|
||||
}
|
||||
}
|
||||
if _, err := os.Lstat(filepath.Join(state, "models", "flux.safetensors")); err != nil {
|
||||
t.Error("the weights were not shown to ComfyUI:", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(state, "custom_nodes")); err != nil {
|
||||
t.Error("ComfyUI lists custom_nodes in its base directory first thing, and it is not there:", err)
|
||||
}
|
||||
if get("/quit") != http.StatusNotFound {
|
||||
t.Error("a path the router never asks for was let through")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Errorf("stopped, the gate said %v", err)
|
||||
}
|
||||
case <-time.After(25 * time.Second):
|
||||
t.Fatal("the gate did not stop")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIExitingEndsTheGate(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI_EXIT", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
err := RunGate(t.Context(), GateConfig{Listen: "127.0.0.1:0", Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: io.Discard})
|
||||
if err == nil || !strings.Contains(err.Error(), "exit status 3") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type brokenWriter struct{}
|
||||
|
||||
func (brokenWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") }
|
||||
|
||||
func TestComfyUIKeepsWritingWhenWhatTheGateWritesToIsGone(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: brokenWriter{}})
|
||||
}()
|
||||
ok := 0
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for ok < 20 && time.Now().Before(deadline) {
|
||||
resp, err := http.Get("http://" + listen + "/health")
|
||||
if err == nil {
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
ok++
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Fatalf("the gate ended: %v", err)
|
||||
default:
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if ok < 20 {
|
||||
t.Fatalf("ComfyUI answered %d times; output it could not hand on stopped it", ok)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"8": {"class_type": "FluxGuidance", "inputs": {"guidance": "{{guidance|3.5}}", "conditioning": ["2", 0]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|20}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["8", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|4}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -493,6 +493,9 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
}
|
||||
|
||||
// Control configures the control plane.
|
||||
@@ -723,7 +726,10 @@ type Endpoint struct {
|
||||
Models []string `koanf:"models"`
|
||||
// Engine names the inference engine, which decides what its metrics are
|
||||
// called. Empty means vLLM's names.
|
||||
Engine string `koanf:"engine"`
|
||||
Engine string `koanf:"engine"`
|
||||
// Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
// name or its JSON; flux-schnell unless set.
|
||||
Workflow string `koanf:"workflow"`
|
||||
Pool string `koanf:"pool"`
|
||||
Weight float64 `koanf:"weight"`
|
||||
Priority int `koanf:"priority"`
|
||||
|
||||
@@ -1611,6 +1611,7 @@ type RequestView struct {
|
||||
TTFTMs int64 `json:"ttft_ms"`
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Images int64 `json:"images,omitempty"`
|
||||
CostMicros int64 `json:"cost_micros"`
|
||||
// What the guardrails did, by name.
|
||||
GuardAction string `json:"guard_action,omitempty"`
|
||||
@@ -1644,7 +1645,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
|
||||
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
|
||||
@@ -499,7 +499,11 @@ func (a *API) kbSearch(w http.ResponseWriter, r *http.Request) {
|
||||
// Said, not swallowed: a search that quietly lost its meaning half
|
||||
// looks like a collection that does not know the answer.
|
||||
if a.kb.Embed != nil {
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, in.Query)
|
||||
question := in.Query
|
||||
if m, ok := a.ctl.Catalog().Resolve(c.EmbedModel); ok {
|
||||
question = m.QueryPrefix + question
|
||||
}
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, question)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, c.EmbedModel+" did not embed the question: "+err.Error())
|
||||
return
|
||||
|
||||
@@ -8,12 +8,14 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
@@ -321,6 +323,9 @@ type Model struct {
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
// ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
// pictures: the workflow, and the file it loads.
|
||||
ComfyUI *ComfyUI `json:"comfyui,omitempty" koanf:"comfyui"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it — unless
|
||||
@@ -363,6 +368,13 @@ type Model struct {
|
||||
// Capabilities is what this model can do beyond plain completion.
|
||||
Capabilities Capabilities `json:"capabilities,omitempty" koanf:"capabilities"`
|
||||
|
||||
// QueryPrefix is put before a question an embedding model embeds for a
|
||||
// knowledge-base search; documents are embedded without it. Models
|
||||
// trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
// "Instruct: Given a web search query, retrieve relevant passages that
|
||||
// answer the query\nQuery: " for one.
|
||||
QueryPrefix string `json:"query_prefix,omitempty" koanf:"query_prefix"`
|
||||
|
||||
// Role "guard" is a model that judges the gateway's traffic for the
|
||||
// guardrails rather than serving clients: hidden from /v1/models and the
|
||||
// chat page, refused to any key that names it, critical by default, and
|
||||
@@ -450,6 +462,98 @@ type Capabilities struct {
|
||||
Tools bool `json:"tools,omitempty" koanf:"tools"`
|
||||
Vision bool `json:"vision,omitempty" koanf:"vision"`
|
||||
Embeddings bool `json:"embeddings,omitempty" koanf:"embeddings"`
|
||||
// Images is a model that makes pictures, asked for at
|
||||
// /v1/images/generations. Implied by an external model whose API is
|
||||
// ComfyUI's.
|
||||
Images bool `json:"images,omitempty" koanf:"images"`
|
||||
}
|
||||
|
||||
// MakesImages reports whether a model answers /v1/images/generations.
|
||||
func (m Model) MakesImages() bool {
|
||||
return m.Capabilities.Images || m.ServesComfyUI()
|
||||
}
|
||||
|
||||
// EngineComfyUI is the engine that runs ComfyUI on a node.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// ServesComfyUI reports whether a model is made by ComfyUI, on the
|
||||
// cluster's own nodes or on a server outside it.
|
||||
func (m Model) ServesComfyUI() bool {
|
||||
if m.External != nil {
|
||||
return m.External.API == APIComfyUI
|
||||
}
|
||||
return len(m.Profiles) > 0 && m.Profiles[0].Engine == EngineComfyUI
|
||||
}
|
||||
|
||||
// ComfyUI is a ComfyUI model's workflow and checkpoint.
|
||||
type ComfyUI struct {
|
||||
// Workflow is a built-in's name (flux-schnell, the default, or
|
||||
// flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
// format.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
// source's own name unless set, which a source that is a set of files
|
||||
// must.
|
||||
Checkpoint string `json:"checkpoint,omitempty" koanf:"checkpoint"`
|
||||
}
|
||||
|
||||
// ComfyWorkflow is what a ComfyUI model's replicas are sent: the workflow
|
||||
// as the catalog names it, and the checkpoint it loads.
|
||||
func (m Model) ComfyWorkflow() (workflow, checkpoint string) {
|
||||
if m.External != nil {
|
||||
return m.External.Workflow, m.External.Model
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
workflow, checkpoint = m.ComfyUI.Workflow, m.ComfyUI.Checkpoint
|
||||
}
|
||||
if checkpoint == "" && !strings.HasSuffix(m.Source, "/") {
|
||||
checkpoint = path.Base(strings.TrimPrefix(m.Source, "hf:"))
|
||||
}
|
||||
return workflow, checkpoint
|
||||
}
|
||||
|
||||
// checkWorkflow refuses a workflow that is neither a built-in nor a file.
|
||||
func checkWorkflow(model, w string) error {
|
||||
// A file is read where the router runs; a name is checked here.
|
||||
if w != "" && !strings.ContainsAny(w, "/.") && !slices.Contains(comfy.Builtins(), w) {
|
||||
return fmt.Errorf("model %q: no built-in workflow %q (there are %s; or name a JSON file)",
|
||||
model, w, strings.Join(comfy.Builtins(), ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) validateComfyUI() error {
|
||||
comfyProfiles := 0
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == EngineComfyUI {
|
||||
comfyProfiles++
|
||||
}
|
||||
}
|
||||
if comfyProfiles == 0 {
|
||||
if m.ComfyUI != nil {
|
||||
return fmt.Errorf("model %q: comfyui: is for a model whose profiles run engine comfyui", m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if comfyProfiles != len(m.Profiles) {
|
||||
return fmt.Errorf("model %q: every profile runs engine comfyui or none does: "+
|
||||
"a request for pictures cannot be answered by an engine that writes text", m.Name)
|
||||
}
|
||||
if len(m.Adapters) > 0 {
|
||||
return fmt.Errorf("model %q: ComfyUI serves no adapters", m.Name)
|
||||
}
|
||||
var w string
|
||||
if m.ComfyUI != nil {
|
||||
w = m.ComfyUI.Workflow
|
||||
}
|
||||
if err := checkWorkflow(m.Name, w); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, ckpt := m.ComfyWorkflow(); ckpt == "" {
|
||||
return fmt.Errorf("model %q: its source is a set of files, so comfyui.checkpoint must name the one the workflow loads",
|
||||
m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Catalog is the set of models the cluster should serve.
|
||||
@@ -690,6 +794,9 @@ func (m Model) validate() error {
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if err := m.validateComfyUI(); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
@@ -1094,8 +1201,20 @@ type External struct {
|
||||
// than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
// goes with the peer's own list, and is sent only while the peer is up.
|
||||
Peer string `json:"peer,omitempty" koanf:"peer"`
|
||||
// API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
// a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
// sent the workflow below with the request's prompt in it. Model is then
|
||||
// the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
API string `json:"api,omitempty" koanf:"api"`
|
||||
// Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
// (flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
// plane.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
}
|
||||
|
||||
// APIComfyUI is External.API for a ComfyUI server.
|
||||
const APIComfyUI = "comfyui"
|
||||
|
||||
// IsPeer reports whether a model is offered by a federated cluster.
|
||||
func (m Model) IsPeer() bool { return m.External != nil && m.External.Peer != "" }
|
||||
|
||||
@@ -1110,6 +1229,21 @@ func (e *External) validate(model string, profiles int) error {
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
switch e.API {
|
||||
case "", "openai":
|
||||
if e.Workflow != "" {
|
||||
return fmt.Errorf("model %q: a workflow is for api: comfyui", model)
|
||||
}
|
||||
case APIComfyUI:
|
||||
if e.Peer != "" {
|
||||
return fmt.Errorf("model %q: a federated model speaks OpenAI's API, not ComfyUI's", model)
|
||||
}
|
||||
if err := checkWorkflow(model, e.Workflow); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func comfyModel(source string, c *ComfyUI, profiles ...Profile) Model {
|
||||
if len(profiles) == 0 {
|
||||
profiles = []Profile{{Name: "gpu", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1}}
|
||||
}
|
||||
return Model{Name: "flux", Source: source, ComfyUI: c, Profiles: profiles}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelMakesPicturesFromItsSourcesFile(t *testing.T) {
|
||||
cat, err := New([]Model{comfyModel("hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors", nil)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
if !m.MakesImages() || !m.ServesComfyUI() {
|
||||
t.Error("a ComfyUI model does not say it makes pictures")
|
||||
}
|
||||
if w, ckpt := m.ComfyWorkflow(); w != "" || ckpt != "flux1-schnell-fp8.safetensors" {
|
||||
t.Errorf("workflow %q, checkpoint %q", w, ckpt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsRefusedWhatItCannotDo(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
m Model
|
||||
want string
|
||||
}{
|
||||
"a set without a checkpoint": {comfyModel("https://mirror/flux-dev/v1/", nil), "comfyui.checkpoint"},
|
||||
"an unknown workflow": {comfyModel("/w/f.safetensors", &ComfyUI{Workflow: "sdxl"}), "no built-in workflow"},
|
||||
"mixed engines": {comfyModel("/w/f.safetensors", nil,
|
||||
Profile{Name: "a", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1},
|
||||
Profile{Name: "b", Engine: "vllm", GPUClass: "6000-24g", TP: 1}), "every profile"},
|
||||
"comfyui settings without the engine": {Model{Name: "x", Source: "/w/x", ComfyUI: &ComfyUI{},
|
||||
Profiles: []Profile{{Name: "a", Engine: "vllm", GPUClass: "6000-24g", TP: 1}}}, "engine comfyui"},
|
||||
"no card": {comfyModel("/w/f.safetensors", nil, Profile{Name: "a", Engine: EngineComfyUI, TP: 1}), "gpu_class"},
|
||||
} {
|
||||
if _, err := New([]Model{c.m}); err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: %v", name, err)
|
||||
}
|
||||
}
|
||||
ok := comfyModel("https://mirror/flux-dev/v1/", &ComfyUI{Workflow: "flux-dev", Checkpoint: "flux1-dev-fp8.safetensors"})
|
||||
if _, err := New([]Model{ok}); err != nil {
|
||||
t.Errorf("a set with its checkpoint named was refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -360,7 +361,7 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
names := append([]string{m.Name}, m.Aliases...)
|
||||
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
out = append(out, endpoint.Info{
|
||||
info := endpoint.Info{
|
||||
ID: rep.ID,
|
||||
URL: strings.TrimRight(rep.Endpoint, "/"),
|
||||
Models: names,
|
||||
@@ -369,7 +370,18 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Pool: rep.Pool,
|
||||
Weight: rep.Weight,
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
if prof.Engine == catalog.EngineComfyUI {
|
||||
workflow, ckpt := m.ComfyWorkflow()
|
||||
w, err := c.workflow(workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; its replica is not offered",
|
||||
"model", m.Name, "replica", rep.ID, "err", err)
|
||||
continue
|
||||
}
|
||||
info.Workflow, info.UpstreamModel = w, ckpt
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
@@ -412,23 +424,45 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
if m.IsPeer() {
|
||||
id = "peer:" + m.Name
|
||||
}
|
||||
engine, workflow := endpoint.EngineExternal, ""
|
||||
if m.External.API == catalog.APIComfyUI {
|
||||
w, err := c.workflow(m.External.Workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
engine, workflow = endpoint.EngineComfyUI, w
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: id, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
Engine: engine, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
Peer: m.External.Peer, PeerCluster: cluster, PeerTags: tags,
|
||||
Workflow: workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// workflow is a ComfyUI workflow as the router is sent it: a built-in's
|
||||
// name as it is, a file read here, on the control plane the catalog says it
|
||||
// is on, and sent as the workflow itself.
|
||||
func (c *Controller) workflow(w string) (string, error) {
|
||||
if !strings.ContainsAny(w, "/.") {
|
||||
return w, nil
|
||||
}
|
||||
b, err := os.ReadFile(w)
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// fingerprint summarizes a list so an unchanged one can be skipped.
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)))
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)),
|
||||
sha256.Sum256([]byte(i.Workflow)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -111,7 +111,14 @@ func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []Ad
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
// An on-demand model that is up was asked for, by somebody this
|
||||
// control plane has not heard of: it started before a restart or a
|
||||
// change of leader. Without this it reads as wanted by nobody and is
|
||||
// drained on the next pass — every awake on-demand model, at every
|
||||
// failover. It gets a whole idle timeout from now instead.
|
||||
if _, asked := r.wanted[m.Name]; !asked && m.Scaling.OnDemand() {
|
||||
r.wanted[m.Name] = r.now()
|
||||
}
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
@@ -1409,6 +1409,29 @@ func TestUseKeepsAnOnDemandModelAlive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAwakeOnDemandModelSurvivesARestartOfTheControlPlane(t *testing.T) {
|
||||
// A control plane that has just started, or just become leader, knows
|
||||
// nobody asked for anything. Taking that to mean nobody wants the
|
||||
// on-demand models a node reports running drained every one of them at
|
||||
// every failover, under whoever was using them.
|
||||
h := newHarness(t, onDemandModel(10*time.Minute), gpuNode("1", 8))
|
||||
h.r.Adopt("1", []AdoptedShard{adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(5 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
if reps := h.r.Replicas(); len(reps) != 1 || !reps[0].State.Counts() {
|
||||
t.Fatalf("the adopted replica of an on-demand model was taken down: %+v", reps)
|
||||
}
|
||||
// Still an on-demand model: unused for a whole idle timeout, it goes.
|
||||
h.advance(6 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State.Counts() {
|
||||
t.Errorf("an adopted replica nobody used outlived its idle timeout: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelWithMaxZeroStaysOff(t *testing.T) {
|
||||
// min 0 max 0 is configured and deliberately not running, which is a
|
||||
// different and equally useful thing from "load it when wanted".
|
||||
|
||||
@@ -64,6 +64,15 @@ var Reference = []Block{
|
||||
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ComfyUI", Title: "ComfyUI", Doc: "", Fields: []Field{
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is a built-in's name (flux-schnell, the default, or\nflux-dev) or a JSON file on the control plane, in ComfyUI's API\nformat.", Nested: ""},
|
||||
{Key: "checkpoint", Type: "string", Doc: "Checkpoint is the file the workflow's {{checkpoint}} names: the\nsource's own name unless set, which a source that is a set of files\nmust.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "ComfyUIConfig", Title: "ComfyUIConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is ComfyUI's directory, where main.py is. Readable by the user\nengines run as.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
|
||||
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
|
||||
@@ -84,6 +93,7 @@ var Reference = []Block{
|
||||
{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
|
||||
{Key: "models", Type: "[]string", Doc: "", Nested: ""},
|
||||
{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow for engine: comfyui, a built-in's\nname or its JSON; flux-schnell unless set.", Nested: ""},
|
||||
{Key: "pool", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
@@ -104,6 +114,8 @@ var Reference = []Block{
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
@@ -171,6 +183,7 @@ var Reference = []Block{
|
||||
{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
|
||||
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
|
||||
{Key: "comfyui", Type: "*ComfyUI", Doc: "ComfyUI is how a model whose profiles run engine comfyui makes its\npictures: the workflow, and the file it loads.", Nested: "catalog.ComfyUI"},
|
||||
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it — unless\nMaxLoadedAdapters is set.", Nested: ""},
|
||||
{Key: "max_loaded_adapters", Type: "int", Doc: "MaxLoadedAdapters, above zero, has each replica hold at most this many\nadapters at once, loaded when a request first asks for one and unloaded\nleast recently used first to make room. The router sends a request for\nan adapter to a replica that already holds it while that one has room\nto take it. vLLM profiles only: llama-server loads adapters at start\nand cannot load one afterwards. Zero loads every adapter on every\nreplica at start.", Nested: ""},
|
||||
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
|
||||
@@ -181,6 +194,7 @@ var Reference = []Block{
|
||||
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
|
||||
{Key: "max_output", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
|
||||
{Key: "query_prefix", Type: "string", Doc: "QueryPrefix is put before a question an embedding model embeds for a\nknowledge-base search; documents are embedded without it. Models\ntrained with an instruction on the query side want it, Qwen3-Embedding\n\"Instruct: Given a web search query, retrieve relevant passages that\nanswer the query\\nQuery: \" for one.", Nested: ""},
|
||||
{Key: "role", Type: "string", Doc: "Role \"guard\" is a model that judges the gateway's traffic for the\nguardrails rather than serving clients: hidden from /v1/models and the\nchat page, refused to any key that names it, critical by default, and\nalways running. Guard says how its answers are read.", Nested: ""},
|
||||
{Key: "guard", Type: "*GuardModel", Doc: "", Nested: "catalog.GuardModel"},
|
||||
{Key: "origin", Type: "*Origin", Doc: "Origin is the registry repository the weights came from, at which\ncommit, and each file's checksum there. Written when a model is served\nfrom the registry; for one that was not, worked out from the mirror's\nrecord of where it fetched the file. It is what lets the control plane\nsay that the repository has published different weights since, and\nmove the model to them.", Nested: "catalog.Origin"},
|
||||
@@ -444,6 +458,7 @@ var Reference = []Block{
|
||||
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
@@ -488,6 +503,7 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
|
||||
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
|
||||
@@ -496,6 +512,7 @@ var Reference = []Block{
|
||||
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
|
||||
{Key: "trtllm", Type: "TRTLLMConfig", Doc: "TRTLLM is TensorRT-LLM, which the agent does not install: a node\noffers it only when its interpreter is named here.", Nested: "engines.TRTLLMConfig"},
|
||||
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
|
||||
{Key: "comfyui", Type: "ComfyUIConfig", Doc: "ComfyUI makes pictures; offered by a node whose ComfyUI is named here.", Nested: "engines.ComfyUIConfig"},
|
||||
}},
|
||||
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
|
||||
|
||||
@@ -22,6 +22,7 @@ var Sources = []string{
|
||||
"internal/agent/engines/sglang.go",
|
||||
"internal/agent/engines/trtllm.go",
|
||||
"internal/agent/engines/sim.go",
|
||||
"internal/agent/engines/comfyui.go",
|
||||
"internal/agent/supervisor/supervisor.go",
|
||||
"internal/tlsconf/tlsconf.go",
|
||||
"internal/control/catalog/catalog.go",
|
||||
|
||||
@@ -42,7 +42,7 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
|
||||
// answer than saying the prompt is too large, and it would be done
|
||||
// silently, to a request that was already being quietly moved.
|
||||
canonical, cand := r.resolve(name)
|
||||
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
|
||||
if why := checkContextFits(cand, req.contextTokens(), req.MaxTokens); why != "" {
|
||||
continue
|
||||
}
|
||||
res, err := r.acct.Check(p, canonical, req.estimatedTokens())
|
||||
|
||||
@@ -56,6 +56,7 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
func(w http.ResponseWriter, req *http.Request, _ store.User) {
|
||||
web.ReplyRenderer(r.cfg.MaxBodyBytes)(w, req)
|
||||
}))
|
||||
r.registerImagesPage(mux, pages)
|
||||
}
|
||||
|
||||
// chatSend dispatches one turn on behalf of a signed-in person.
|
||||
@@ -256,6 +257,12 @@ func (r *Router) principalFor(req *http.Request, user store.User, asked string)
|
||||
// by name, "big-model" came first and every visitor who pressed Send without
|
||||
// touching the picker got placeholder text.
|
||||
func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
return r.modelsOf(req, user, false)
|
||||
}
|
||||
|
||||
// modelsOf is the models a person may use from a page: those that make
|
||||
// pictures, or those that write text.
|
||||
func (r *Router) modelsOf(req *http.Request, user store.User, pictures bool) []web.ChatModel {
|
||||
principal, err := r.principalFor(req, user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -283,7 +290,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
@@ -299,7 +306,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{
|
||||
Name: name, Simulated: entry.Simulated(), OnDemand: true,
|
||||
})
|
||||
|
||||
@@ -48,12 +48,23 @@ type Info struct {
|
||||
Peer string
|
||||
PeerCluster string
|
||||
PeerTags bool
|
||||
// Workflow is the ComfyUI workflow sent for an endpoint whose engine is
|
||||
// EngineComfyUI: a built-in's name or the workflow's JSON.
|
||||
Workflow string
|
||||
}
|
||||
|
||||
// EngineExternal marks an endpoint that is a provider outside the cluster.
|
||||
// It publishes no metrics, so it is never scraped and never reads as stale.
|
||||
const EngineExternal = "external"
|
||||
|
||||
// EngineComfyUI marks a ComfyUI server. The router speaks ComfyUI's API to
|
||||
// it, not OpenAI's, and it publishes no metrics either.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// Scraped reports whether an engine publishes metrics the router reads. One
|
||||
// that does not is never read as stale for having none.
|
||||
func Scraped(engine string) bool { return engine != EngineExternal && engine != EngineComfyUI }
|
||||
|
||||
// Serves reports whether the replica serves the given model.
|
||||
func (i Info) Serves(model string) bool {
|
||||
return slices.Contains(i.Models, model)
|
||||
@@ -257,7 +268,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
||||
continue
|
||||
}
|
||||
var age time.Duration
|
||||
stale := l.info.Engine != EngineExternal
|
||||
stale := Scraped(l.info.Engine)
|
||||
if !l.scrapedAt.IsZero() {
|
||||
age = now.Sub(l.scrapedAt)
|
||||
stale = age > r.cfg.StaleAfter
|
||||
|
||||
@@ -102,6 +102,8 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/score", r.handleInference("/v1/score"))
|
||||
mux.HandleFunc("POST /v1/audio/transcriptions", r.handleInference("/v1/audio/transcriptions"))
|
||||
mux.HandleFunc("POST /v1/audio/translations", r.handleInference("/v1/audio/translations"))
|
||||
// Pictures, made by ComfyUI from a workflow the catalog names.
|
||||
mux.HandleFunc("POST "+imagesPath, r.handleInference(imagesPath))
|
||||
// Answered by the guardrails' judge model.
|
||||
mux.HandleFunc("POST /v1/moderations", r.handleModerations)
|
||||
// Codex, and whatever else speaks only the Responses API, translated to
|
||||
@@ -319,7 +321,7 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
exempt := []string{"/health", "/ready", "/leader", "/", "/about", "/_static/", "/_lang/", "/claim/",
|
||||
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/"}
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/org", "/org/", "/setup/"}
|
||||
// The docs are public unless the site said otherwise, and then guarded by
|
||||
// the session check in front of them — or, on a gateway with keys and no
|
||||
// accounts to sign in to, by the key like any other path.
|
||||
@@ -391,7 +393,11 @@ type parsedRequest struct {
|
||||
ProfileOverride string
|
||||
PromptPrefix string
|
||||
PromptTokens int
|
||||
MaxTokens int
|
||||
// ContextTokens is what has to fit in one context: the prompt, except
|
||||
// for embeddings, where each input is embedded on its own and the
|
||||
// longest one is what a context must hold.
|
||||
ContextTokens int
|
||||
MaxTokens int
|
||||
|
||||
// ThreadID is the conversation this request continues, empty for the
|
||||
// stateless path every request took before threads existed.
|
||||
@@ -473,11 +479,19 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// A picture asked of a model that writes text, or the other way
|
||||
// round, is refused before it waits for either.
|
||||
if why := r.imageRouteRefusal(path, parsed.Model, parsed.UpstreamBody); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
|
||||
// Before authentication work and before admission: a request that
|
||||
// cannot fit should not consume a quota reservation or a queue slot on
|
||||
// its way to being refused.
|
||||
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
|
||||
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
|
||||
if why := checkContextFits(entry, parsed.contextTokens(), parsed.MaxTokens); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
@@ -736,6 +750,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
|
||||
// the truth than charging nothing.
|
||||
u.PromptTokens = int64(req.PromptTokens)
|
||||
}
|
||||
u.Images = int64(res.outcome.Images)
|
||||
r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
|
||||
r.acct.Record(req.reservation, u)
|
||||
}
|
||||
@@ -825,6 +840,12 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
|
||||
// plan §14). Measured over the whole prompt, not the routing prefix —
|
||||
// see promptBytes.
|
||||
p.PromptTokens = promptBytes(&b) / 4
|
||||
p.ContextTokens = p.PromptTokens
|
||||
if path == "/v1/embeddings" {
|
||||
if longest, ok := longestInput(b.Input); ok {
|
||||
p.ContextTokens = longest / 4
|
||||
}
|
||||
}
|
||||
|
||||
// Tags are the gateway's: an engine has no use for them, and its logs
|
||||
// are no place for what a client wrote there.
|
||||
@@ -897,6 +918,29 @@ func promptPrefix(b *inferenceBody) string {
|
||||
// bounded head of the conversation; a token estimate wants all of it, and using
|
||||
// one for the other meant TPM limits undercounted exactly the requests that
|
||||
// consume the most, and nothing could tell whether a prompt fit in a context.
|
||||
// longestInput is the length of an embeddings request's longest input, when
|
||||
// input is a list of strings: a batch of texts, each its own context.
|
||||
func longestInput(raw json.RawMessage) (int, bool) {
|
||||
var list []string
|
||||
if len(raw) == 0 || raw[0] != '[' || json.Unmarshal(raw, &list) != nil || len(list) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
n := 0
|
||||
for _, s := range list {
|
||||
n = max(n, len(s))
|
||||
}
|
||||
return n, true
|
||||
}
|
||||
|
||||
// contextTokens is what has to fit in one context; requests built inside
|
||||
// the router set only PromptTokens.
|
||||
func (p *parsedRequest) contextTokens() int {
|
||||
if p.ContextTokens > 0 {
|
||||
return p.ContextTokens
|
||||
}
|
||||
return p.PromptTokens
|
||||
}
|
||||
|
||||
func promptBytes(b *inferenceBody) int {
|
||||
n := 0
|
||||
add := func(raw json.RawMessage) {
|
||||
@@ -1088,7 +1132,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
entry["capabilities"] = map[string]bool{
|
||||
"tools": m.Capabilities.Tools, "vision": m.Capabilities.Vision,
|
||||
"embeddings": m.Capabilities.Embeddings,
|
||||
"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(),
|
||||
}
|
||||
data = append(data, entry)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// /v1/images/generations, OpenAI's shape, made by ComfyUI: the request goes
|
||||
// through the same checks as any other — key, allowlist, limits, admission,
|
||||
// guardrails on its prompt — and only at the replica is it turned into a
|
||||
// workflow, queued and waited on, and its pictures sent back as base64.
|
||||
|
||||
const imagesPath = "/v1/images/generations"
|
||||
|
||||
// imageRequest is what a request for pictures asks. Beyond OpenAI's fields,
|
||||
// those a diffusion model has and DALL·E does not: seed, steps,
|
||||
// negative_prompt and guidance.
|
||||
type imageRequest struct {
|
||||
Prompt string `json:"prompt"`
|
||||
N int `json:"n"`
|
||||
Size string `json:"size"`
|
||||
ResponseFormat string `json:"response_format"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
Guidance *float64 `json:"guidance"`
|
||||
|
||||
width, height int
|
||||
}
|
||||
|
||||
// The bounds of a request. A FLUX latent is 1/16 of the picture, so a side
|
||||
// that is not a multiple of 16 is not one it can make; the rest keep one
|
||||
// request from holding a card for the length of a film.
|
||||
const (
|
||||
maxImages = 4
|
||||
minSide = 256
|
||||
maxSide = 2048
|
||||
maxSteps = 100
|
||||
defaultSide = 1024
|
||||
)
|
||||
|
||||
func parseImageRequest(body []byte) (imageRequest, error) {
|
||||
var in imageRequest
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
return in, err
|
||||
}
|
||||
if strings.TrimSpace(in.Prompt) == "" {
|
||||
return in, errors.New("prompt is required")
|
||||
}
|
||||
switch {
|
||||
case in.N == 0:
|
||||
in.N = 1
|
||||
case in.N < 0 || in.N > maxImages:
|
||||
return in, fmt.Errorf("n is between 1 and %d", maxImages)
|
||||
}
|
||||
switch in.ResponseFormat {
|
||||
case "", "b64_json":
|
||||
case "url":
|
||||
// A URL would have to be served by somebody and kept for some time,
|
||||
// and a picture made from a prompt is the prompt's owner's: it is
|
||||
// not kept here.
|
||||
return in, errors.New(`response_format "url" is not offered: the picture comes back as b64_json and is kept nowhere`)
|
||||
default:
|
||||
return in, fmt.Errorf("response_format %q is not b64_json", in.ResponseFormat)
|
||||
}
|
||||
in.width, in.height = defaultSide, defaultSide
|
||||
if s := strings.TrimSpace(in.Size); s != "" && s != "auto" {
|
||||
w, h, ok := strings.Cut(strings.ToLower(s), "x")
|
||||
wi, err1 := strconv.Atoi(w)
|
||||
hi, err2 := strconv.Atoi(h)
|
||||
if !ok || err1 != nil || err2 != nil {
|
||||
return in, fmt.Errorf("size %q is not WIDTHxHEIGHT", s)
|
||||
}
|
||||
in.width, in.height = wi, hi
|
||||
}
|
||||
for _, side := range []int{in.width, in.height} {
|
||||
if side < minSide || side > maxSide || side%16 != 0 {
|
||||
return in, fmt.Errorf("size %dx%d: each side is a multiple of 16 from %d to %d", in.width, in.height, minSide, maxSide)
|
||||
}
|
||||
}
|
||||
if in.Steps < 0 || in.Steps > maxSteps {
|
||||
return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
|
||||
}
|
||||
if in.Guidance != nil && (*in.Guidance <= 0 || *in.Guidance > 30) {
|
||||
return in, errors.New("guidance is above 0 and at most 30")
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// imageRouteRefusal says why a request cannot be answered at the path it was
|
||||
// sent to, before it takes a place in a queue: a picture asked of a model
|
||||
// that writes text, or text of one that makes pictures. Empty when it can,
|
||||
// and for a model the catalog does not know, which a hand-configured
|
||||
// endpoint may serve as it likes.
|
||||
func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
m, known := r.modelMap()[model]
|
||||
if path == imagesPath {
|
||||
if known && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s does not make pictures", m.Name)
|
||||
}
|
||||
if _, err := parseImageRequest(body); err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if known && m.ServesComfyUI() {
|
||||
return fmt.Sprintf("%s makes pictures: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// generateImages answers a request at a ComfyUI replica. It returns what the
|
||||
// proxy would: a failure of this replica that another may not have is
|
||||
// retryable, and nothing is written until the pictures are all in hand.
|
||||
func (r *Router) generateImages(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
if req.Path != imagesPath {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s only", ep.ID, imagesPath)
|
||||
}
|
||||
in, err := parseImageRequest(body)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
var upstream struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &upstream)
|
||||
tpl, err := comfy.Load(ep.Workflow)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
seed := comfy.RandomSeed()
|
||||
if in.Seed != nil {
|
||||
seed = *in.Seed
|
||||
}
|
||||
p := comfy.Params{
|
||||
Checkpoint: upstream.Model, Prompt: in.Prompt, NegativePrompt: in.NegativePrompt,
|
||||
Width: in.width, Height: in.height, N: in.N, Seed: seed, Steps: in.Steps,
|
||||
}
|
||||
if in.Guidance != nil {
|
||||
p.Guidance = *in.Guidance
|
||||
}
|
||||
workflow, err := tpl.Fill(p)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
|
||||
if t := r.cfg.Timeouts.Total; t > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, t)
|
||||
defer cancel()
|
||||
}
|
||||
start := time.Now()
|
||||
images, err := r.comfy.Generate(ctx, ep.URL, ep.APIKey, workflow)
|
||||
if err != nil {
|
||||
var ce *comfy.Error
|
||||
switch {
|
||||
case errors.As(err, &ce):
|
||||
return proxy.Outcome{Retryable: ce.Retryable}, err
|
||||
case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
|
||||
return proxy.Outcome{}, proxy.ErrClientGone
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return proxy.Outcome{}, fmt.Errorf("%s made no picture within %s", ep.ID, r.cfg.Timeouts.Total)
|
||||
}
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
|
||||
data := make([]map[string]any, 0, len(images))
|
||||
for _, im := range images {
|
||||
data = append(data, map[string]any{"b64_json": base64.StdEncoding.EncodeToString(im.Data)})
|
||||
}
|
||||
out, err := json.Marshal(map[string]any{"created": time.Now().Unix(), "data": data})
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
// The seed that made them, so a picture liked can be made again.
|
||||
w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(out)
|
||||
took := time.Since(start)
|
||||
return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(out)), TTFT: took, Decode: took,
|
||||
Images: len(images)}, nil
|
||||
}
|
||||
|
||||
// ctxCauseIsClient reports whether a context ended because the client went
|
||||
// away, rather than because a deadline set here passed.
|
||||
func ctxCauseIsClient(ctx context.Context) bool {
|
||||
return !errors.Is(context.Cause(ctx), context.DeadlineExceeded)
|
||||
}
|
||||
|
||||
// registerImagesPage mounts the page that makes pictures, beside the chat
|
||||
// and for the same reasons: a model tried without a client, with the
|
||||
// browser holding a session and never a key, and the request sent down the
|
||||
// one path every API request takes.
|
||||
func (r *Router) registerImagesPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetImages(func() bool {
|
||||
for _, m := range r.modelMap() {
|
||||
if m.MakesImages() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
mux.HandleFunc("GET /images", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
contexts, _ := r.chatContexts(req.Context(), user)
|
||||
current, err := r.pickContext(req.Context(), user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
current = ""
|
||||
}
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Images(user, r.modelsOf(req, user, true), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /images/generate", pages.Guard(r.imagesSend))
|
||||
}
|
||||
|
||||
// imagesSend makes pictures on behalf of a signed-in person.
|
||||
func (r *Router) imagesSend(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
http.Error(w, "could not read the request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
Prompt string `json:"prompt"`
|
||||
Size string `json:"size"`
|
||||
N int `json:"n"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
// Context is what the pictures are charged to, checked as the
|
||||
// chat's is.
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Model == "" {
|
||||
http.Error(w, "a model and a prompt are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
principal, err := r.principalFor(req, user, in.Context)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
// Rebuilt from the fields the page has, so it can set nothing else.
|
||||
out := map[string]any{"model": in.Model, "prompt": in.Prompt, "size": in.Size, "n": in.N}
|
||||
if in.Seed != nil {
|
||||
out["seed"] = *in.Seed
|
||||
}
|
||||
if in.Steps > 0 {
|
||||
out["steps"] = in.Steps
|
||||
}
|
||||
if s := strings.TrimSpace(in.NegativePrompt); s != "" {
|
||||
out["negative_prompt"] = s
|
||||
}
|
||||
upstream, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
http.Error(w, "could not build the request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
next := req.Clone(auth.WithPrincipal(req.Context(), principal))
|
||||
next.Body = io.NopCloser(bytes.NewReader(upstream))
|
||||
next.ContentLength = int64(len(upstream))
|
||||
next.Header.Set("Content-Type", "application/json")
|
||||
next.Header.Del(knowledgeHeader)
|
||||
r.handleInference(imagesPath)(w, next)
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// imageRouter has a ComfyUI model, flux, beside a text model, and returns
|
||||
// what the ComfyUI server was sent.
|
||||
func imageRouter(t *testing.T) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
return imageRouterWith(t, nil)
|
||||
}
|
||||
|
||||
func imageRouterWith(t *testing.T, edit func(*config.Server)) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
var mu sync.Mutex
|
||||
var queued map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/metrics":
|
||||
t.Error("ComfyUI was scraped for metrics")
|
||||
case "/prompt":
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
mu.Lock()
|
||||
queued = b.Prompt
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p1"}`)
|
||||
case "/history/p1":
|
||||
mu.Lock()
|
||||
n := 1
|
||||
if latent, ok := queued["4"].(map[string]any); ok {
|
||||
n = int(latent["inputs"].(map[string]any)["batch_size"].(float64))
|
||||
}
|
||||
mu.Unlock()
|
||||
var images []string
|
||||
for i := range n {
|
||||
images = append(images, fmt.Sprintf(`{"filename":"%d.png","type":"temp"}`, i))
|
||||
}
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[%s]}}}}`,
|
||||
strings.Join(images, ","))
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png-bytes")
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
|
||||
External: &catalog.External{URL: srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
|
||||
if edit != nil {
|
||||
edit(&cfg)
|
||||
}
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:flux", URL: srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
return rt, &queued, &mu
|
||||
}
|
||||
|
||||
func postImages(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestAPictureIsMadeByComfyUIAndComesBackAsBase64(t *testing.T) {
|
||||
rt, queued, mu := imageRouter(t)
|
||||
rec := postImages(t, rt.Handler(), imagesPath, map[string]any{
|
||||
"model": "flux", "prompt": "a lighthouse at dusk", "size": "768x512", "seed": 7})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var out struct {
|
||||
Data []struct {
|
||||
B64 string `json:"b64_json"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil || len(out.Data) != 1 {
|
||||
t.Fatalf("body %s: %v", rec.Body.String(), err)
|
||||
}
|
||||
if b, _ := base64.StdEncoding.DecodeString(out.Data[0].B64); string(b) != "png-bytes" {
|
||||
t.Errorf("picture = %q", b)
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "7" {
|
||||
t.Errorf("seed header = %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
wf, _ := json.Marshal(*queued)
|
||||
for _, want := range []string{`"ckpt_name":"flux1-schnell-fp8.safetensors"`, `"text":"a lighthouse at dusk"`,
|
||||
`"width":768`, `"height":512`, `"seed":7`} {
|
||||
if !strings.Contains(string(wf), want) {
|
||||
t.Errorf("%s not in the workflow sent: %s", want, wf)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAndTextAreAskedForAtTheirOwnPaths(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
h := rt.Handler()
|
||||
for name, c := range map[string]struct {
|
||||
path string
|
||||
body map[string]any
|
||||
want string
|
||||
}{
|
||||
"text of flux": {"/v1/chat/completions", map[string]any{"model": "flux", "messages": []any{}}, "ask for them at " + imagesPath},
|
||||
"picture of chat": {imagesPath, map[string]any{"model": "chat", "prompt": "x"}, "does not make pictures"},
|
||||
"no prompt": {imagesPath, map[string]any{"model": "flux"}, "prompt is required"},
|
||||
"odd size": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "size": "1000x1000"}, "multiple of 16"},
|
||||
"url wanted": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "response_format": "url"}, "b64_json"},
|
||||
"too many at once": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "n": 9}, "n is between"},
|
||||
} {
|
||||
rec := postImages(t, h, c.path, c.body)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), c.want) {
|
||||
t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelSaysItMakesPictures(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if !strings.Contains(rec.Body.String(), `"images":true`) {
|
||||
t.Errorf("/v1/models: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsCheckedInTheCatalog(t *testing.T) {
|
||||
ok := catalog.Model{Name: "f", Source: "external", External: &catalog.External{
|
||||
URL: "http://10.0.0.147:8188", Model: "flux1-dev.safetensors", API: catalog.APIComfyUI, Workflow: "flux-dev"}}
|
||||
if _, err := catalog.New([]catalog.Model{ok}); err != nil {
|
||||
t.Fatalf("refused: %v", err)
|
||||
}
|
||||
for name, e := range map[string]catalog.External{
|
||||
"unknown api": {URL: "http://a:1", Model: "m", API: "a1111"},
|
||||
"unknown workflow": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Workflow: "sdxl"},
|
||||
"workflow, openai": {URL: "http://a:1", Model: "m", Workflow: "flux-dev"},
|
||||
"comfyui from peer": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Peer: "other"},
|
||||
} {
|
||||
e := e
|
||||
if _, err := catalog.New([]catalog.Model{{Name: "f", Source: "external", External: &e}}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAreChargedByThePictureAndRecorded(t *testing.T) {
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"flux": {PromptPer1k: 1000, PerImage: 50000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-1", Hash: sum[:], Prefix: "bbbb", Name: "t",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
b, _ := json.Marshal(map[string]any{"model": "flux", "prompt": "three posters", "n": 3})
|
||||
req := httptest.NewRequest(http.MethodPost, imagesPath, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
id := rec.Header().Get("X-Request-Id")
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, err := rt.store.UsageByRequest(t.Context(), id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok {
|
||||
if u.Images != 3 {
|
||||
t.Errorf("recorded %d pictures, want 3", u.Images)
|
||||
}
|
||||
if want := (u.PromptTokens*1000+999)/1000 + 3*50000; u.CostMicros != want {
|
||||
t.Errorf("charged %d micros, want %d (%d prompt tokens and three pictures)", u.CostMicros, want, u.PromptTokens)
|
||||
}
|
||||
return
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("no usage was recorded")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThePicturesPageOffersPictureModelsAndTheChatTheOthers(t *testing.T) {
|
||||
e := newFakeEngine(t, "chat")
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
})
|
||||
rt.SetEndpoints(t.Context(), append(rt.reg.Infos(), endpoint.Info{ID: "c", URL: e.srv.URL, Models: []string{"chat"}}))
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
user := store.User{ID: "usr-1", ProjectID: project}
|
||||
names := func(pictures bool) []string {
|
||||
var out []string
|
||||
for _, m := range rt.modelsOf(httptest.NewRequest("GET", "/", nil), user, pictures) {
|
||||
out = append(out, m.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := names(false); len(got) != 1 || got[0] != "chat" {
|
||||
t.Errorf("the chat offers %v", got)
|
||||
}
|
||||
if got := names(true); len(got) != 1 || got[0] != "flux" {
|
||||
t.Errorf("the pictures page offers %v", got)
|
||||
}
|
||||
|
||||
// The page's request is rebuilt: a field it has no business sending
|
||||
// does not reach the model.
|
||||
body := `{"model":"flux","prompt":"a kite","size":"512x512","n":1,"seed":3,"response_format":"url"}`
|
||||
rec := httptest.NewRecorder()
|
||||
rt.imagesSend(rec, httptest.NewRequest("POST", "/images/generate", strings.NewReader(body)), user)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "b64_json") {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "3" {
|
||||
t.Errorf("seed %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
}
|
||||
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
|
||||
h := rt.Handler()
|
||||
// The /admin pages are mounted only with a cluster view, which this
|
||||
// router has none of; these are the session pages it does mount.
|
||||
for _, path := range []string{"/profile", "/chat", "/org", "/org/org-x"} {
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/org", "/org/org-x"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if strings.Contains(w.Body.String(), "api key") {
|
||||
|
||||
@@ -224,6 +224,8 @@ type Outcome struct {
|
||||
Decode time.Duration
|
||||
// Usage is the engine-reported token usage, if it was present.
|
||||
Usage *Usage
|
||||
// Images is how many pictures were made, by an endpoint that makes them.
|
||||
Images int
|
||||
// Bytes is how many bytes were relayed to the client.
|
||||
Bytes int64
|
||||
// Reply is the assistant's text, when CaptureReply asked for it. Truncated
|
||||
|
||||
@@ -276,6 +276,10 @@ func splitList(v string) []string {
|
||||
// ordinary path, so the caller's limits, budget and guardrails apply to it
|
||||
// and it is counted to them.
|
||||
func (r *Router) embedFor(ctx context.Context, p auth.Principal, model, text string) ([]float32, error) {
|
||||
// A question, not a document: what the model wants before a query.
|
||||
if _, m := r.resolve(model); m.QueryPrefix != "" {
|
||||
text = m.QueryPrefix + text
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"model": model, "input": []string{text}})
|
||||
hreq, err := http.NewRequestWithContext(auth.WithPrincipal(ctx, p), http.MethodPost, "/v1/embeddings", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kb/kbsync"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -253,3 +255,25 @@ func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
|
||||
t.Errorf("a salt without passages:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuestionIsEmbeddedWithTheModelsQueryPrefix(t *testing.T) {
|
||||
// Qwen3-Embedding and its like are trained with an instruction before
|
||||
// the query, never before the documents: the documents come through
|
||||
// /v1/embeddings as they are; the question gets the prefix here.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.Capabilities.Embeddings = true
|
||||
m.QueryPrefix = "Instruct: retrieve passages\nQuery: "
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
// The test engine answers with the input's length in bytes and words.
|
||||
v, err := rt.embedFor(t.Context(), auth.Principal{}, "emb", "kaç gün izin var")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "Instruct: retrieve passages\nQuery: kaç gün izin var"
|
||||
if int(v[0]) != len(want) || int(v[1]) != len(strings.Fields(want)) {
|
||||
t.Errorf("embedded %v, not %q", v, want)
|
||||
}
|
||||
}
|
||||
@@ -29,6 +29,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -53,7 +54,9 @@ type Router struct {
|
||||
reg *endpoint.Registry
|
||||
poller *scrape.Poller
|
||||
px *proxy.Proxy
|
||||
m *metrics
|
||||
// comfy asks ComfyUI replicas for pictures.
|
||||
comfy *comfy.Client
|
||||
m *metrics
|
||||
// adm is nil when admission control is disabled.
|
||||
adm *admission.Controller
|
||||
|
||||
@@ -311,6 +314,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
kv: kvindex.New(),
|
||||
kvClient: newKVClient(),
|
||||
m: metrics,
|
||||
comfy: &comfy.Client{HTTP: &http.Client{Transport: &http.Transport{DialContext: (&net.Dialer{Timeout: cfg.Router.Timeouts.Dial}).DialContext}}},
|
||||
px: proxy.New(proxy.Config{
|
||||
DialTimeout: cfg.Router.Timeouts.Dial,
|
||||
ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader,
|
||||
@@ -370,7 +374,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
prices := make(map[string]accounting.Price, len(cfg.Auth.Pricing))
|
||||
for name, p := range cfg.Auth.Pricing {
|
||||
prices[name] = accounting.Price{
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k,
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, PerImage: p.PerImage,
|
||||
}
|
||||
}
|
||||
r.recorder = accounting.NewRecorder(st, r.auth, accounting.RecorderConfig{
|
||||
@@ -386,6 +390,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
accounting.NewPricing(prices, accounting.Price{
|
||||
PromptPer1k: cfg.Auth.DefaultPrice.PromptPer1k,
|
||||
CompletionPer1k: cfg.Auth.DefaultPrice.CompletionPer1k,
|
||||
PerImage: cfg.Auth.DefaultPrice.PerImage,
|
||||
}),
|
||||
r.recorder, log)
|
||||
r.threads = r.store
|
||||
@@ -487,6 +492,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
|
||||
Pool: e.Pool,
|
||||
Weight: e.Weight,
|
||||
Priority: e.Priority,
|
||||
Workflow: e.Workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -1162,7 +1168,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
}
|
||||
// A filter per attempt, for the model this attempt is served by.
|
||||
filter := r.outputFilter(uctx, req, model, &preq)
|
||||
out, perr := r.px.Do(uctx, w, preq)
|
||||
var out proxy.Outcome
|
||||
var perr error
|
||||
if dec.Endpoint.Engine == endpoint.EngineComfyUI {
|
||||
out, perr = r.generateImages(uctx, w, req, dec.Endpoint.Info, body)
|
||||
} else {
|
||||
out, perr = r.px.Do(uctx, w, preq)
|
||||
}
|
||||
releaseAdapter()
|
||||
uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
|
||||
endSpan(uspan, perr)
|
||||
|
||||
@@ -783,3 +783,28 @@ func TestAStoppedServiceAnswers503AndStillListsModels(t *testing.T) {
|
||||
t.Errorf("started again, still refused: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestABatchOfEmbeddingsIsCheckedTextByText(t *testing.T) {
|
||||
// llmgw-kb sends many passages in one request; each is embedded on its
|
||||
// own, so their sum over the context is no reason to refuse them. One
|
||||
// passage over the context still is.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.ContextLength = 2048
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
send := func(input []string) *httptest.ResponseRecorder {
|
||||
b, _ := json.Marshal(map[string]any{"model": "emb", "input": input})
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/embeddings", bytes.NewReader(b)))
|
||||
return rec
|
||||
}
|
||||
passage := strings.Repeat("x", 6000) // about 1500 tokens
|
||||
if rec := send([]string{passage, passage, passage, passage}); rec.Code == http.StatusBadRequest {
|
||||
t.Errorf("four passages that each fit were refused: %s", rec.Body.String())
|
||||
}
|
||||
if rec := send([]string{"short", strings.Repeat("x", 40_000)}); rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("a passage over the context was let through: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -96,8 +96,8 @@ func (p *Poller) Sync(ctx context.Context) {
|
||||
|
||||
want := make(map[string]endpoint.Info, len(infos))
|
||||
for _, in := range infos {
|
||||
if in.Engine == endpoint.EngineExternal {
|
||||
continue // a provider elsewhere publishes no /metrics
|
||||
if !endpoint.Scraped(in.Engine) {
|
||||
continue // a provider elsewhere, or ComfyUI, publishes no /metrics
|
||||
}
|
||||
want[in.ID] = in
|
||||
}
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many pictures a request made, for a model that makes them: what a
|
||||
-- per-picture price is charged by, and what its tokens do not say.
|
||||
ALTER TABLE usage ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,3 @@
|
||||
-- How many pictures a request made, for a model that makes them: what a
|
||||
-- per-picture price is charged by, and what its tokens do not say.
|
||||
ALTER TABLE usage ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
@@ -442,8 +442,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
|
||||
`INSERT INTO usage (ts, key_id, project_id, model, served_model,
|
||||
prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
|
||||
request_id, replica, queue_ms, ttft_ms,
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -457,7 +457,7 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
|
||||
r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
|
||||
r.CostMicros, r.Status, r.LatencyMs, r.UserID,
|
||||
r.RequestID, r.Replica, r.QueueMs, r.TTFTMs,
|
||||
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit); err != nil {
|
||||
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit, r.Images); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -871,12 +871,12 @@ func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool,
|
||||
err := s.db.QueryRowContext(ctx,
|
||||
`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
|
||||
completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms,
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images
|
||||
FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
|
||||
Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
|
||||
&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
|
||||
&u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardRule, &u.GuardMs,
|
||||
&u.EndUser, &tags, &hit)
|
||||
&u.EndUser, &tags, &hit, &u.Images)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Usage{}, false, nil
|
||||
}
|
||||
|
||||
@@ -241,6 +241,8 @@ type Usage struct {
|
||||
ServedModel string
|
||||
PromptTokens int64
|
||||
CompletionTokens int64
|
||||
// Images is how many pictures were made, for a request for them.
|
||||
Images int64
|
||||
// CostMicros is what this request spent against the budget.
|
||||
CostMicros int64
|
||||
Status int
|
||||
|
||||
@@ -92,7 +92,10 @@ type AccountPages struct {
|
||||
// with it: the policies in force, whose judges the mirror page serves.
|
||||
hasGuard bool
|
||||
// hasSettings is set when /admin/settings is registered.
|
||||
hasSettings bool
|
||||
hasSettings bool
|
||||
// images says whether some model makes pictures; nil when the
|
||||
// pictures page is not mounted.
|
||||
images func() bool
|
||||
guardLibrary func() guard.Config
|
||||
// guardFor names the guardrail policies in force for someone, and
|
||||
// whether they keep nothing; nil when there are none.
|
||||
@@ -327,6 +330,10 @@ func (p *AccountPages) logout(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
clearSessionCookie(w, r)
|
||||
// What the pages kept in this browser goes with the session: the chat's
|
||||
// conversation, htmx's copies of the pages visited. The chat keys its
|
||||
// own by account as well, for a browser that ignores this header.
|
||||
w.Header().Set("Clear-Site-Data", `"storage"`)
|
||||
http.Redirect(w, r, "/login", http.StatusSeeOther)
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
@@ -113,6 +115,43 @@ func TestTheChatPageOffersWhatItWasGiven(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheKeptConversationBelongsToWhoeverKeptIt(t *testing.T) {
|
||||
// One browser, two people: an administrator signs out, a new account signs
|
||||
// in, and the chat must not open on the administrator's conversation.
|
||||
h := Chat(store.User{ID: "u-42", Email: "a@example.com"}, []ChatModel{{Name: "m"}}, nil, "", "v1", ChatKnowledge{})
|
||||
w := httptest.NewRecorder()
|
||||
h(w, httptest.NewRequest(http.MethodGet, "http://x/chat", nil))
|
||||
page := w.Body.String()
|
||||
if !strings.Contains(page, `<meta name="chat-owner" content="u-42">`) {
|
||||
t.Fatal("the page does not say whose conversation it keeps")
|
||||
}
|
||||
if !strings.Contains(page, `const KEY = PREFIX + "." + document.querySelector('meta[name="chat-owner"]').content;`) {
|
||||
t.Error("the conversation is not kept under the account")
|
||||
}
|
||||
// The unowned key from before, and anyone else's, are dropped before the
|
||||
// kept conversation is read.
|
||||
drop, read := strings.Index(page, "localStorage.removeItem(k)"), strings.Index(page, "const kept = load_()")
|
||||
if drop < 0 || read < 0 || drop > read {
|
||||
t.Error("someone else's conversation is not dropped before this one is put back")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSigningOutClearsWhatTheBrowserKept(t *testing.T) {
|
||||
p := NewAccountPages(fakeAccounts{user: store.User{ID: "u1", Email: "a@example.com"}}, nil, nil, "v0",
|
||||
slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
r := httptest.NewRequest(http.MethodPost, "http://x/logout", strings.NewReader("csrf=tok"))
|
||||
r.Header.Set("Content-Type", "application/x-www-form-urlencoded")
|
||||
r.AddCookie(&http.Cookie{Name: SessionCookie, Value: "good"})
|
||||
w := httptest.NewRecorder()
|
||||
p.logout(w, r)
|
||||
if w.Code != http.StatusSeeOther {
|
||||
t.Fatalf("status %d", w.Code)
|
||||
}
|
||||
if got := w.Header().Get("Clear-Site-Data"); got != `"storage"` {
|
||||
t.Errorf("Clear-Site-Data = %q, want \"storage\"", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSimulatedModelSaysSo(t *testing.T) {
|
||||
// big-model returns Greek letters at a fixed rate. Offered next to a real
|
||||
// model with nothing to tell them apart, it reads as a broken model rather
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type imagesData struct {
|
||||
Nav Nav
|
||||
Contexts []ChatContext
|
||||
Context string
|
||||
User store.User
|
||||
Models []ChatModel
|
||||
CSRF string
|
||||
Version string
|
||||
Now string
|
||||
}
|
||||
|
||||
// Images renders the page that makes pictures, for one signed-in person.
|
||||
// Its models are those that make them, as the chat's are those that write.
|
||||
func Images(user store.User, models []ChatModel, contexts []ChatContext, current, version string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
noStore(w)
|
||||
render(w, r, "images", imagesData{
|
||||
Nav: navFrom(r, "images"),
|
||||
Contexts: contexts, Context: current,
|
||||
User: user, Models: models, CSRF: csrfFor(r),
|
||||
Version: version, Now: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// SetImages has the menu offer the pictures page while some model makes
|
||||
// them. Asked on every page, so it must be cheap: the catalog, not a request.
|
||||
func (p *AccountPages) SetImages(available func() bool) { p.images = available }
|
||||
@@ -0,0 +1,36 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestThePicturesPageDrawsInEveryLanguage(t *testing.T) {
|
||||
for _, lang := range []string{"en", "tr"} {
|
||||
req := httptest.NewRequest("GET", "/images", nil)
|
||||
req.Header.Set("Accept-Language", lang)
|
||||
rec := httptest.NewRecorder()
|
||||
Images(store.User{Email: "a@b.c"}, []ChatModel{{Name: "flux-schnell", Hardware: []string{"6000-24g"}}},
|
||||
[]ChatContext{{Value: "personal", Label: "Personal"}}, "personal", "v0")(rec, req)
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("%s: %d %s", lang, rec.Code, rec.Body.String())
|
||||
}
|
||||
page := rec.Body.String()
|
||||
for _, want := range []string{`<option value="flux-schnell">`, `id="prompt"`, `/images/generate`} {
|
||||
if !strings.Contains(page, want) {
|
||||
t.Errorf("%s: %s missing", lang, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(page, "images.") {
|
||||
t.Errorf("%s: an untranslated key is on the page", lang)
|
||||
}
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
Images(store.User{}, nil, nil, "", "v0")(rec, httptest.NewRequest("GET", "/images", nil))
|
||||
if !strings.Contains(rec.Body.String(), `class="empty"`) {
|
||||
t.Error("no models, and the page does not say so")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"images.title": "Pictures",
|
||||
"images.size": "Size",
|
||||
"images.count": "How many",
|
||||
"images.negative": "Negative prompt",
|
||||
"images.negative_hint": "what the picture should not have; FLUX schnell takes little notice of it",
|
||||
"images.seed": "Seed",
|
||||
"images.seed_hint": "the same seed and prompt make the same picture; empty is a new one each time",
|
||||
"images.steps": "Steps",
|
||||
"images.steps_hint": "more is slower and finer; empty leaves it to the model (4 for schnell, 20 for dev)",
|
||||
"images.prompt_placeholder": "Describe the picture…",
|
||||
"images.make": "Make",
|
||||
"images.kept": "Pictures stay on this page until you leave it; nothing of them is kept on the gateway. What is recorded is the usage: how many pictures, and what they cost.",
|
||||
"images.empty": "No model you may use makes pictures right now.",
|
||||
"images.empty_why": "Either the cluster has no picture model, or your project's allowlist leaves it out. A model that makes pictures is a ComfyUI server in the catalog (<code>api: comfyui</code>).",
|
||||
"images.js.making": "Making… {s} s",
|
||||
"images.js.made": "{n} made in {s} s.",
|
||||
"images.js.failed": "Not made: {why}",
|
||||
"images.js.download": "download",
|
||||
"images.js.reuse_seed": "use this seed",
|
||||
"images.js.seed": "seed {seed}"
|
||||
}
|
||||
@@ -11,5 +11,6 @@
|
||||
"nav.account": "Account",
|
||||
"nav.sign_out": "Sign out",
|
||||
"nav.docs": "Docs",
|
||||
"nav.settings": "Settings"
|
||||
"nav.settings": "Settings",
|
||||
"nav.images": "Pictures"
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"images.title": "Görseller",
|
||||
"images.size": "Boyut",
|
||||
"images.count": "Adet",
|
||||
"images.negative": "Negatif istem",
|
||||
"images.negative_hint": "görselde olmaması gerekenler; FLUX schnell buna pek kulak asmaz",
|
||||
"images.seed": "Tohum",
|
||||
"images.seed_hint": "aynı tohum ve istem aynı görseli üretir; boş bırakılırsa her seferinde yenisi",
|
||||
"images.steps": "Adım",
|
||||
"images.steps_hint": "fazlası daha yavaş ve daha ince; boşsa modele bırakılır (schnell için 4, dev için 20)",
|
||||
"images.prompt_placeholder": "Görseli tarif edin…",
|
||||
"images.make": "Üret",
|
||||
"images.kept": "Görseller siz sayfadan ayrılana kadar burada kalır; ağ geçidinde onlardan hiçbir şey saklanmaz. Kaydedilen kullanımdır: kaç görsel üretildiği ve maliyeti.",
|
||||
"images.empty": "Şu anda kullanabileceğiniz, görsel üreten bir model yok.",
|
||||
"images.empty_why": "Ya kümede görsel modeli yok ya da projenizin izin listesi onu içermiyor. Görsel üreten model, katalogda bir ComfyUI sunucusudur (<code>api: comfyui</code>).",
|
||||
"images.js.making": "Üretiliyor… {s} sn",
|
||||
"images.js.made": "{n} görsel {s} sn'de üretildi.",
|
||||
"images.js.failed": "Üretilemedi: {why}",
|
||||
"images.js.download": "indir",
|
||||
"images.js.reuse_seed": "bu tohumu kullan",
|
||||
"images.js.seed": "tohum {seed}"
|
||||
}
|
||||
@@ -11,5 +11,6 @@
|
||||
"nav.account": "Hesap",
|
||||
"nav.sign_out": "Çıkış yap",
|
||||
"nav.docs": "Belgeler",
|
||||
"nav.settings": "Ayarlar"
|
||||
"nav.settings": "Ayarlar",
|
||||
"nav.images": "Görseller"
|
||||
}
|
||||
+5
-1
@@ -15,7 +15,8 @@ import (
|
||||
// between different programs.
|
||||
type Nav struct {
|
||||
// Current is the page being shown, so its link is marked rather than
|
||||
// offered: "cluster", "people", "quotas", "usage", "mirror", "chat", "keys".
|
||||
// offered: "cluster", "people", "quotas", "usage", "mirror", "chat", "images",
|
||||
// "keys".
|
||||
Current string
|
||||
// Admin shows the administration links, and Role which of them: a
|
||||
// billing role sees usage and quotas only.
|
||||
@@ -34,6 +35,8 @@ type Nav struct {
|
||||
// Org offers the organization page: to an administrator, and to anyone
|
||||
// who manages an organization.
|
||||
Org bool
|
||||
// Images offers the pictures page, while some model makes them.
|
||||
Images bool
|
||||
}
|
||||
|
||||
type navKey struct{}
|
||||
@@ -49,6 +52,7 @@ func (p *AccountPages) navFor(r *http.Request, user store.User, current string)
|
||||
Guard: p.hasGuard,
|
||||
Settings: p.hasSettings && user.IsAdmin,
|
||||
CSRF: csrfFor(r),
|
||||
Images: p.images != nil && p.images(),
|
||||
}
|
||||
// Only when there is an organization to show: a link that leads to "there
|
||||
// are none" on every page is noise.
|
||||
|
||||
@@ -915,3 +915,15 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
|
||||
.settings-form label { display: block; margin: .8rem 0; }
|
||||
.settings-form input[type=text] { width: 100%; box-sizing: border-box; }
|
||||
.sr-only { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }
|
||||
|
||||
/* The pictures page: the chat's bar and composer, and a gallery under them. */
|
||||
.images-num { width: 12rem; font: inherit; padding: .3rem .45rem; margin-left: .4rem;
|
||||
border: 1px solid var(--line); border-radius: 6px; background: var(--card); color: var(--fg); }
|
||||
#status.error { color: var(--bad, #b42318); }
|
||||
.images-gallery { display: grid; grid-template-columns: repeat(auto-fill, minmax(min(100%, 18rem), 1fr));
|
||||
gap: 1rem; margin-top: 1rem; }
|
||||
.images-gallery figure { margin: 0; border: 1px solid var(--line); border-radius: 8px; background: var(--card);
|
||||
overflow: hidden; }
|
||||
.images-gallery img { display: block; width: 100%; height: auto; }
|
||||
.images-gallery figcaption { font-size: .8rem; color: var(--muted); padding: .45rem .6rem 0; }
|
||||
.images-prompt { font-size: .82rem; margin: .25rem .6rem .55rem; overflow-wrap: anywhere; }
|
||||
@@ -7,6 +7,7 @@
|
||||
<script defer src="/_static/katex/auto-render.min.js"></script>
|
||||
<script defer src="/_static/pages.js"></script>
|
||||
<meta name="csrf" content="{{.CSRF}}">
|
||||
<meta name="chat-owner" content="{{.User.ID}}">
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>{{t "chat.title"}}</h1>
|
||||
@@ -131,10 +132,22 @@
|
||||
// In this browser's storage, and only there: the gateway keeps no
|
||||
// conversation. Every read and write is guarded — a private window, or a
|
||||
// browser that refuses storage, simply starts empty each time.
|
||||
//
|
||||
// Kept under the account's id: a browser is often shared, and the next
|
||||
// person to sign in on it must not be handed the last one's conversation.
|
||||
// Anyone else's left here is dropped on the way in, as is the one from
|
||||
// before the key carried an owner.
|
||||
const sys = document.getElementById("system");
|
||||
const temp = document.getElementById("temperature");
|
||||
const tempOut = document.getElementById("temp-out");
|
||||
const KEY = "llmgw.chat.v1";
|
||||
const PREFIX = "llmgw.chat.v1";
|
||||
const KEY = PREFIX + "." + document.querySelector('meta[name="chat-owner"]').content;
|
||||
try {
|
||||
for (let i = localStorage.length - 1; i >= 0; i--) {
|
||||
const k = localStorage.key(i);
|
||||
if (k && (k === PREFIX || k.startsWith(PREFIX + ".")) && k !== KEY) localStorage.removeItem(k);
|
||||
}
|
||||
} catch (_) { /* no storage: nothing to drop */ }
|
||||
const load_ = () => { try { return JSON.parse(localStorage.getItem(KEY) || "null"); } catch (_) { return null; } };
|
||||
const save = () => {
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
{{define "images"}}<!doctype html>
|
||||
<html lang="{{lang}}"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>{{t "images.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
<script defer src="/_static/pages.js"></script>
|
||||
<meta name="csrf" content="{{.CSRF}}">
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>{{t "images.title"}}</h1>
|
||||
<p class="sub">{{.User.Email}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
{{if gt (len .Contexts) 1}}
|
||||
<form method="get" action="/images" class="chat-context">
|
||||
<label class="inline-label">{{t "chat.context.using"}}
|
||||
<select id="context" name="context" data-autosubmit>
|
||||
{{range .Contexts}}<option value="{{.Value}}"{{if eq .Value $.Context}} selected{{end}}>{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<noscript translate="no"><button type="submit" class="link">{{t "chat.context.switch"}}</button></noscript>
|
||||
</form>
|
||||
{{else if .Contexts}}{{with index .Contexts 0}}<input type="hidden" id="context" value="{{.Value}}">{{end}}{{end}}
|
||||
|
||||
{{if .Models}}
|
||||
<form id="make" class="images-form">
|
||||
<div class="chat-bar">
|
||||
<label class="inline-label">{{t "chat.model"}}
|
||||
<select id="model">{{range .Models}}<option value="{{.Name}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}</option>{{end}}</select>
|
||||
</label>
|
||||
<label class="inline-label">{{t "images.size"}}
|
||||
<select id="size">
|
||||
<option value="1024x1024">1024 × 1024</option>
|
||||
<option value="1024x768">1024 × 768</option>
|
||||
<option value="768x1024">768 × 1024</option>
|
||||
<option value="1280x720">1280 × 720</option>
|
||||
<option value="768x768">768 × 768</option>
|
||||
<option value="512x512">512 × 512</option>
|
||||
</select></label>
|
||||
<label class="inline-label">{{t "images.count"}}
|
||||
<select id="n"><option>1</option><option>2</option><option>3</option><option>4</option></select></label>
|
||||
</div>
|
||||
<details class="chat-settings">
|
||||
<summary>{{t "chat.settings"}}</summary>
|
||||
<label>{{t "images.negative"}} <span class="opt">{{t "images.negative_hint"}}</span>
|
||||
<textarea id="negative" rows="2"></textarea></label>
|
||||
<label>{{t "images.seed"}} <span class="opt">{{t "images.seed_hint"}}</span>
|
||||
<input type="number" id="seed" min="0" step="1" class="images-num"></label>
|
||||
<label>{{t "images.steps"}} <span class="opt">{{t "images.steps_hint"}}</span>
|
||||
<input type="number" id="steps" min="1" max="100" step="1" class="images-num"></label>
|
||||
</details>
|
||||
<div class="chat-composer">
|
||||
<textarea id="prompt" rows="3" placeholder="{{t "images.prompt_placeholder"}}" required></textarea>
|
||||
<button type="submit" id="go">{{t "images.make"}}</button>
|
||||
</div>
|
||||
</form>
|
||||
<p id="status" class="note" aria-live="polite"></p>
|
||||
<div id="gallery" class="images-gallery"></div>
|
||||
<p class="note">{{t "images.kept"}}</p>
|
||||
{{else}}
|
||||
<p class="empty">{{t "images.empty"}}</p>
|
||||
<p class="note">{{th "images.empty_why"}}</p>
|
||||
{{end}}
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const say = {
|
||||
making: {{t "images.js.making"}},
|
||||
made: {{t "images.js.made"}},
|
||||
failed: {{t "images.js.failed"}},
|
||||
download: {{t "images.js.download"}},
|
||||
reuseSeed: {{t "images.js.reuse_seed"}},
|
||||
seed: {{t "images.js.seed"}},
|
||||
make: {{t "images.make"}},
|
||||
decimal: {{t "_meta.decimal"}},
|
||||
};
|
||||
function fill(s, vals) { return s.replace(/\{(\w+)\}/g, (m, k) => (k in vals ? String(vals[k]) : m)); }
|
||||
function fixed(n, places) { return n.toFixed(places).replace(".", say.decimal); }
|
||||
// One request at a time, and nothing kept: a picture is a few megabytes,
|
||||
// and this page is for trying a model, not for keeping what it made.
|
||||
(function () {
|
||||
const form = document.getElementById("make");
|
||||
if (!form) return;
|
||||
const csrf = document.querySelector('meta[name="csrf"]').content;
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const status = $("status"), gallery = $("gallery"), go = $("go");
|
||||
|
||||
form.addEventListener("submit", async (e) => {
|
||||
e.preventDefault();
|
||||
const prompt = $("prompt").value.trim();
|
||||
if (!prompt || go.disabled) return;
|
||||
const body = {
|
||||
model: $("model").value, prompt, size: $("size").value, n: Number($("n").value),
|
||||
context: $("context") ? $("context").value : "",
|
||||
};
|
||||
const neg = $("negative").value.trim();
|
||||
if (neg) body.negative_prompt = neg;
|
||||
if ($("seed").value !== "") body.seed = Number($("seed").value);
|
||||
if ($("steps").value !== "") body.steps = Number($("steps").value);
|
||||
|
||||
go.disabled = true;
|
||||
status.classList.remove("error");
|
||||
const start = performance.now();
|
||||
const tick = () => { status.textContent = fill(say.making, { s: fixed((performance.now() - start) / 1000, 0) }); };
|
||||
tick();
|
||||
const timer = setInterval(tick, 1000);
|
||||
try {
|
||||
const resp = await fetch("/images/generate", {
|
||||
method: "POST", headers: { "Content-Type": "application/json", "X-CSRF-Token": csrf },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
const text = await resp.text();
|
||||
let data = null;
|
||||
try { data = JSON.parse(text); } catch (_) {}
|
||||
if (!resp.ok || !data || !data.data) {
|
||||
const why = (data && data.error && data.error.message) || text.trim() || resp.status;
|
||||
throw new Error(String(why));
|
||||
}
|
||||
const seed = resp.headers.get("x-llmgw-seed") || "";
|
||||
const took = (performance.now() - start) / 1000;
|
||||
status.textContent = fill(say.made, { n: data.data.length, s: fixed(took, 1) });
|
||||
show(data.data, seed, body);
|
||||
} catch (err) {
|
||||
status.textContent = fill(say.failed, { why: err.message });
|
||||
status.classList.add("error");
|
||||
} finally {
|
||||
clearInterval(timer);
|
||||
go.disabled = false;
|
||||
}
|
||||
});
|
||||
|
||||
// Newest first, and a batch in its own order: the last of it is
|
||||
// prepended first.
|
||||
function show(items, seed, body) {
|
||||
items.map((it, i) => [it, i]).reverse().forEach(([it, i]) => {
|
||||
const src = "data:image/png;base64," + it.b64_json;
|
||||
const fig = document.createElement("figure");
|
||||
const img = document.createElement("img");
|
||||
img.src = src;
|
||||
img.alt = body.prompt;
|
||||
img.loading = "lazy";
|
||||
const cap = document.createElement("figcaption");
|
||||
const line = document.createElement("span");
|
||||
line.textContent = body.model + " · " + body.size.replace("x", " × ") +
|
||||
(seed ? " · " + fill(say.seed, { seed }) + (items.length > 1 ? " #" + (i + 1) : "") : "");
|
||||
const dl = document.createElement("a");
|
||||
dl.href = src;
|
||||
dl.download = (body.model + "-" + (seed || "x") + "-" + (i + 1) + ".png").replace(/[^\w.-]+/g, "_");
|
||||
dl.textContent = say.download;
|
||||
dl.className = "link";
|
||||
cap.append(line, " ", dl);
|
||||
if (seed) {
|
||||
const again = document.createElement("button");
|
||||
again.type = "button";
|
||||
again.className = "link";
|
||||
again.textContent = say.reuseSeed;
|
||||
again.addEventListener("click", () => {
|
||||
$("seed").value = seed;
|
||||
form.querySelector("details").open = true;
|
||||
$("prompt").focus();
|
||||
});
|
||||
cap.append(" ", again);
|
||||
}
|
||||
const p = document.createElement("p");
|
||||
p.className = "images-prompt";
|
||||
p.textContent = body.prompt;
|
||||
fig.append(img, cap, p);
|
||||
gallery.prepend(fig);
|
||||
});
|
||||
}
|
||||
|
||||
$("prompt").addEventListener("keydown", (e) => {
|
||||
if (e.key === "Enter" && (e.metaKey || e.ctrlKey)) { e.preventDefault(); form.requestSubmit(); }
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body></html>{{end}}
|
||||
@@ -28,6 +28,9 @@
|
||||
<span class="menu-gap" aria-hidden="true"></span>
|
||||
{{- end}}
|
||||
<a href="/chat"{{if eq .Current "chat"}} aria-current="page"{{end}}>{{t "nav.chat"}}</a>
|
||||
{{- if .Images}}
|
||||
<a href="/images"{{if eq .Current "images"}} aria-current="page"{{end}}>{{t "nav.images"}}</a>
|
||||
{{- end}}
|
||||
<a href="/profile"{{if eq .Current "keys"}} aria-current="page"{{end}}>{{t "nav.account"}}</a>
|
||||
<a href="/docs"{{if eq .Current "docs"}} aria-current="page"{{end}}>{{t "nav.docs"}}</a>
|
||||
<form method="post" action="/logout" class="inline">
|
||||
|
||||
Reference in new issue
Block a user